 Okay, so I wonder I'm gonna have a moderator for this talk no one's no one's moderating so I'm just gonna get get started Thanks everyone for coming. I know it's Early and apparently the attendance shows that my name is Micah Abbott. I'm a senior quality engineer at Red Hat I work on the Red Hat atomic host product and the Red Hat core OS product. I'm here talking about white quality as everyone's job I'll give a quick disclaimer though when I proposed my abstract I had a Idea of what I wanted to talk about in terms of quality is everyone's job But when I just developed the slides it transformed more into building How to build a culture around that how to build a team that that respects that So I will still touch upon what how quality is everyone's job, but there might not be as much focus As I planned so first some bad news. I'm not an expert in this There's probably people who can speak to this more intelligently and profoundly None of the advice I'm gonna give is particularly novel if you were to Google some Software development practices and quality practices you could probably come up a lot of the stuff I'm gonna tell you about I'm also really bad at following my own advice So there's lots of things I haven't put into practice in my own day-to-day job my own organization And there's no guarantee that any advice I'm gonna give you is gonna work for your team software development, especially quality is really complicated and It really you really have to know your own team to figure out what's gonna work. What's not gonna work But I do have good news. I've been doing this for almost 20 years So I have what I think is a good amount of experience to make some observations Give some opinions on the matter I am ISTQB certified for those of you who don't know what ISTQB is It's the international software testing qualification board. They're trying to promote Standardized QE practices and best values. I went through their foundation level course Pass their test got the certification I've been participating an agile an agile team for almost four years now This talk will be pretty heavy on the agile methodology down the line I'm a big fan of it having lived both sides of agile and other ways of doing development and I've got a psychology degree and the only reason I mentioned that is because A lot of what I'm going to talk about is building teams how to form relationships within teams And I think the background in psychology really is helpful in navigating those waters So first I want to talk about how we did QE in the past. I'm sure most of you are familiar with the waterfall model. It looks like this you've got Requirements that are made that fall down in design, implementation, verification, etc There's a lot of problems with this model in my opinion and I'm going to start going through them Number one you end up with multiple silos of responsibility Like product management development quality assurance Everyone tends to own that their little piece of work and doesn't necessarily pay attention to what other people are doing in the whole chain And sometimes that generates silos when silos for example You sometimes end up in a quality organization where one team is doing test automation development And then the other team is doing the execution and they might not even be communicating very well or communicating outside of that that larger silo Like I said before everyone tends to stay in their own lane They own their piece of work until they get to the end and then they pitch it off to the next Group and then the next group has to deal with whatever output was given was generated And it doesn't really encourage communication between the groups you like I said when you are so Singularly focused on your output. You aren't really Going back up the chain to talk about how they did something what they did or getting involved in that process earlier on And the one thing I noticed in these models explicitly missing from there are groups like support and documentation There's no box on that that chain that says, you know supports should be providing input here Unless you have a really good product manager or program program manager might be pulling people in like that Typically support and documentation are those and those types of organizations are left out of the discussion When we're owning the only what the when each stage is when the output of each stage is owned by a certain group Any mistakes are then just passed on to the next group and this encourages things like throwing code over the wall Which you end up like I'm sure you maybe you've seen Monty Python where the code comes over the wall QE has to take it and Oftentimes you end up in a situation like this where the code doesn't compile It doesn't build or you end up with a kernel panic and QE is just frustrated because clearly the developers didn't test their code And I touched upon before it's isn't really conducive to having teams work together When you're just throwing code over the wall, you're not really in a good relationship to to collaborate on on the product itself and I found in my experience. It's a poor model for providing feedback to other teams that engineers in my experience when Working with other development teams in this model. You rarely get the hey, thanks for testing this feature Hey, thanks for finding that bug. It's just sort of like everyone's doing their job and focused on the next The next bit of output they have to generate and There's really little to no effort to improve this process You everyone just keeps doing the same thing over and over again and the same mistakes are made often and no Little to no improvement is made I Additionally these interactions that you do have with other groups is when something is typically when something's on fire like There's a deadline that's been missed There's a bug that's been found and you have to release the product and it tends up to be ends up being like a high stakes lots of emotion Environment where everyone's trying to place blame on the other person Or the other group I should say So moving to the present We're starting to see waterfall models get phased out and agile is getting phased in at Red Hat in particular where we have a strong Commitment to doing agile in as many places as possible. We have an agile architect actually who helps drive that message across all the group all the teams And this you know, we end up with in this model. We end up with QE and development under the same umbrella in the same team So we end up succeeding or failing together. So everyone's intrinsically linked to the outcome of the product If QE does a bad job or if that does a bad job Now there is no QE and death anymore. It's just one team that team either succeeded succeeds or fails And in this model, I think it promotes professional growth because you're allowed to Work on things that wouldn't typically fall under your purview like QE can work on dev tasks Dev can work on QE tasks and in some cases, you know, you might own some of the relinch process You get to work on some of the relinch stuff as well And also agile allows you to constantly improve the process So you can look at what happened in the past make an analysis suggest changes try those changes and then Go forward. Hopefully for the better Now when you start practicing agile, you usually end up practicing DevOps as well And that's when, you know, a team the team owns all the phases of development or multiple phases of the development Like for example, the actual development test release and monitoring There could be other phases of development in your model But these are the ones that kind of stick out to me and this and this Model of owning all the phases you need to have excellent communication in your team And you got to work well together in team and this becomes your This is like the key point like you have to have great teamwork in order to see it in this model So going in the future looking to the future, we're probably going to see more agile adoption more DevOps adoption We're going to see more automation on obviously things like continuing to get integration delivery become really important Even more important than they are now because we want to be able to move faster and produce software That's a higher quality at the same same time and I threw in this because Everyone's talking about our artificial intelligence and machine learning I Haven't seen this in practice yet in the QE space, but it's probably going to happen I can imagine someone writing something that would analyze test results and can figure out commonly Commonly found bugs or errors or whatnot And obviously When I'm here to emphasize it's gonna be a lot more teamwork. You got to have good teamwork to be successful going forward Much like they did in anchorman So I'm gonna talk about what I think what I describe as a highly functional team I don't know if I read this or I made it up myself, but it's a team that works well together It's able to solve problems together and achieve your goals together consistently You're not gonna do that every week or every day or whatever, but overall you have a really highly functional unit of People And one of the cornerstones of this I think is open communication you need to be able to talk to your team members completely honestly and Let them know what's happening in inside the team and with the product itself and to do that You should really everyone in the team should feel respected and show respect to the other people on the team And this breeds a tea an environment where no one's afraid to Share an opinion or share some feedback that might be controversial because they know they're not going to get judged When they make that that that statement. I Also think everyone in the team should have a confidant because you're not always gonna want to just go to your team of eight or ten People and say hey, this is what I think we should do because maybe you're a little Where how it's gonna sound you should be able to go to somebody on your team whether it's a architect or a senior team lead or a scrum master for example and say Here's the idea. I have how do you think I should phrase it? How do you think I should present it to the team and that just having that person to bounce ideas off of like can really Help boost your confidence and like get over the hump of feeling scared to talk to your team with What might be controversial opinion? Another course don't I think is appreciation. I think overall in this industry do a really bad job of showing appreciation for each other We should be able to Tell people that hey you what you did is important what you did has value and again That breeds that environment where everyone's starting to feel like they're part of the overall team and they're they're they're part of the overall success of the product of the software development process even a simple thing like Thanks for finding that bug or thanks for you know updating the Jira ticket like That's just like common courtesy, but we seem to to forget to do that In our day-to-day jobs and when you get that kind of feedback especially from someone in a senior position like an architect or someone That's above you or something you have more respect for like that can really improve your day and improve your whole experience on the team and this kind of goes into this This picture which if you haven't seen before it's called Maslow's hierarchy of needs It was proposed by a psychologist named Abraham Maslow in 1943 in a book He wrote and it it shows that we can to reach self-actualization and in software development terms This means like being able to take Be creative with your solutions be creative with your problem solving and taking risks etc So in software development, we probably mostly have our basic needs covered food water security safety Hopefully those are covered when you're working like if not then reevaluate what you're doing Well, you know the food courts closed. There's uber eats or something, you know But when I talk about when I talked about everyone feeling valued and being respectful and whatnot that really keys it falls into the psychological needs So the person like it says here esteem needs prestige and feeling accomplishment like when you get that feedback from your team Saying you're doing a great job. Thank you for doing that like that That helps you fill that that hierarchy and then the relationships That you build on your team fills that piece of the hierarchy So then you can go once you have those Those needs addressed in your team, then you can go and take those risks offer those controversial opinions take some creative Chances with the software that you're working with So how do we get there? How do we get to this highly functional team where everyone's owning the process? Everyone's concerned with software quality Number one you have to nurture the creation relationships within your team You don't have to be best friends with everyone, but you should be friendly with them You should Feel like they're up here that you can talk to that you can interact with And treat them with the respect and value that I mentioned earlier I think it's also important to meet in person at least once a year Especially here at red hat one here, but at red hat we have We have a lot of teams that have remote workers on the team I have for example, I work on for example, there's people in Westford, Massachusetts in Raleigh in San Francisco, Toronto, and then some folks even in Europe So it's really difficult to get everyone in place Once a year, but if you can it allows you to build those relationships. I talk about In such a much more deeper and more meaningful way Because you just just the value of human interaction is like I mean, I can't I can't put more emphasis on it And that's just in the workplace itself Like once you're done with the work, then you go out to dinner Then you go out and have drinks and you get to figure out More about the persons that you're working with and I think that is going to help you in the long term Have a more successful team If you're practicing agile, I think you should utilize those agile activities like In-person stand-ups or video stand-ups I find that Just seeing the faces of the people I work with on video conference because we are all remote is Reinforces like hey, there's another person on the other side of the world that Is depending on me to do my job correctly and vice versa. They're depending on me to do my job as well If you can do in-person stand-ups like that's even better because you do get that immediate face-to-face feedback And in agile, we have the concept of retrospectives where we Talk about what we did in the last sprint the last Period of time and we talked about what went well What didn't go well and we figure out how we can improve the process To get closer to like a real the the best we can be And the closer we get to the best we can be Clearly we're going to get better quality software out of it And like I said before give feedback to your team Celebrate your achievements that your team has made you know, whether it's on social media or on email or whatever Say talk about how what this person did in their in their job in the sprint or whatnot Suggest your improvements when things don't work out. Don't don't hold back. I mean do it respectfully clearly, but You need to keep giving that feedback to improve the team as a whole and show gratitude I mentioned before like just saying thank you for Things that happen every day Really go the different go the distance. I think especially if you do it consistently And finally remember we're all human. We all have the same Not the same, but we all have similar dreams hopes feelings desires Treat other people like they're human. They're not just another face on the screen or whatnot or a You know email address on your inbox So once you have let's say you have your highly functional team and it's running well Now you need to start putting your feelers out beyond your team To again to spread the word of how Software quality is important to everybody and how it's all their their responsibility So this is when you need to get the non-development non QE people involved Bring in people like tech writers program managers support engineers community managers Get them to participate in your development process Let them provide feedback to you and let and let and provide feedback to them as well Like maybe the support engineers aren't As timely with contacting you with critical information or the need for critical information So we can talk to them about that. How can we improve that process or For example program manager if we can talk about how maybe they're not generating requirements that are specific enough or Or whatnot, but get everyone participating on the process And again use the same guidelines I I've talked about before in terms of respect and gratitude and whatnot So now you've got a highly functional team core team You've started to build relationships with external teams now We can start to leverage the strength that we've built with the core team and the external teams This is when you can really start to talk about making quality Everyone's job So you get involved in the development process as soon as possible In the waterfall model, you saw how the the requirements planning happens at the very beginning How many people have participated in requirements planning as a Q engineer for some piece of software? I mean, I know I haven't honestly I don't know if you have then you're doing the good you're doing a good job Well done But I think that that we're missing that a lot from the from the industry And this kind of goes into the this v-model so the v-model shows on the left hand side We've got the software development process starting with the requirements planning down to architecture design that implementation And on the right side, we have the testing and validation portion of the model and you can notice that each for each piece of Development whether to requirements architecture design we have a test Uh corollary so each one of these testing pieces has the ability to validate what was generated on the other side Um, and again, like I said, I don't follow my advice. I'm very bad at following my own advice So I'm not even close to being this this good if you're if you're operating at this level My hat's off to you because it does require a lot of work. It does require a lot of teamwork and But this should be the goal we're striving for you should be striving to Have tests that are able to validate each piece of the left hand side of the model Ah, that didn't work. Okay. So I wanted all those to come out sequentially, but I guess my edit was wrong so Yeah Well, I guess I'm doing a bad job So like I said, uh So in terms of getting involved attend these decision-making meetings if you can if they'll allow you to get in there And even if you can just get in there and listen you can get a feel for You know where the direction of the product's going and then Alter your your how you figure out how you're going to test this product down the line Understand the business deeds that are being met understand the use cases and start asking questions about how this software is going to be tested If you can do that at these early stages at the planning stages the requirements stages Then you can maybe uncover problems in the requirements that will be impossible to test. Um, I Don't have a good example of it, but you can imagine what requirement that would be Come down from product management And then when qe gets it finally at the end of the you know at the bottom of the model if they're not involved They're like I I don't know how to test this like there's no possible way. I don't have a million man hours to do this Um And start to plan your development test test together and this is one is another Uh point where I'm I'm not doing as good a job as I should in our on our team. Um, but if you can plan that that development piece of work and then Side by side plan how you're going to test that and develop them in parallel Then you're going to be able to pick out the bugs in the software a lot earlier Then if the development finishes first and then three weeks later qe picks it up and says, oh, I'm going to write some tests for it You're going to miss bugs and then when you get involved with Development to fix these bugs. They're going to say, well, why didn't you catch this earlier? That sort of thing And if you do that you'll have success kid to show for it so to wrap up waterfall model is bad agile is better if you're doing it properly um You got to live or die together a team like That's the whole I think that's one of the key points of the agile model Practice that open communication be be open with your feedback be honest with your feedback Oh, that goes right into my next point honest feedback Uh and strive to include those external teams get their feedback because they have a different perspective on the software development process They then you might and that's gonna ultimately grow your team as a To be more successful And like I said getting involved early on often ask questions And try to catch the problems that we're going to face as qe as early as possible And that's it so So anybody have any questions comments No, great go Go do good Thank you everybody All right, I'm going to think about it. Um, it's kind of open-ended, but I'm curious to hear thoughts So just so my situation I work on fedora, right? Yep It's it's interesting because it's not a product. It doesn't have a development process, right? Yeah, people think about hey, how do we make fedora do agile? I was thinking about situations like operating systems fedora, rel, whichever you're working on very complex Interlocking products, which don't have a single development team a single development model How do you apply these practices to those do you think? Yeah, that's so in case I'll repeat just for the recording in case it wasn't caught but adam's asking In the fedora ecosystem, you know, there's many different development teams many different developers How do you apply these? tactics to that model You know, I think It kind of goes hand in hand with how fedora operates now like you just got to I mean I know you adam you reach out to everybody, right? You try to be respectful Yeah, you try you try you're trying to do the right thing You're trying to talk to them and get them involved. You're obviously not going to be able to deploy agile across the entire fedora Yeah system, but I think just Trying to get to that that personal side of things like being honest And respectful and showing gratitude towards these these developers They're going to start to come around And start to interact with you more and then you can have a better dialogue of like Well, you know your code sucks. Well, not that we shouldn't say that but you know Your code has some issues, let's say Maybe we could talk about how to improve that or we can talk about what kind of tests we can write to catch The regression we've seen or the problems we've seen I don't think there's a really good answer to that because it's Number one. I'm not an expert. I said that earlier But I think I bet you if you talk to One of the agile practitioners in red hat, they might have some ideas for you Um, you know, since you work for red hat, you could talk to uh, Jen Krieger. She's the architect. She might have an idea So Yeah, yeah, I I would just start with being like Let's start at the bottom like just Be nice Be a be a good person, you know and encourage that sort of that that sort of behavior. I think You know as humans when we see We tend to mirror the behavior we see. I mean, I have two kids and I know that They're basically a mirror of every mistake I've made and every good thing I've done. So It's sort of a human nature thing even as adults where we mirror what we see So if you're being nice to somebody they're more likely to be nice to you and then you can certainly have those dialogues Yep I have a question to her. Yes. Yes. So I mean, I agree with everything that you said and I try to agree with I agree with everything that you said But I noticed that a lot of times It seems that the success rate is much higher when you have qe and development together But the other teams, especially p.m. For example, it's really hard to get them on board Yeah, so do you have any advice for improving that? Yeah, I mean you need I think in those types of in that sort of scenario where you have those external teams that haven't bought in You need to get I would suggest probably getting your management involved and having them reach out to the other side and say You know, this is what we're driving towards. This is what we wanted. We want everyone on board with ensuring software quality And outside of that like I said, maybe you can drop in on our meetings, you know reach out to them I know for example on my team my program manager is ben briard And I've got a pretty decent relationship with him Like I can reach out to him and ask questions And the only reason I in the only way we did that is that, you know, he dropped in the office. I was in the same place We said hello, we introduced each other and we just started talking like people, you know And then now I've got this open line of communication to him where I can say like What is this product requirement mean? Like what are you intending for this, you know? Or do you think we're going in the right direction with the development choices we made that kind of thing? So like I said, it really starts with just those personal Relationships and if you can't establish that personal relationship Then I think you have to involve your management and get your management to bridge that gap Ahead of time Easier than done. I know but that that's the best the best advice I can give you You're welcome Anything else? No, all right. Well, thank you all for coming. I know it's early and Have a good rest of the conference I'm Anisha Narang and I work at Red Hat. I've been in the QE job role for almost five years now I worked with a couple of test automation tools include Selenium python cucumber water and off lately. I started working with protractor Um, other than that, I like traveling and meeting new people So so before I get started, I would quickly like to know that how many of you are QEs in the room Okay, that's a good number and how many of you already worked with protractor or know what protractor is Okay, that's interesting So I'll quickly run you through the agenda of the talk today So we'll quickly see how to get started with protractor We'll have a look at jasmine, which is a web application testing framework that protractor uses by default Next we'll have a look at the global variables and how to use the element locators Um, understanding promises and control flow. Next, we'll also have a look at page object pattern How that helps in maintaining the test better protractor browser logs Generating html reports and a demo at the end What is protractor protractor is an end-to-end test framework for angular and angular js applications It is a node js application built on top of web driver js It is a wrapper around web driver js to control browsers It is specific to angular and understands the internals of the application It knows when angular is done processing and the application is ready for the automated tests to run against One of the advantages of protractor is that it supports angular specific locator strategies Now, what do we mean by angular specific locator strategies? We will have a look at it in the further slides It's pretty simple to get started with protractor Assuming that we have node js installed on our machine. We will use npm to install protractor We say npm install hyphen g protractor Now this will install two command line tools for you. One is protractor and the second is web driver manager You need to start the selenium server before you can have your tests up and running So for that you need, but web driver manager will Take care of all the required binaries and you don't have to install anything separately But make sure that you do a web driver manager update before you do a web driver manager start So now the web driver manager will help you get an instance of selenium server running at localhost colon www.wd slash hub URL that's mentioned Writing your first test. So it's pretty simple to get started with writing your first test It needs two files. One is the configuration file and the second is the spec file The configuration file tells you where to talk to the selenium server Where the spec files are located the spec file can be one file or many files And the framework which is jasmine by default and all the other configurations are taken by default So here if you have a look at the file it reads the selenium address Which is exactly the same URL that we mentioned in the previous slide Next is the specs which we are mentioning as spec.js Which is just a spec file. It could be many files as well And next is the framework, which is the jasmine framework that protractor uses by default Now one This is the spec file So the describe and its syntax is coming from the jasmine framework And browser is a global variable that protractor exports You see browser dot get would load a particular load the particular URL And the expect is the assertions. A spec file can have one or many Assertions that we may want as per the test So before I go ahead, I would quickly like to give you an overview about the The application that we are going to refer while we are testing So here is the angular js.org if you look at it So the test that we will be referring to is here So if we if I input the text anisha in this textbook, I expect that it should show hello anisha there This is the test that we are going to refer to in the entire talk going forward Now now that we are done writing the spec How do we run the test? It's pretty simple. We just have to do protractor contor js It will open the browser perform the test for you and show you the results Before we go ahead and see the demo We will quickly understand a bunch of essentials that we might want to understand Jasmine Jasmine is a behavior driven development framework for testing the javascript code The test suite is defined in the describe function which acts as two parameters the first is the string and the second is a function A string which describes the test suite mainly and function describes the block of code that implements it The second is a global function called it which again acts as two parameters The first is the string which describes the test that in the plain text and second is a function Which is the exact block of code that we are trying to use. So here. This is a way simple Two-line code that we have we create an element object and we capture The text of that element and we are expecting that text to equal hello world global variables Protractor exports these global variables. The first is browser It's a wrapper around an instance of web driver used for navigation and page wide information The browser dot get method with load will load the page for you It under it expects angular to be present in the application that you're testing if the application does not contain Angular does not have the angular library. It will throw an error saying the angular library does not not found Next is element element is a helper function for finding and interacting with DOM elements on the page that you're testing The element function will accept one parameter Which is the locator strategy and return an element finder object for you to interact on the On the DOM of the page by a collection of element locator strategies You can search the elements on the page using by dot id by dot css by dot name by dot model And couple of other locator strategies that we will look forward element locators So one of the most important things that whenever we are doing UI testing is that we need to find the right element Know the state of the application and interact with the element So the couple of element most widely used element locators that protractor has by dot css find an element using a css selector By dot id find an element with a given id by dot model find an element with a certain ng model by dot binding Find an element bound to a given variable. Now if you look at the last two, they are not Very commonly used because these are the ones that are angular specific locator strategies what I mentioned in the initial slides by dot model and by dot binding are the any Angular specific locator strategies and locators are passed to the element function as element by dot css And some and the css that we want to pass there Now, how do we find the element? How do we locate that particular element is we go to that particular element Do a right click do an inspect and we can see here. So now if you see here you will see that Here you see ng model your name and how I am interacting with the element is by using by dot model your name That's how I'm interacting with the particular element on the page Which is the text box that we referred initially Actions the element function returns an element finder object So now the element finder object knows how to find the element on the page But it will not go ahead and interact with the element on the page until and unless an action method has been called On that object. So here if we look at the code on the left side We have the element object and we have the element function which is being passed a locator strategy in it And now we have the action methods the action methods are nothing but the dot click dot send keys dot clear These are all the action methods that we might want to perform on the web application for our test to proceed So all actions are asynchronous and all action methods return a promise. What is a promise? We will look forward in the further slides. So now that they return the promise So to log the text of an element what we have to do is we have to do element dot get text dot then And do a console dot like bog text We have to use the dot then here Now understanding promises and control flow web driver j s and thus protractor apis are entirely asynchronous All functions return promises now until I started working with protractor. I had no idea There was a concept called promises that existed a promise is nothing but the eventual result of an operation Promise helps you decide what to do when a particular operation succeeds or fails Web driver j s maintains a queue of pending promises called the control flow to keep the execution organized control flow coordinates this dueling and execution of commands operating on a queue Now we don't need to write promises because the control flow mechanism of protractor allows us to write promises in a synchronous way We can write our code in a synchronous way despite of enjoying the asynchronous behavior of the application Promises let's make a quick attempt to understand what promises are Promises represent the eventual result of an operation Promises are objects the promise object is used for asynchronous computation. Now, let's assume that promise is an object We're trying to make an sttp get request which is api docs id where we expect that it will return the particular title of document So we it might return the name of that particular document It might not return because of some error occurred So now we define we take that promise object and use the success operation where we say We want to log the title of the published doc And we use the promise object to define what happens when the sttp request fails because of some error So we are trying to log the request failed with the response and the status code Now the control flow control like I said control flow coordinates the scheduling and execution of commands operating on a queue So now how the code would look like when we are not when there is no control flow and how the code would look like when we have control flow If we do not have control flow now that we know that every action returns a promise So we will have to resolve each promise in the code using the dot then So here if you look at the code we have browser dot get it Moves to google.com and then if you notice we have the dot then operation Which says return particular element finding an element With the locator again. We have a dot then which sends a particular keys Which is a web driver into that text box again. We use the dot then and we send And we click on a particular Element so this this is the block of code that we would have to write if we do not have control flow in protractor But now that we have control flow as a part of protractor It's very simple for us. We can write the code in a synchronous way saying browser dot get element By dot ig dot send keys and go ahead and click the button quickly page object pattern The page object pattern simply models the page elements as objects within the test code Page object pattern is one of the best practices that we can follow whenever we are writing UI tests It helps us to manage and maintain the test in a better way It reduces the amount of duplicate code and one of the advantages is that if the UI changes We have to make change at only one place the fix needs to be done at only one place Now how would the code look like if we have if we do not have If we are not using page object pattern, we use the describe We specify the name of the test suit in the it and then we go ahead and load the page, which is browser dot get element by dot model, which is your name, which is nothing but that text box that I showed We are sending keys as annisha in there and then we are Creating a greeting object where we are capturing the element where we are expecting the text to change as per The test and then we do an expect on that particular greeting dot get text to equal. Hello, annisha protract So the jasmine expectations also have the ability to understand promises and that's how we see the expect greeting dot get text Um is working and it is showing you the results to expect. Hello, annisha Now um now when we're using the page object pattern, how would the code look like? So earlier we saw that how We would write the test when we are not using page object pattern, but now we are using page object pattern And we have to create a particular we have to create the object Which is so on the left that you see is the page object file Which is angular home page and defining the function We describe the two elements that we're trying to interact with one is name input and the second is greeting Um, next we also define a bunch of functions Which is this dot get which is nothing But you can reuse this function every time to navigate to a particular url Instead of writing browser dot get in every test file that you're trying to use Next is a set name method that we have created which is again Nothing, but we are trying to interact with the same element and send the keys So here if we notice we've parameterized it so that we can reuse the same block of code and the same function for any test that we are trying to write and use the same element there Next is get greeting text. So this will return the text From that particular element, which is nothing but it does a return greeting dot get text We export the particular Page object to be used in the test spec So now we also need to make sure that we are making the appropriate changes in the spec files when we have created a page object So we do a require saying dot angular home page To the angular homepage object and the path has to be relative to the spec file that we are using Now it is again The test that we are trying to describe which says it should greet the named user Angular home page dot get which is again the get method from the left that we are trying to use that set name We are passing the parameters We know that we've parameterized it on the left and we are adding our expect conditions So all the expect conditions are supposed to be a part of the spec file and we see it here So we have expect angular home page dot get greeting text Which is again the function that we're calling using the function from the left to equal. Hello, Anisha And that's how we can always reuse the page object file in any other test that we might Want to use which involves the same elements that we are trying to use Protractor browser logs assertion. So often when we're doing UI testing we might miss out on some console log errors that on the browser console log errors that might be That might occur some assets might not load correctly So this is one library that allows you to verify that there are no browser console log errors How do we use that? It allows us during the browser console logs in each test for warnings and errors If a particular asset is even not able to load it is showing a 400, but it's not reflecting on the UI This particular library will capture those errors We do an npm install browser protractor browser logs We define a particular object we do a require on it and then we add this block of code in the prepare.js Which is loaded at the start of every test before the test runs Now, how do we go ahead with it is we define we define the logs object and we define a before each function Which resets that particular object before each test is run and we have an after each test So after each test is finished running it will check the browser console logs and verify if there are any errors or not Now the last thing is generating reports. How do we generate reports using jasmin spec reporter? So we use jasmin spec reporter and we use the protractor jasmin to html reporter Just to make sure that we have html reports and the jasmin spec reporter allows us to have descriptive report on the console So by default jasmin does not allow you to have those descriptive reports on the console So we use jasmin spec reporter and protractor jasmin to html reporter for having html reports and below is the How the html report would look like So now we will quickly have a look at the code how the code looks like Okay So if we look here, we have the specs folder here where I've implemented the page object pattern The first file that you see is the name input spec.js That's the spec file that we have and second is pages, which is the page object file. So we quickly have a look at the code How that looks like So if you look closely, it's exactly the same code that we described We have different elements and we have the different functions and we're exporting it And now we will have a look at the spec file. How does the spec file look like? So if you see the require it the path is relative to the spec file that we have We have the describe function here. We have the it function here The angular home base dot get method. We are making a call We're doing the set name and we are adding the expect condition Other than that, we will also have a look at how The prepare.js looks like Okay, so this is how the prepare.js would look like we have the jasmin spec reporter We have the jasmin browser to html reporter We have the browser logs the three libraries that we discussed during the talk We have all of them here and here is the block of code for adding html reporter Next is the block of code for add jasmin spec reporter And the last is for the protractor browser console logs You see the before each and the after edge function there and we are making a call to each of these functions How the conf.js looks like is this so we add the on prepare as a part of conf.js Just to make sure that all the functions that we have described in the prepare.js are called When the tests are run, so we'll Quickly run the test here. How to run the test is like we I told initially it's very simple We use protractor conf.js So it opens the browser it goes to that particular text performs the test that was pretty quick to miss out so This is how it looks like on the console And this is how the html report would look like You can find the demo code on this url. It's a It's on my github add slash protractor 101's hyphen demo, I guess And this was the recording that I did of the same demo that I gave live, which was a success though So we don't need that And thank you any questions. Thanks Yeah, sure You wanted to see on the console or you wanted to see the report So, um, so if the if a particular test fails Then there will be a screenshot attached on the right in the in the html report that where the test case fails So that's also a way. Um, that that is also one feature that you can use if you might want to have a look here So you see, um I've mentioned the safe part the screenshots folder takes screenshots only on failures So you can use that feature as well Questions So does this Um Yes, yes Yes, so it will So the question is that does it actually capture The network errors or just the js errors. So yes, it will capture any errors that will come as a part of the browser console logs Misha, you've used selenium before. Yes. How would you compare selenium to? Okay, so Protractor is again built on web driver js, which is nothing but selenium itself So it's a wrapper around it. So if you are testing an application, which is uh angular based protractor is the best tool to use Yeah, thanks any more questions All right. Thank you. I hope you had a All right, I guess it's time to get started So hi everyone. It's good to see you. My name is greg. I've been working on the set distributed software system Since 2009. It's been a long time Last time I gave this talk we took about 37 minutes and I tried to cut it down But we'll see how it goes. Um, which means that if you have a question that comes up that you want to ask Maybe raise your hand to She yell out it while I'm talking so that in case we run out of time you get your questions answered Who here once came because they think they want to learn something about sef Good Okay, who here mostly does testing during their day job? Okay, who mostly writes software during their day job All right. Good mix. Excellent. Okay So we got like three slides that I'm going to do as fast as I can about sef just so we know what problem we're dealing with So sef is a distributed storage system And provides object block and file interfaces to people who want hundreds of pet terabytes or petabytes of data um A sef cluster is is built on top of the reliable autonomic distributed object store Which is responsible for actually storing data and serving i o and that rados cluster is composed of mostly object storage demons which actually are involved in serving every i o and doing data replication And recovering from failures and then a very small number of these monitors which we're not gonna Which we really don't need to worry about right now that just sort of say okay Here's using the cluster but aren't involved in data movement So if you write an application that runs against the sef rados cluster Then you've got a big cluster even a small one is probably several dozen Dozen demons running all together and then the application just talks to it via a library binding instead of library bindings So if you want to test the sef cluster It's a little complicated. You can't just sort of go like if you want to do power off testing You can't just flip one switch and turn it off Or maybe you have a big giant data center switch that you can turn off But then you don't have a test system running either So we needed some way To do testing of sef and so we came up with toothology toothology is the academic study of cephalopods So it's just a pun on the name And it started about seven years ago So in 2011 I had been working as a very wet behind-the-years software developer for a couple years And it was mostly me and a couple other people sitting in a room together And I was the new guy someone else had started the project several years earlier But we really needed to formalize sef testing At least once we had a situation Where we ran it where we ran a test like you know fio or something and we said There was a bug and they tracked and we tracked it down and we said oh Like this if condition is is backwards. So we swapped it And then a few weeks later one of the other three people in the room ran a test and tracked down And found a bug and they tracked it down and they said, huh this if condition is backwards and they swapped it And we eventually realized that they were the same if condition and it wasn't you didn't take account of enough state But because we were just sort of running test ad hoc Out of shell scripts on our laptops when we felt like it then we didn't have a good way of identifying that problem And so we really needed a solution Luckily we hired a guy who was much smarter about testing and administration than i was named tv And he set out to try and build us a test system The first one involved using auto tests, but it just didn't work great And we couldn't find anything that was accessible to us that would work for distributed systems Because they were all mostly set up around single nodes and we really really needed to be able to say Hey, that machine needs to turn off or we need to kill this process running on that machine over there So tv sat down and he wrote some python code and he called it an orchestra module And the purpose of orchestra was to let us Ssh into remote machines and do things to them And it sort of it presented it presented these machines as python objects One of them was a cluster object and you could run commands on every node in the cluster Or you know a filtered set of nodes in the cluster or get a single node out of the cluster and save that and run a bunch of commands on that one node And You know that worked it let us control clusters. And so then he created the tuthology test running system Tuthologies Tuthology was built up around orchestra And its job was to take a list of tasks that you gave it like set up a self cluster and then You know mount a kernel client and then run some tests on it and run those tests those tasks on target nodes Um in ways that are defined by rules Which I will just illustrate here real quickly So a target a set of targets is just a list of users and machines that you can log into Login to with ssh without passwords so that the system can automatically log in Roles are a mapping from Tines of demons into nodes in the cluster now In our examples, these are going to be set based because we're testing staff But but the roles that are available are not defined by the tuthology framework They're interpreted by the tasks And so in this role we have you know a monitor on each of three nodes and a couple of osd's And a meta this is a metadata server and then a client role And then the tasks are lists of Other kinds of python code and we got a question It's still up okay, so I actually know very little about ansible as a as a as a front end technology, but yes Ansible was not really available at the time Or maybe was available, but no one had heard of it Yeah So and then but like this list of tasks each of these tasks is actually a separate python code file And so the seph one says you know install and set up seph This is a one one for mounting the kernel client And then this is work this is a work unit one which is a generic one that lets us invoke Shell code tests and it's going to invoke it on all of the clients in the system And this particular one is the dbench dot shell script Oh, the tasks are kind of interesting. They can be something called context managers Which basically means that they don't need to execute all in one go The seph task actually sets up a cluster and then it runs the yield command Another command and then another test can run after it while the seph cluster is running And then when they return execution flow these back to the seph task Then it does the tear down of the cluster in the cleanup And then tuthology automatically can combine individual yaml files into a single test so you can like Have a have a file that's your Targets to run on because these are my machines and then you just like keep on changing the The particular test you want to run by changing the key client dbench dot yaml file to something else And it automatically combines those yaml fragments into a single test job Which I think is familiar in many test systems now, but I don't know it was the first time I saw it And when it's done with a job the tuthology logs all that output into an archive directory you specify So you have a set of this is probably pretty small But you have a remote folder that contains all the logs from all the nodes that were in the system You have a tuthology log file. That's the actual test running running output You have a summary dot yaml file that specifies whether the test failed or succeeded if it failed It has a reason whatever whatever failure condition triggered it the original config that was used the version of seph it was run on stuff like that Now that was seven years ago and tv was smart, but it was still only you know a couple weeks or maybe a couple months from time for him So it's a little bigger and different today Tuthology today is mostly but not exclusively used inside of the upstream sepia lab This is about 250 machines that are Available to the seph community. I think they're physically hosted by the red hat open source and standards group but it's just It's it's not controlled by anyone it's anyone except for the seph community itself And it's devoted to running seph tests full time and we grant ssh access to any engaged developer So it's a big system and it takes a little more than just sitting there at your desktop and being like oh Like run this test now. So first of all We have a locking server so that we can say hey I need three nodes. Give me three nodes That aren't going to get trampled on by someone else And you can use this to grab a target's file in addition to actually locking them Second of all, maybe all the servers in the system are busy right now So you can't lock any so you can schedule a job to run later at some later date Instead of running it immediately from your desktop And when you do that it just assembles the ammo files puts them into a beanstalk queue And then we have a tuthology server running with a bunch of worker Processes that sit there and go. Hey is there a job I can run off of the beanstalk queue. Oh, there is Okay, this job takes three nodes. Let me lock three nodes. Hey, I locked three nodes. Let me execute this job Hey, I'm done with the job. Let me store the log files And instead of yeah So we have the lock server and we have schedule and instead of actually scheduling individual jobs We instead tend to specify schedule suites So suites obviously Well suites do test a specific category of the step interface We have suites for the file system. We have suites for the underlying rados object store We have suites for s3 compliant gateway We got suites for a lot of different things And in particular instead of being a full collection a full listing of all the tests we want to run suites are actually directories of the ammo fragments that we can combine And this and this is really useful because it means that if we want to add a new test for a new api We built up that instead of having to do all of the different combinations of things from the tests against that api We just add a new api a new test this api yaml fragment and the system picks it up against all the other stuff So in this example, this is a trimmed down version of our rados verify suite and we've got A couple of different yaml configs that are going to run on everything And we've got a description of the cluster and we can choose whether we want to do thrashing against the cluster or not Which we'll talk about later We have a couple of different local object stores that run on a single hard drive And so we can configure different ones of those and then we have different tasks We've got one for testing whether the monitors recover for whether the api is actually work and for Something that we call classes And so when we run us runs to theology suite then it'll first say, okay Well, you know we run we include both the yaml files in the top level directory and then I go into the clusters directory And I'm going to pick out some files and oh This directory is tagged so that we actually use both files in this directory for all the tests But in the thrash directory, we pick one yaml file So I'm going to pick the no we're not going to thrash the yaml file in the object store directory We're going to run on xfs with our file store back end and I'm going to run the class tests And then the next test is all the same things except running the api tests And the next test is all the same things except for running the monitor tests And then the next test we're going to switch to blue store and iterate through all the all the tests again So it's it's a combinatorial explosion But it's very useful Because when I add a new thing, it's like hey, I need to verify that my new I don't know That my new thing works that I add a new task that says hey like Test that I successfully mount and unmount correctly if I have if I found a bug that does it 30 times more 30 times in a row It doesn't work And then it's still test against all the backends and things We have a whole bunch of suites So coverage is pretty good And I mentioned the thrash directory So this is a very important part of testing sef sef's purpose in life is to deal with failures So we actually need to test it against failures. You may have heard of the netflix chaos monkey That's a very popular concept today It probably we probably read a news article before you wrote this, but it's just sort of a thing you have to do So it's a it's one of those tasks that runs in the background And it runs around and does things to the cluster We have we have thrashers that you can configure to turn osd's on and off randomly But you know only so many at a time so the cluster should still work or maybe enough that it doesn't still work You can configure it And we have thrashers that let you change the way data is sharded across the cluster or force a force a A reshuffle of the data so it has to move while io is still happening Things of that nature and this is these are very important to testing this the recovery state space of sef So that's sort of an overview of the toothology You know user interface, but it's also important to talk about how we use it So first of all we test the development branch I can Write some code up and say all right. I think this might be ready or you know I think this one thing works I want to make sure that this one thing works, but i'm not ready for a pull request yet So I can push that code to our sefci dot get repository And we have a system that automatically builds packages out of anything that gets pushed there And then I can schedule a job In the cp lab to test that to test that code And I can get it to email me or i or anyone else can just go look at our polpito website That reports all the jobs passes and failures and lets you look at logs Second of all after branches are tested, you know by the individual developer We test every pull request that comes into the system. We use github We can get we get a lot of pull requests from people that we don't know and a lot of pull requests from people We do know but don't have quite as good testing as we do So we have to test all that before we merge into the system Tech leads and people who do a lot of reviews have some tools to help them But basically we go through we we make sure that a branch is in good enough state that we think it's worth Actually testing on we merge it into an integration branch with three or four or five other pull requests And we run them through whatever suite is appropriate or whatever sets of suites are appropriate and we look for problems And again those results are publicly available If there are no problems then hooray we can merge that code assuming everyone's ready Everyone agrees if there are problems we figure out why we go back and report to that pull request And hopefully we get an update that goes through it again But it's very important nothing merges into the set project without passing the appropriate set of tests So our master branch, you know things get in sometimes, but it's pretty stable And finally we actually do nightly testing so sets a big complicated distributed system Sometimes there are races that will only turn up one in 10 runs or one in 100 or even one in several thousand And so we run nightly tests all the time against, you know, the in development code We run tests nightly tests against the against these stable branches that we that we are supporting upstream and it's long term stable They will things of that nature Depending on the tests they'll mostly run every they run between every other week and every night And those results are also publicly available So that's all the ways that tuthology is great There are some problems with it some gaps That i'm going to tell you about but I do want to Before I tell you the awful things about my project I want to say, okay, I think we're pretty good at testing We we do a lot of functional coverage I've sort of run through just the framework about that we use for it But we have a lot of specific stuff inside of that framework to make it work Well, we have this seftest rados thing that can issue arbitrary numbers like, you know We run it for hours at a time of of rados operations Of reads and writes and and more complicated things against the cluster and verify that the results are exactly what we expect to get back And we run that against all kinds of thrashers that do all kinds of terrible things to the system We deliberately inject failures with like two demons we inject failures within demons We have we have code in the seft base That we can set to just trigger an assert at particular and important places and make sure that the system recovers correctly We have code that can inject Delays and delivering messages or in granting locks to try and expose those races We we go out and we as it says we fiddle with the raw objects or with the raw disk state And we make sure that the cluster detects that and recovers it in the ways we and then recovers from it Or at least responds to it in the ways that we expect And do theology is not the only way we do testing We have a make check build target in our repository that runs We have it's not as good as we'd like it to be but we do have unit tests that test actual like code modules built around the g-test framework all the way up to up to user interface tests that turn on a very small local stuff demon cluster And issue all the commands that you can issue to the system and make sure that that reacts correctly when you issue The command or when you issue the command incorrectly Some of the other components in the system such as our sef ansible which we use for deployment in most cases Or the sef volume thing that we use for provisioning. They all have they have their own test frameworks that they get used for But with that said there are some pretty important gaps that we've discovered over the years First of all, tuthology handles demons by sshing into into the test node and directly invoking them That gives us a lot of cool features very cheaply Like the ability to to issue standard in and receive standard out and and signal stuff without going through hoops But it means that we don't test well now it's system d or in the past upstart or says five-minute scripts And sometimes that can be a problem like when you expect your script to only allow three restarts in ten minutes But actually it's just restarting infinitely And so when there's a failure that you expect that it causes an assert that you expect to kill the demon It just never goes away Second of all tuthology does its own package installs and cluster configuration now when tuthology was created in 2011 We really expected that most of our users were going to write their own puppeter chef scripts And so this wasn't a big deal But as we moved forward we we had the stuff deploy thing We now use a lot of stuff ansible. We may in the future start using a lot of rook Which is the kubernetes operator for storage from the cncf And other partners like susa uses a thing called deep-sea and none of these package Installation and cluster configuration systems get tested in our nightlies Which means that they might be doing something wrong and we don't ever notice it First of all finally well not finally Performance testing is very important. We're a storage system like we're distributed storage system So no one expects us to get a hundred percent of every iop on visible to every client But they do get angry if we degrade performance by 15 percent from release to release And we really don't have any way to do that in tuthology right now And finally scale testing Is not really feasible It's like tuthology's integrate integration testing system We have that expectation that has to will finish within hours or else the scheduling doesn't really work right And we want them to run as little space as possible because we always want more space There are some proposed solutions and some things that we've actually done So we we aren't doing it yet, but we are talking about you know This week or yes this next week We're we're going to talk about starting to do things like we want to expand the tuthology framework api So we can restart and signal using the init system instead of issuing commands like hey Kill all dash nine on this particular note over there Or hey run sefto sd with the sets of arguments on that note over there and make sure that it doesn't die We are talking about on wednesday morning, although we haven't announced yet We're talking about writing a new api Within the tuthology framework so that we can request installs and then they can be satisfied by a thing That installs you think is a fanciful or a thing that installs you bring using greek and kubernetes Or a thing that installs using deep-sea and that the rest of the tasks don't care about that and just get given an install And we can switch to new ones as they become important and we actually test that these behave correctly We don't have a great solution to performance testing right now And so in the short term we're we actually have a new performance suite Which is running the performance test that we do have from another solution But you know it runs on random nodes in the system So we're trying we're gathering data right now and seeing hey Can we like have this performance test fail if it's 10 slower than we expect or is that going to turn up every third run Things of that nature in the long term We really need to do some kind of analysis solution based off of the performance numbers And the nodes that they run on and the variations that we've seen and do an alerting But that'll be some kind of machine learning thing which we just don't have the Have the expertise or knowledge to do right now Finally for scale testing We we made a decision that actually an integration test framework is is not a good place for for long running scale tests Because you know it's an integration system So in the both in the community and within companies that build products around it We've got groups that are designing tests to run longer term in their own lab allocations We might slice out part of the cp lab for some of these and from time to time They might you know, we might get space from heart from from partners or from downstreams, but That's the immediate solution And we'd also like to in the medium term figure out how to mock up tests that simulate longer term systems For instance in seph Usually the amount of data in the system is not where our problems come from problems tend to come from things like Our s3 compatible object solution having too much metadata for the system to handle correctly But the actual data objects are fine So we can start like injecting very large metadata indices And then making sure that those behave correctly without having the data to back them up But none of that exists yet Finally there are some weaknesses about Tuthology itself in order the framework itself not just how we use it Um right now tuthology has become strongly tied to seph Orchestra should be easy to use elsewhere. It's just a control system And when tb built it he really did design it to support other any other multi node system But nobody else uses it. So it's sort of gotten stuck a little more together Um, I say should have what I mean is you know to avoid this problem It may or may not have been worth the expense But it really would have been good to find at least one other community to use tuthology with us when we built this um If you have this problem today, you should find something that exists instead of rolling your own At this point, we're not really worrying about it too much like They're stable code bases that fit our needs Switching to something would be monstrously expensive and get us very little But we do occasionally keep our eye out and say hey Is there some other distributed storage system that could use a test framework? Maybe you'd like to work with us Although we haven't gotten any takers yet Another problem with tuthology is that it's strongly tied to the actual cpa lab It's supposed to deploy anywhere and there are other people running it But sometimes hard coded values like an expectation that you that you can pull packages from a sep.com domain Sneak in and if you have your own development system because you have an old fork or something Then you can't pull it from sep.com. You need to pull from your own local service The documentation is limited like we have a group of people that maintain and run the service And they you know every several years go and go go have a frenzy of updating for some portion of the documentation But there's a lot to work through if you actually want to set up a system So Right Okay, so There but it's more than just other groups If you need to like write write a test and debug it then you have to actually write the python in your local machine Push it to a repository that these that the tuthology framework can pull from you need to have machines locked You need to invoke that and you need to wait through the Wait through the machines to get imaged and for the test to install and to run before you can get any feedback You need local sep or you need built sep packages now when tvag initially wrote this We actually just all standardized on a particular version of a distribution And so we could build sep on our machine with just make And then it would scp the files over to the remote nodes and invoke them directly But that became infeasible as we got more people and as we needed to test more distributions and more combinations of things And so we need packages now, but that means you need to wait for sep packages to to appear And that make and all this stuff makes it hard for third parties to contribute and for new sep developers to Do anything with their patches because they write a patch and they're like I don't know what I can do. I ran make check, but I don't think it tested it at all There are some solutions If we had to do this again and this was important to us, we should have made regular tear down and set up part of what we do We have a lot of you know infrastructure and configuration as code, but we don't invoke most of it very often We have in the past found groups that came to work with us and they had trouble with setup and install and we sort of did A one one time walk through with them to help them get going and we didn't take a lot of that back Or and and make a changes to the system to to prevent it being a problem in the future We made some paint changes like the packages deliberately But maybe we should have considered how people who didn't have lab access would would be able to respond to that and done some kind of Workaround or bifurcation or whatever And in the past we actually had someone write something called tuthology open stack And that was pretty cool. It was a simple script you could invoke from wherever And and if you had access to a to an open stack cluster Like including a public one it would set up a tuthology instance and run a suite and then shut down and archive the logs to some storage system So you could you could just do one shot tests But it had a few problems. Um, the person who was most interested and it moved on to other things the The the open stack apis at the time were were ludicrously unstable and unworkable So it used the open stack cli commands directly But those have also changed since then so it no longer really functions very well There is at least one group still using it But they it's but the fork they're using while they did fork tuthology to use it and it's diverged pretty wildly at this point We have some better news. There's a thing called vstart runner So in the sef repository, there's a script called vstart.shell and it turns on a local sef cluster Just on on your laptop or your development box or whatever And the vstart runner provides you a restricted api a subset of the whole tuthology framework api And if you stick to that api that it can run it against those local demons There's a lot of tests in the file system tree or in the file in the sef of s file system suite that use this There are some tests in the rados gateway Obam s3 and swift compatible object store suite that use this And it means that as we hopefully transition to more of the tests being built against this api then Then you know external first time contributors can start running tests with it on the thing that they most care about And we are investing now a lot more work to foster a community We well, I I just started a sef test weekly weekly meeting a couple months ago to discuss things If you're interested, it's wednesday morning is my time in the pacific coast and they said that url And i'm gonna start sending out reminder emails We had a cephalocon conference several months ago in beijing that sort of sparked this We met a bunch of groups that we had never heard of before that were using sef and and several of them were like Oh, yeah, we have a development team and we were on toothology and it was kind of a pain to get going But now we've got it hard-coded to our needs and there were like four of them And we're like and they they had all done, you know their own their own localization port And so we're trying to work with them To Or with some of those groups to get those patches back I mean that maybe not directly because you know they took their shortcuts too But figure out what those patches were and what we need to change in the upstream system to make it work better And eventually we haven't started yet, but but there are now stable open stack apis I think lib cloud is is the one that's come up and that actually works now We'd like to rebuild toothology open stack so that we can make so that it keeps working And you know, it's easier for people to deploy systems And that'll have other benefits for us too like the ability to test the toothology framework itself in an open Stack cloud instead of rolling it out to our infrastructure and hoping that nothing breaks Um This is sort of the performance thing that I talked about before we don't have good trend analysis tools in toothology We can robustly fail individual tests by looking for core dumps by seeing if there are errors or warnings in the logs By having you know a task run a test condition and and failing on whatever arbitrary thing you cared about in python It's and it's and it's easy to look at the individual runs whether they passed or failed You know sweets show up as green or yellow or red and then individual jobs do the same But there's not a lot of granularity there to say. Oh a particular job has failed the last 10 times we've run it Run it or hey, this branch is broken or whatever In the future, you know Hopefully we get to fixing this but so far it's just not something that's been this hard enough But a lot of people would would find this to be a problem Um The sweets explode in size like I said is common editorial So as you keep on adding ammo fragments, it gets really really large The the main rados suite is up over 124,000 jobs now Um, and there's no way to prioritize those tests within the framework There's no way of saying. Oh, this is a very important failure to analyze New users don't really or resource constrained users don't know what they should run And it's and you just sort of have to watch it and know whether a test matters if you're trying to get faster feedback We do have some solutions There's a new well It's a couple years old now, but there's a subset functionality that That you can run so that instead of using all the common editorial tests It makes sure that every fragment gets used but maybe not all of the fragments And so you can sort and you can specify. Hey, I want to run subset one of 200 And as long as you then go on to run subs at two of 200 and three of 200 and four of 200 all the way up Then you will eventually get the entire common editorial tests and all of the individual pieces get run in every suite But it lets you run run run them down much smaller So you can get a reasonable test system out of 397 jobs instead of a couple hundred thousand when we use this Um All right. Yeah, so that's that we also have other other things so you can filter That so when you build a test and it gets a name or description That's just the names of all the ammo files So you can filter against you can filter and only take the tests that match a particular yaml fragment Or you can exclude the tests that match a particular yaml fragment If you have a failure in a suite run That you decide it's a problem Then you can go fix that code and then you can rerun only the tests that failed because you think that those are Going to be the ones that are most most useful for you And in the future we may do more stuff, but again, it's not become a critical issue yet What is a critical issue for us is that scheduling is very primitive Like I said, it's a beanstalk queue and jobs just sort of get picked off and on the main like test node system Or the test running system We've got like I don't know the exact number But it's something in the neighborhood of 50 toothology worker demons that just sit there and go Hey, do you have a job for me? Oh, hey, you've got a job for me. Let me knock to lock two nodes or lock five nodes or lock 10 nodes And if you lock five or 10 nodes, then you're then you've got 49 other people competing with you and they only want to lock two nodes They always win And sometimes because it's just a queue we'll have nightlies get scheduled But you know I get impatient and I jump jump the nightly queue with with my test run And then a week later the scheduled nightly nightly job is actually starting to get picked off the queue But at that point, you know, it's the stuff shot one on master branch from a week ago And we're running it through now, but oh by the way, we just like 20 minutes ago scheduled a new run on this week's code And that's just not very useful because eventually this the queue gets backed up enough that we start killing nightlies But we've tested our old code and not our new code There are and this is sort of an illustrative of an issue So we've gotten several hacks and we talk every several months about how this is a problem But then if you ask someone what problems they have with toothology, they forget about this one Because our minds have sort of been molded to it So this is something you want to be aware of when you're building test systems Right now all of our jobs run on two nodes because you know We can mostly squeeze our jobs down into two nodes and then they all and then they actually get executed One of our queuey people looks at the beanstalk queue and says, oh, it's like 7 000 jobs long now And we have three three rados nightly runs in there. So I'll just kill the two oldest nightly runs Um In the soon it's not done yet But a really easy fix is to do something a little more intelligent with the locking where we where the workers can Can order up can order up in queue for locking and say, hey, I need five nodes And then we wait for five nodes and then those get locked before you try and execute any other tests in the future We've we're not quite sure how it's gonna how it's gonna happen yet We've talked about actually just switching to kubernetes and the and it can solve some of these problems But we really need to write a more robust scheduler or you know, install a more robust scheduler so that we can do more intelligent things Oh, yeah, I've only got like two minutes left. Good thing. I'm almost done And then you know, we want to be more intelligent about the way our nightly runs work Okay, so are we gonna stick with it? Well, yes, we're keeping our tests It's proven effective over seven years. We're we're incredibly stable for the things we're doing We've got it running right now. It doesn't take a lot of maintenance to keep doing working the way it is working Um, but we are actively exploring ways to make it better for new users and and developers Should you do automated? Yes, you should do automated testing But let me tell you you should probably not write your own framework There are a lot more of them available today than there were seven years ago And a lot of projects have over the last decade written custom frameworks So if you want to build your own framework, you you should think real hard about it Are you really sure something doesn't work for you? If you're really sure then like, isn't there something that does mostly what you do? What do they do? Why is what they do insufficient and what makes your needs different from other people's? You need to answer all these questions before you build your own testing framework or you're doing the wrong thing Um If you answer those questions and you're really sure you need to you know Do the simplest thing you can small frameworks can always be combined into larger ones And try to keep your components discreet so they can be replaced later That is the end of my talk for more information a lot of URLs And we have like 30 seconds for more questions Hi, so I mainly I work on fedora qa So I'm generally interested in integration testing So something I notice often comes up with systems like this I'm guessing the system is mainly built to test a potentially broken sef in a known good environment Like you have known good fedora environments known good Ubuntu environments and then you run an unknown sef code based on those Is it possible to sort of invert the flow? So you run a known good sef in a possibly broken environment and if not, have you thought about that? We haven't thought about it. We we at this point we freshly freshly image all our machines for every test because Doing that when you're trying to test your system. It's really obnoxious But you could build broken images and run the test against that and see what happens right So you can't at least sort of drop it images. I mean, I mean it's it's not something we've ever ever discussed or actually tried But yes, we could drop in a broken image and see what happened to it Yes Question is how difficult it is to deploy to the logic to test a sef cluster. Yeah, so it's it's pretty difficult I haven't ever done it personally If you go way back in time using This pathology open stack one made it pretty easy and I think maybe if you go grab SUSE's fork, it still is But I haven't actually done that myself And there's a whole lot of services running here that we just sort of tangentially pushed on like the Service that builds that builds set packages the service that hosts and serves them up the fat service that the pulpedo service That stores job results It's not not as easy as we'd like it to be but if you're interested We would love the input on what makes it hard Do you get a hundred percent test passes on commits or do you aim for something lower? um Do we get a hundred percent test passes on commits? not quite we so Sometime like like we just run enough jobs that honestly we notice whenever a package repository goes away And so it's not unusual to have like one or two jobs out of a couple hundred fail on that Just because of some network blip or connectivity issue. Um we When there are failures we audit them and we don't We don't merge any anything with new failures, but sometimes we do have Known failures in a system that are pending on some other pr and we merge and we merge in that state That has punished us in the past sometimes pretty badly So like there's a reason people say should be 100 we're a lot closer to it than Most systems like us that I know of but we don't we don't have a 100 programmatic requirement. It's all human driven All right, well, I'll be around wearing a sep shirt for the rest of the day and also tomorrow So come grab me if you want to have any questions and thanks for your time Really spelled but that's how that's the pronunciation. It's all game. I see how I'm a senior manager of quality engineering at red hat For the last six plus years almost seven years now I've been building quality engineering teams to responsible for testing automating tests for several different Products a red hat you can see some of them listed here such as red hat satellite pole project, which is an upstream project ansible tower And I would like to talk to you today about doing web UI automation specifically using selenium and python You probably noticed that the I don't have a slide So I'm actually using a web page to show you the data There's a reason why I chose to do that and not actually have slides For one thing you can follow along so you can either clone the git repository that's right there on the site You can visit that page, which is devconf usa18.readthedocs.io And the reason why I decided to do this is because devconf the whole idea about devconf is about open source, right? So sharing really trying to collaborate with other people So a couple of things that you get out of if you were to clone this git repository for one thing You're going to have all the the source code for the document itself I know that sometimes people have to write documents and they want to use maybe something that is Easy for you to script against a write code. So restructure text is what I've used here It's it can be a little bit daunting and scary to write anything using restructure text So if anything if you don't want to do any Anything related to automation you don't want to know about python or anything at least you can use this to say Well, this is how you build a document with images and tables and links and that stuff So I'm hoping that you will though Want to do automation and want to use python and selenium, but this is one of the things you can get out of The other thing that I want to also Point out is that I am going to try to cover as much as I can in about 30 minutes That's a lot to unpack. So I'll do my best But once again, if you clone this repository if you just bookmark this page You will actually get access to all the configuration files that I'm using here I'm going to show you how to automate stuff using python I'm going to show you how to use py test And then I am actually also going to show you how to plug it all in into a continuous integration Continuous delivery environment So what's in this whole thing for you? Why am I taking so much time of the 30 minutes that I have to talk about this is that I interview people a lot for jobs And a lot of times they don't have they they come in and they say well, I've done this I read this or I studied this in college But I don't have anything to show for if you take this And you start modifying it At least you have some kind of Portfolio that you can bring into your interviews and say I know how to write tests I know how to use py test. I know how to use continuous integration So I think that you know ultimately you could use this To hopefully get your next position or you know at least to brush up on on your skills right now And a true sense of true spirit of open source What I do ask is that if you make any Changes and if you make any improvements, please send me a patch We'll apply it I will give you credit for it and then you can help me improve the document for anyone else who wants to use it Okay, so there's a lot like I said that I want to unpack here but when we talk about web UI automation Your job as a quality engineer or as a tester Is to actually try to identify the things there are the elements that are on the web page so that you can Interact with it And I'm not going to lie to you web UI automation is a pain a but it's is hard Especially I would say that writing good robust web UI test is really hard to do that Because you have to I'm going to show you this little picture here. This is a a DOM object You can think of a web page as a Two ways that you can interact with a web page you can look at the source code for it Which is the html or you can think of it as a tree Representation where you have this DOM object and you can see that you have the top node That represents the entire web page that you're working on and then for each node that you see here It either represents a section of the page or an element that you want to interact with So why is it such a pain to to actually do web UI automation? Is that a lot of times designers and the developers that will change the look and feel of the page And depending on how you choose i'm going to show you some of the schemes some strategies for you to identify the elements here Depending which strategy you choose if you're relying on an element to be on this node here So it's like two nodes deep or three nodes deep And you build your strategy around something called xpath If that element moves to another node, then your code is broken the xpath expression that you're going to write is just It's completely useless, right? So you have you find yourself Continuously having to rewrite your expression as your strategy. The other thing that is also I guess I should point out. It's not really Maybe it may not be obvious But the whole point is you want to automate and computers are much faster than the human beings So a lot of times when you're working on a web page people put like a lot of you know Ajax see things and there's a bunch of javascript Frameworks that people are using there. So sometimes you have to say, you know when you're devising your test I'm going to click on something something is going to happen I need to make sure that my code is smart enough to wait for that one thing that i'm expecting to happen So it's really an art. It's a pain about so my advice is that try to keep it to you know Keep your tasks ui automation as few as possible make them robust make them cover As much ground as you can to like for integration tests Unless your job is actually to do web ui automation. I guess at that point you don't have much of a choice But you can the way they integrate the interact with the elements and the elements like buttons menus images and whatnot is to Find a locator and if you are here earlier for an issues talk She was talking about how you can look at the source code for example of a web page and this is the you know google's Search box there. I am highlighting the the specifically the search box because in there I can look at information that I can use Right now it's what I have highlighted here is the id of the box, which is lst-ib And that's going to be something that i'm going to be using for for the demo here So you can search for things by text, which is what I recommend then you can search for things by There are css ids or css values and that's the list that I have here And tag names html tag names I Put xpath as the last option here because i'm trying to encourage people not to use it And the reason why is you know for one thing it's pretty fragile if things move around then your xpath changes And then you break a test But some xpath expressions are evil. They're really really hard to understand. So like this example here Because I don't have a good way to get a hold of the element that I want I need to find an element and then walk backwards To find the one that I wanted because there was no clear way direct way for me to get to the element that I wanted to To get to so that's my my advice so selenium is the Framework that a lot of people use so anisha talked about protacker and jasmine earlier today But selenium is the one that i'm really used to and I think it seems to predominate around like people who are doing Wabio automation But it works on it works on different platforms. It's open source, which is pretty cool And it also has a bindings for several different programming languages We're going to be talking about python today, but it does offer bindings for other languages And you require what you do is you install the selenium Application the framework itself For your system And then it provides you with a domain specific language that gives you helper methods To interact with the web elements. So it lets you click on things select things scroll through the web page Really control like the elements of the web page And in addition to selenium you need something called a web driver Which lets you interface with the with an actual browser. Okay, so firefox chrome Microsoft edge safari or anything like that. So you need the combination of those two to actually do web ui automation One easy way that used to be a really nice Tool and it's still there is but this changed a little bit is selenium id And it's an add-on that you install on your on your browser The old version was in my opinion was a little bit easier. They provided a little more value But it was a pain to install it required an older version of firefox The one of the advantages of it is that the selenium id lets you record your interaction So you you turn it on and you start clicking away and and typing stuff and moving things around the web UI It records every step of the way everything that you've done And then when you're done you can just replay it and you could literally say this is my automation Though I wouldn't go as far to say that that's automation, but but you could literally do that right so The other advantage that the old version of the selenium id ahead. It was that it let you Export things Into different languages so all the actions use the language the domain specific language click select go get stuff like that But then he translated that into python So now you're really closer to having full automation because you could just use this thing And then record it export it and add your python code and then you were done The new version which is what you're seeing on the right hand side of the screen there it actually There are good things about it and the bad things the good things is that it's a whole it's really easy to install So if you have the latest version of firefox or now it supports chrome So what whichever browser you have you can just install it through the add-ons page and That's nothing else for you to do. So that's pretty cool The bad things about the new version is that they removed all the bells and whistles that they had And one of them is not being able to export to python So that kind of stinks But what I want to show you here is that you know just to give you an idea what web UI automation really is all about So i'm going to hit the record button here And now i'm just going to give you an example of going to google.com And then i'm going to click here on the search box on the right hand side You can see that there are some two steps two commands were executed and recorded there's an open slash Slash is the root address that I typed right here. So google.com It says that I clicked at something that has an id equals to lst-ib If you remember the screen the screenshot that I showed before that's the google search box Now I am going to you know type some text. So i'm going to um i'm going to search for red hat And then i'm going to hit the search button And it performed the search right all the steps were recorded on the right hand side So if I stop recording now I can Change the speed let me make because this goes really fast and it goes back to what I said about computers being really fast But I can replay this and what you should see now is that it's going to open google It's going to type red hat in the search box It's going to hit the google search button and you know it's um it's going to display the list So i'm running really slowly here for you guys to see it right all the steps of the way But this is I guess an example UI automation Right, so it's too bad that the new version of the id has changed But um, I highly recommend it to play around see see if you like it. It gives you good ideas As you as you are recording your actions, you can actually right click on the browser and it adds a Extra menu option that gives you things for example. You want to say, um, you know verify that a certain Text is present verify that a certain field is present before you move on to the next thing Or assert that the thing that i'm looking for is there so you can see all of that exposed through the uh the interface So trying to move really quickly here. Um selenio id is a really nice tool I hope that you read through the webpage and there's some points there to talk about advantages and disadvantages of it Soroko is another one. It's the same exact thing does the same exact thing as selenio id but it's for chrome only it doesn't really Give you the same I guess the same experience selenio id does but it's still there now the selenio id supports chrome I would recommend you to use selenio id just to play around But the real magic happens when you start automating things using python or some other programming language, right? Because now we have really full control over it and the idea here is as I mentioned earlier You want to use the selenio module? It expose a lot of different methods and these in my opinion they're very intuitive because The name of the method tells you exactly what you're trying to accomplish So find element by id find element by x path and then you know It should be very intuitive what you're what you're attempting to do and I would say that most methods are named in such a way That are very easy to follow along Right, so you can see for example here are some fragments of html and this is the this box here in the bottom You can see this is how you would use selenium to access something by id something by the class name something by You know tag name and so on and so forth So it's pretty straightforward to install this to get it running All you have to do is if install selenio however you get your your python packages nowadays, right? And then you want to install the web driver for the specific browser that you want to interact with control i'm going to recommend google chrome only because firefox requires something called a gecko driver and firefox has been churning out newer versions of of itself Quicker than the gecko driver guys can actually update their drivers So your experience is not going to be that great So i highly recommend go with chrome and you're going to be pretty safe, right? If you're on a mac you can just brew cask install chrome driver But if you're on linux, just you know, I guess yum install dnf install or whatever it is The package that you use the package manage system that you use Here's something cool that I really liked about also using python and selenium is as you start doing ui automation You probably don't know what to do you probably don't know like what am I what strategy am I going to use here? Right? I don't even know what i'm trying to find So I have an example here that is basically going to do two things one And if you were to run this in python, it's going to be really quick But um it it's going to basically open google.com It's going to assert that the title of the page says google And then what i'm going to do is i'm going to open google again and now i'm going to modify google's logo I'm going to inject my own images in there So this shows you that you can interact with the browser elements And it also shows that you have a really powerful way to really inject anything to the page. So you know, it's really really powerful This screen here shows on the bottom you have the code that I just showed you I added a breakpoint So um if you were to when you If you are to clone my repository just mind that i'm using python 3.7 So breakpoint is something that it's only available in python 3.7 There are other ways to insert breakpoints This breakpoint would not be found on the source code So you're not going to have any trouble, but if you want to follow along if you're watching the video later on Just be aware the breakpoint works only in 3.7. So what i'm going to do here is i'm going to basically i'm going to load this script on a python repo I'm using something called ipython And because of the breakpoint you can see that I am at the line that says start google chrome It says browser equals webdriver.chrome So i'm just going to walk through it and the reason why i want to do is because i want to show you how You can interact and why this is a useful idea Because if you're starting out doing web UI It's a good idea for you to be able to Interact with it and not have to write a script run it fail Try it again fail again run it right you don't want to do that so you can actually do this really Interactively and slowly so we're just going to hit the next thing. So here is opening google And then i am actually doing a couple of things i am going to find the element So i'm using element equals browser dot find element by id i'm past the search box id to it And then because i am in the testing business I want to make sure that i got my element back and that's what the assert there is for it says Please make sure that i found the thing that i want to interact with If you don't find the thing that you're trying to interact with then you won't be able to do anything with it Right, so that's the whole idea. So now, you know, we're going to type You know red hat into the search box it's going to Perform the search and that's the end of that that part of the code I'm going to browse to google again I am going to now i'm going to make sure that i got my box And now is where the fun begins because i'm going to inject some code and that's me That's my picture. I just replaced it. It looks squashed. So i'm going to Change it so that it looks a little bit better Right, but that's the whole idea so interact with it if you had more elements here So you want to say you want to search that you know the about link works So you could actually do it right here test it When you got to the point where you think that you got everything that you needed then write the code and you got your script Right, so that's the whole idea the last thing that you probably want to remember is that you want to quit You want to close your browser, right so that it doesn't just especially if you're running this locally Which i wouldn't recommend for a real scenario. You don't want to have like a million web browsers open on your on your system Right, so that's the whole idea of using python and selenium But you don't want to sit there and type all this stuff You want to automate it and that's where we start talking about Using python to write your tests The idea here is that you don't want to sit around and type stuff You want to be able to have something that can be run as executed anytime you want as many times as you want Especially i would recommend that you want to do it through some kind of continuous integration environment, right? And what i have here is that you know because we already have all the We have selenium installed we have a web drive installed what i have here is what is referred to as unit test python unit test Like test it relies on python's built-in unit test module And the idea here is that you create a class for every it's a python class For every test that you want to run right so like a test suite I'm going to speed up really quickly because i got my 10 minute there But basically you can see a couple things i want to highlight There's a setup here which takes care of creating the browser instance And then there's a cleanup method which takes care of closing your browser So that's going to be very important. This is also referred to as a fixture For your test and that's something that you want to happen every time you run a test There's some a little there's more control over this so you can have this happen for every test It could be for the entire module it could be for the class for the class itself So there's some Ways that you can control but the interesting thing here is that That the first line that line 11 line 12 takes care of creating the browser for you and closing it up And then the rest of the code now is just basically please open google and assert things for me Right, so make sure that the first task make sure that the title is present The next one is i'm going to type red hat in the search box And then i want to make sure the red hat shows up in the in the title of the window itself So this is the the whole idea Because we're you know, I don't have a lot of a lot of time. I have the actual code here This is how you actually run it So if you clone again if you clone the repository You just run python put the entire path there and you're going to see your browser start popping up and running things And it's going to be really really quick. Okay So that's python unit test style But if you really want to learn The coolest thing the newest technology out there If you really want to impress people and say, hey, I want to work as an automation person a tester of python Bless you So I would recommend that you look into pytest and pytest has a lot of really cool features I'm not going to go through it right now because I don't have the time But I will highlight that there are plugins which are really really cool And then you have fixtures which have really like fine grain control You can really tell like, um, you know, give it a name and you can specify When do you want the fixture to be applied? What should happen when what should happen when you use the fixture? There's some cool things called parametrize, which lets you do Pass the data to your test So you write test once and you change the data value and then it gives you the you know the The different variations of the tests, right? So what I'm going to do here. I'm going to show you is I am installing a plugin called python selenium And the reason why I chose to put that in here is because when you install python selenium like this If you remember that the previous code I had to create my own setup and I had to say Please make sure to create my browser and then please shut it down This plugin automatically gives you that fixture for free For all of your tests Right. So the way that you tell your test which driver which I guess browser you want to use is by specifying the command line So you say that's that's driver chrome and if you look at the code above, there's no setup. There's no tear down I just wrote my test using just plain python. There's no magic here But they're the magic. I guess there is is this selenium Object here, right? There's this argument that's being passed to it. And this is the fixture that you get for free There's no declaration. You don't need to import anything. He's automatically available for you, right? So I think this is really really really cool Some other plugin that I have here Is x dist and this allows you to imagine that you have like a thousand tests You don't want to run your tests sequentially unless you have to you want to parallelize it, right? You want to get your tests really quickly. So by installing this Python dash x dist you can pass the and flag And then you can specify if you just pass pass dash and auto I think you will figure out how many cores are available in your system and then we'll figure out like how many threads I'm going to spin up and I'm going to split your tests Execute them in parallel and then return all the test results back to you, right? So that's really really powerful here And that's why I wanted to show you that you definitely want to play with with this plugin Then what I want to show you is that the next thing that I want to show you is there's something called sauce labs I'm going to try to get back to it because I want to if I run out of time At least I want to get to the continuous integration part And if I have time I'll come back and I'll I'll talk about sauce if I don't have time Then if you want to ask me about sauce labs just about ping me on you know outside of the room and I'll I'll let you know But the final I guess piece Of um of this presentation is about ci So what's the use of of you having written like a thousand automated tests? And they're just sitting there and it requires someone to automatically going there and and run something or click on something But so the idea here is that you don't want to be bothered right? You want to be drinking you want to drink your coffee? You want to you know check the news online while your tests are being executed? That's really the truth. You don't want to be bothered with by anyone So git lab is just one free Service out there. It's completely free. Okay that will give you a ci CD so it's continue it's continuous integration continues delivery process environment that uh, it's completely free And what I have here when you look at the source code for this repository You will see that there's a file called dot git lab dash ci dot eml And what I've done here really to show what you can do the power of ci is that I created three stages And these are the things that must be executed in a certain order And for each stage that I have here what I'm doing is I have different jobs that will be executed Right, so I have a generate docs. I have a test pie test and I have test sauce labs and then you know for each Stage for each job. I have all the commands Everything that I that I need to happen here, but the coolest thing is this so some of these tests Will require running my automated test the ones that I showed you before running on chrome or running on firefox I don't want to run this on my system So using git lab pipelines what he does is he uses a docker rise And I hope you know, I don't know if you know how many of you are familiar with docker But you know talking about containers of virtualization, right? So What he does is it will spin up a dockerized Firefox and chrome for you and your tasks will be executed against it So there's no infrastructure that you've required on your end, right? So that's really really powerful That you have I think that a picture will probably be a better way to represent here But this is what if you go to git labs and you look at the ci pipeline page You'll see that the three stages So what happens here is that this document that i'm using for this presentation is generated There are some steps involved in the job. So I do some linting Which means I want to make sure that the restructure text syntax that i'm using is properly formatted And if it's not the job fails, I don't get my document There's no new version of the document for me, right? But what I do is I generate the document and assuming the document is actually finished and has no issues It gets published to read the docs Which is that site that you see over there on the url address bar and then Once that is done. I run my tests in parallel All right, so I run my tests on chrome. I run my tests on firefox And assuming that these run then I run my tests on sauce labs. How many more minutes do I have? You got three Sweets. All right, so let me go back to sauce labs. So what is sauce labs? First of all, I work for red hat. I don't work for sauce labs. So I make no money out of this I just want to make sure But I really we use it internally and the whole the whole point of using sauce labs is that You don't want to host your own infrastructure What if you are in the business of testing web UI? Applications for phones for devices, right? Or you have you have a requirement you have to support microsoft, right? Are you going to have a windows box laying around and install everything and maintain it and keep updating whenever new versions of Of the browser come out. Um, I don't think you want to do that You could but I wouldn't recommend it. So using the same exact things that I was using before So using pythons using python using python selenium plugin What I can do here is I can create my matrix of support and that's what you see here. So I have Chrome running on the mac a specific version of the mac I have windows 10 and microsoft edge. I have firefox and linux. I have safari on the mac and there's some Magic here also that you will see that when you run the test itself It does some really cool things and it gives it gives like specific names for the the job so that you know, um, you know like I got a little ahead of myself. Here's the test, right? So I only have two tests My matrix has four different elements So the outcome is that these two tests will be executed against those browser operating system combinations, right? So it will be a total of eight tests Automated in the end for you The way the py task works and when you use the x disk plugin is that the name of the tests are not going to be it's going to be all like Garbage you're not going to be able to tell which which is which so there's some really cool examples here on this File that I that I have here on the on the repository that shows you how you can override the id So you can give it a like a name that makes a little more sense So you can see for example that i'm running a test called tasks on the score page title on chrome Then the same one on microsoft edge, right? so You know it is really cool to use sauce labs because you using their Infrastructure, you're not pay well you pay for it, but you're not paying for in terms of the you know Like the sense of you have to maintain it, right? Because that that's time and money and people that requires and then sauce labs also let you You can actually watch the video of the of the test the recording of it, right? So you can look that's what you're seeing here on the screen For example, I actually have 50 vm's available to to the projects that I work on so I can have my test running 50 Testing parallel, which is pretty powerful thing to have too, right? So I've been told that that's the end of my presentation So that won't and I no time for questions Five minutes for question. All right, cool. So, um, I hope this was useful Like I said, please if you if you clone the repository if you make any improvements if you have any feedback Please let me know At the end of the schedule for the conf that you find a nice little link that says hey vote for this presentation Tell all you how awesome it was. So please do that Because then then we invite me back and I'll love to be back next year here. So With that, I'll take any questions from anyone Anyone Yes question Safe I've been doing that in Java using the third safe thing, but I don't know whether there is some utility in python further Yeah, I mean, I would say that I believe that the plugin will Handle that the question was running tests in parallel being thread safe, right? I believe that the plugin will handle all that for you But I think the other factor that is important for you to keep in mind Is that if you have a specific setup or tear it down? Let's just say for example that your tests require a database, right? So you don't want your tests Relying on the same database if they're going to be changing the data So that's something that you're going to have to to handle also So not just being thread safe, but also the the content that you're going to use All right. Thank you. You're welcome Yes question Is there like a is there like a repo of any like standard sample tests? I may have missed it, but That's a good question. Um, so Yes, there is but it wouldn't be intuitive because for example I'm going to put a link here and I hope that you have time to to jot it down But for example, um, I work I work at red hat And we actually Have all of our tests are fully open sourced Okay, so you can actually go if you look on github Dot com and you look for satellite qe Under satellite qe you're going to find all the repositories everything that we that we write for the automation that we Execute at red hat Robotello is something that I wrote 2011 it's still being around But when you go to Robotello and then you look under the tests folder There then There's some a couple of Different folders, but this one over here you can see for example, like how do I do api tests? How do I do cli tests? So yes, there's there are many Repositories out there, but I don't think that there's anything that is like the definite source of truth for that So you have to look around so you can look at this Um, and you know, feel free to ping me email me And I'll point you to something interesting You're welcome One more question perhaps anyone I'm preventing you from getting lunch. Yes You mentioned that the documentation is automatically updated with the change of tests How do you Like for your stakeholders and just for legacy like referring to how do you look at the history? like is there a way to Like what does that look like? Yeah, so I mean this a specific document Wouldn't I guess it wouldn't Match I guess the the scenario that you pointed out because this is just for an example But I would say that you know what you have to do is Anyone who will like to see what the changes were in you go to either get hub or in this case you go to get lab And then you can look at the At the repository you can look at all the commits So this is where we're going to see the history of the changes that went in Right and if you if in the business of writing documents, right? So then you probably submit your your changes and have someone review them before they get merged So they'll have a review process Before anything happens, right? So the pipeline is executed every time there's code changes here All right All right, cool. Thank you very much for your time. I appreciate it. Thank you I request you all to can you settle down next up we have a very Found mentor of mine and a passionate phd student at northeastern university mania abdi She's going to talk about end-to-end tracing in sef with yeager Hello everyone, my name is mania and today i'm talking about end-to-end tracing in sef using yeager This is a joint work between moc, norcy, stern, bu and red hat and Sef is a larger-scale distributed system and It comes at the consist of many nodes and it is highly praised by community and it has been and it has been deployed over Many data centers around the world Now let's look at the sef Architecture and take a look at how sef works Actually, we have sef nodes And from sef nodes we start from the client and here are the client nodes client nodes client nodes sends data to Backend and write data from read data and write data from end to back end and they are there are tons of Clients and then there are osd nodes which are used to store data within the sef and we have the metadata server which is used for storing metadata information for sef fs And sef clients also communicate with sef storage to radios api And then we have monitors and monitors are used for for maintaining the cluster and the cluster status are insufficient for debugging and Let me take give you an example of sef debugging a problem in sef Here as you can see is a very very simple write request that a user can issue and a client can issue to sef And as you can see a very simple write request for very tiny amount of data First issue a check metadata request and the check metadata request involves many many clients many many osd's and then the user Start to write data to back end and again it involves several osd's And then the user starts to update the metadata and finish and commit the write request And it's and it again it involves many other osd's and clients and nodes And now assume that a problem happens here and something goes wrong and The the request crashes or something gets slow As you can see we have many osd's and many components involves in this operation How can we define which osd has the problem and how can we how can we find out which osd could Or which component could have the problem This is one problem with logging that we cannot extract those information from And the other thing is that if even we can find out which problem which node has the problem We have to look at the box that could even be tons of gigabytes of data And looking in look into those bugs is so hard Especially stitching together different parts of the log that can make the problem is Overwhelming and it's so hard. The other thing is that we are unable to show the communication between nodes through the logs and And the self-community The end-to-end tracing end-to-end tracing is a new approach that creates that creates the request flow from the logs And as you can see and as you can see is in this figure It creates a flow for the request that we generate and issue to the user And it's becoming extremely popular everywhere. For example open tracing has the consistent api between Between all softwares for implementing end-to-end tracing into into system And it is and one of its implementation is yeager tracing Self-community also has a start thinking about having end-to-end tracing into their system And they implement blocking self has blocking as as it's end-to-end tracing infrastructure. However, blocking has some limitations and it's And it's great that they are start thinking to use that But we think that it is very limited and it's better to use another approaches that can provide more advanced functionality for us. So here if you look at the table you can see that You can see different features that I list as then Features of open trade as the features of tracing that we want to support and we compare blocking with a Yeager implementation of open as open tracing tracer. Let's look at advanced features advanced features such as Advanced visualization. Can we have advanced visualization with blocking? No, we cannot have because blocking has its own specific type of presenting request and we cannot replace blocking with we cannot replace blocking visualizer with any Visualizer that we want. The other thing is that can we use blocking in production? Unfortunately not and it has two reasons. One of them is that To be able to use blocking we need to start tracing on and off But it cannot be used in a production system because in a production system to be able to debug a system You shouldn't you shouldn't rerun the procedure and you shouldn't reproduce the logs. However We have the open tracing environment which can enable which can enable online tracing using sampling one is that The blocking provides its own api and it provides its own way of tracing However in a data center that has many many applications and have many many components involved in a single request Having a very specific type of tracing is not good and it's better to be able to have all components Traced with a thing with this With a standard api and open tracing again. Yeager using open tracing provides this functionality for us The other thing is that can we Can we leverage community improvements using blocking blocking is a specific to self? However Yeager and open tracing is used by large community of open source software And they can leverage new at new advanced teams new advanced features that are coming and they can improve their Tracing infrastructure So in this type we are trying to adding support for open tracing Infrastructure as fast as possible by layering open tracing underneath api Open tracing how open tracing works and then have And have an introduction on how tracing works and how yeager implements open tracing and how blocking implements open tracing Take a look at anatomy of open tracing Let's start with client users issue request to clients and then we have trace points We have trace points and each trace points is Used by the tracing api and the tracing api associated to metadata information or to context information to these trace points And they are called melody. They are called Span ID and trace ID a collection of trace points. Also, we are calling them span so We have tracing agent the tracing agent receives trace points from tracing from tracing api and source them and cache them temporarily on the on the on its own system and then send them to Yeager tracing back end tracing back end is responsible for getting traces from all over components in the system Stitch traces that are related to a single request together and store them for further use in the future Let's look at how blocking implements these infrastructures and how yeager implements these infrastructures Look at the figures We have osd and rados as our clients and we have trace points that are issued that are generated with user instrumentation And then we have blocking api blocking api takes trace points and using blocking functionality We associate tracing context such as a span ID and trace ID to those to those trace points and then blocking sends the Send those trace points to lttng lgine lttngine works as the tracing trace engine for That's trace engine for blocking and it stores and caches the metadata the trace information temporarily in the in its disk and its buff and its Sorry, and it's mara and then we use bubble trace and also zipkin to stitch together data to collect data from all nodes around the System and then stitch them together and present the request workflow And as you can see in here bubble trace and zipkin together works as the Tracing backend take a look at yeager architecture and see how yeager works again, we have osd and rados as our client nodes and we have Trace points and these trace points are trans and these trace points are sent to yeager agent through Open to open tracing api because yeager is implemented underneath the open tracing api So they when a user issues a trace point that open tracing api associates metadata information associated span ID and trace ID to those to those trace points and sends them back to yeager agent yeager agent stores them Temporarily and then to udp it sends them to yeager collector the yeager collector collects yeager collects traces from different from many different nodes and then Stitch them together and provide stitch them together and provide a unified view for each request in the system as you can see before the yeager collector sends data back to Source data into it's distributed as storage or any other storage type that is provided by the user It first queues those Data into memory and then sends them back. Also yeager provides sampling for us and for sampling We have different type of sampling that is provided by yeager but this yeager collector is issuing the ye the sampling policy to to other agents and ask them to do sampling for the so as one of the Steps of One of our steps of replacing blocking with yeager was how to map api how to map api of blocking with the api of yeager It was not a very one-to-one mapping and it has Some type of complications One of them is that let's take a look at how we start and A trace point and how we start and span in yeager and how we started in blocking in blocking We have to first call trace function And then we have to use and then we have to use init function to initiate to initialize a trace This work together has many different variations over the self code And it should support different type of different type of calling. However, we can However, we try to replace it with source span function in yeager and we Modify the functionality of yeager api to start the span instead of using yeager Using blocking the spans and then we have the event and key valve function in blocking the event function is used for as for annotating The span with the timing stamp and the key valve function is used to annotate the span with With an integer value or with a string value However, on the other hand open tracing api support block kv Which which is used to annotate everything that we want. We can we we don't we are not limited to very specific type of Type of annotation we can have any annotation. We want using Lock kv and then open tracing have any tracer Which which which is used to attach the trade the application The application component to the tracing engine which is presented in the system However, because blocking is not is not using an always on method and we have to turn it on and turn it off We don't have such a functionality in blocking. The other thing is Is that for propagating the information between two different components? We have the inject function in open tracing api and the corresponding one was encode trace in In blocking api. However, it was not again a one-to-one mapping and we need to modify some And we need to have some modifications to to be able to add To add the inject function the other one is the code trace which is used by the receiver of a request and extract the metadata from the other node extract the context data from the other node and then Build a new trace based on that and it is again. It was not a one-to-one corresponding between extract function and the and and the Decode decode trace function and we again have some modification to make those possible Take a look at our implementation What we have done was that we first want to have the least amount of modification that could be that could be added to the system so if you look at the figure one by one on the left, I Present the blocking architecture and on the right. It is the Jaeger architecture We have safe components and we didn't modify any safe components in in our first round of implementation And we have safe components the same then we have blocking api and we use blocking api both for Jaeger and for For our Jaeger functionality and for blocking functionality and we replace the blocking functionality underneath to to instead of inserting to instead of inserting Information into trace points into lttng It sends them to Jaeger agent and then we we enable Jaeger agent on each node and we connect the Jaeger functionality to to our Jaeger agent And Jaeger agent again sends data to Jaeger collector actually after the blocking api We Removed everything and we replace them with the Jaeger functionality Our modification has 200 lines almost 300 lines of code However for adding new trace point because we want to be compatible with open tracing api and we want to compatible with open tracing with open source community So for adding trace point we adding new trace points We are using open tracing api and we are not using blocking api for new trace points that we are added to the system And what was so hard in our modification? There was no contact consistency on context propagation and we couldn't have exactly one-to-one mapping between the Between the components also safe but to be able to to be able to provide Context propagation safe modifies the api of safe functions But it was not what is desired and we implemented to have a more advanced feature And here is an example of traces as you can see in this figure each Each line shows different component and each column shows different component each line shows a single Span and the lengths of each line shows the latency of this of data span as you can see It's it's a little bit small but In here you can see that we can also we can also able to annotate the latency of each Span within the within the trace the other thing that you can show is that the hierarchy the hierarchy that is visible through the span is the is the causality relationship and happens before relationship between different Terraspond and spans in the system Take a look at our evaluation For evaluating our work. We evaluate disk overhead We evaluate memory overhead and we also evaluate the cpu overhead We want to evaluate because all tracing infrastructure comes with an overhead And first of all we want to see what was the overhead that we impose to the system So our experimental setup for our experimental setup. We had two type of nodes one of them was the note that we set up the It was a physical note that we set up self cluster on that and it was it has 64 gigabytes ram. It has to turn you to cpu it has 10 gigabit per second nick and it has two hdd drives and then for the collector note that we want to collect every Traces from the system together. We use a virtual note that has 32 gigabytes ram and 12 virtual cores and a virtual disk We run our experiment for 15 for 15 minutes and the reason was that we during that 15 minutes We issue several read and write requests However, there are some background activity that happens in the step and for those activity We also generate some traces. So we want to be able to capture those races Then we define a time frame of 15 minutes and we capture And we get our statistics during those 15 minutes Is the disk overhead for writing and reading data and also for the For the background Overhead as you can see the x-axis shows the time duration of 15 minutes and the y-axis shows that trace That the amount of data that was generated due to tracing And we run our experiments for different sampling rates for 20 person sampling rates for 50 person sampling rates and 400 person sampling rate and as you can see it was not a linear relationship between different sampling between different sampling rates Because it's a probability based model and then the other thing that is visible here is that this This model of sending data that we send nothing and then send data and then send nothing and then send data is come is coming from the from the fact that we have the We have caching on both agent side and on the collector side We also look at the memory overhead that was imposed by our Infrastructure as you can see in the figure the x-axis again shows the time of 15 minutes within the system And then the y-axis shows the memory overhead of the memory overhead in megabyte in megabyte and again We run our mechanism for our tests our evaluation for 20 percent for 50 percent and for 100 percent of For sampling rate and as you can see the amount of the amount of memory overhead Was slightly different between different types of the between different types of sampling and it was not a huge overhead And it was less than one person one one tenths percent of memory overhead overall system Then we look at also cpu overhead and for looking at cpu overhead again We the x-axis shows the time which was 15 minutes and then the y-axis shows the cpu usage in percentage And as you can see we run it for different sampling ring 20 percent 50 percent and 100 percent and our sum our maximum Maximum cpu overhead was almost eight percent for the for the collector nodes For the future work. We are seeing to have several future works here. First of all, we want to use the out message to We want to use the out log message because Because cpu code has a rich body of log messages and they also they They almost have log messages all over the code on all components We want to use those log messages to annotate our traces because we believe that those information are really valuable and Reproducing them needs a huge effort. So we we prefer to use those log messages This is one of the first effort that we want to have as our future work The other thing is that we are we are trying to add more trace points on different components. For example, currently Steph has several components, but not all of them has trace points We we try to add more trace points to client side to be able to capture more sophisticated and more complicated traces The other thing that we are interested to capture is synchronization point because in order to be able to present a correct view of the system We need to know when our concurrency is finished When our concurrent threads are merged together and provide a single A single and reach to a single point and we need to capture the synchronization point for that And also we want to have we want to be able to find frequent patterns within those Traces within those traces to be able to have a better debugging To provide better debugging opportunity for the users because for example if we can be Retrived if we can be able to retrieve frequent patterns We can detect we can detect what was happening wrong in other patterns But we don't need to look the holes the whole traces to be able to show that we can We can look we can look at specific parts of the trace to be able to Detect the anomaly in several traces And as a summary we enable advanced end-to-end tracing in seph using yeager We replace blocking with yeager and we kept blocking trace points and As there were and we add new trace points using open tracing api our overhead cpu and memory was less than 1 percent And our discovery was less than 60 megabyte per second Thank you Is this an open source project? Is it possible for me to Take what you've done and put it on a seph cluster and then run it? We are planning to have it upstream and yes, it's good. It's it's open source project because But aren't all the bits already available somewhere on github? They are available in my github, but yeah, but it's not an upstream code of seph I mean I can share the repository with you Yeah, it's all available. Thank you Did you actually make the modifications to yeager or was it all using? You know adapting to the api We only adapting to yeager api, but for adding synchronization point We need to have modification to yeager because currently yeager Yeager is not able to capture synchronization points, but We want to add this feature to yeager as well Because it's very important for us You've mentioned that context propagation is hard and it's slow. It sounds like it slowed you down That was resolved. How much faster would it be if it's already a w3c trace? Spec open and I suspect that's going to get approved probably by the end of october Would that make this easier in the future? By context propagation For context propagation There is one thing that is that was so hard in Seph actually the way that they implement blocking and have instrumentation point in seph and it was They modify seph api and I think They modify seph api and it makes things much harder to get rid of those parts And what we are thinking to use is that actually this is available for yeager We can use thread local storage to keep the metadata information In local thread and then retrieve those information whenever we want to create an span instead of modifying all over the code And change every api in the code Because currently this is the way that they it's implemented. It's not very good And one more question in general Distributing tracing is typically not mainstream today because it's very hard to add it to your code This is kind of a first example I've seen of someone that had tracing and tried to move to a different tracing Which seems like an even additional layer of complexity, right? Because you're kind of rewriting something you've already done Are there any tips or lessons learned from that going forward that may be useful for others? The actually the good thing about open tracing is that open tracing actually open tracing is Is the way to answer this question And it provides a standard api for any tracing infrastructure that you want to have underneath For example, you have a specific function and that's api and you use that api for for tracing your System then there is no matter what tracing infrastructure you are using you can use yeager you can use another tracing infrastructure But since you have the same api you can have different modules on the system different software on your system to be instrumented with the same Using the same api and you you don't have the difficulty to going from this yeager Going from this tracing infrastructure to another tracing infrastructure Where does the sampling happen like it in one of the pictures? It looked like you were doing all the trace points and then they were just getting stripped apart so was less storage space or Actually yeager provides several type of sampling You can have you can one of them is the remote sampling that is imposed by the collector And collector decides which no and collector decides on the sampling rate And this is what we have evaluated on our system, but there is another Approaches and there is another types of sampling that they provided through yeager that enables each agent to sample by itself And the different sampling again Well, so but what I mean is when you're doing sampling does that mean that only 20 of the trace points actually get turned on Or do all they got turned on and the data they generate gets thrown away or Where's the cpu if we if we enable it if we enable it on the agent side we for example take One of take every tens of the if we define it as 10 percent We get every tens of the trace points and then propagate those Generate actually trace trace points for them and propagate them But if it's enabled by the collector as the way that we evaluate our system The trace the agent collects all trace points and then sample them and send them back to you to the yeager agent So the key to getting this really efficient is to do the sampling at the agent. How come you couldn't do that? Sampling at the agent. Yeah, we decide which node which actually we decide on the Request to be sampled and then send send the metadata It is sampled on why didn't you use that's the key to getting really low cpu overhead right is too So how come you didn't use that approach? I actually I I didn't have much time to run more experiments because this is the only so is it feasible Yeah, it's feasible. It's just a configuration file that I have to specify I see, okay. It's different configuration that we have to impose to our system I'd like to can you fit this into the broader context of Why you're doing this like is it's just for an alternative debugging thing or What's the use cases for collecting tracing information? Collecting information tracing information used for many things one of them one of them is debugging The other thing is that performance evaluation of the system for example if we want to evaluate What is the what is the if we change if we have an optimization on the system If we have traces from previous execution and if we know how traces was executed previously and with our modification with our Optimization how traces was how how request was executed and what is that traces now? Then we have a better comparison between the two and we can evaluate our optimization better Yeah Any other question Sorry, I couldn't hear you Any more questions altered. Thank you very much. Mania. That was great talk Hi everyone, can you guys hear me at the back awesome? uh, so Welcome to the systems engineering track. No, which track is this? This is soccer quality Welcome to this and ensuring software quality track at dev conf us. Thank you everyone for being here The next talk is property based testing in python and rust by ann mulhorn and i'll hand it over to ann now Thank you. Okay, thanks All right, so you i've been introduced And um now i'm going to talk about who I am I'm actually primarily a software engineer Am I not on I am on Okay, good. I am primarily a software engineer, but um I don't have a naive confidence in the constant correctness of everything I write And so I care a lot about using the tools that'll help me Catch my mistakes design flaws and so forth. And so if i'm serious If i'm writing it. Oh now there if i'm writing a serious piece of software I'll have some sort of continuous integration thing running and then I care a lot about deploying static and dynamic tools That'll help me get stuff right essentially and um So static tools can be many things and dynamic tools are often testing And that's what i'm going to talk about today specifically a particular kind of testing commonly called property based Okay So why do I think i'm qualified to talk about property based testing? Well, I discovered it a long time ago because I was using the language haskell And haskell has the very first implementation of this idea called quickcheck And I was really Impressed by this idea. It seemed very powerful But as everybody here knows haskell has not taken over the world And so when I stopped using haskell I left this really cool idea behind And in fact, I didn't trouble myself much about that until I was working in python in 2015 And I had a library that I was constructing that I knew was absolutely riddled with bugs and that it screamed for property based testing And with I typed quickcheck python Into the google search box And a few seconds later. I was looking at something that looked really really promising And a few hours later. I was running a bunch of tests using this library And the bugs that I had known were there were flying out everywhere okay, and that that's a Somewhat sad story in its way, but I was very impressed with hypothesis I went from not knowing it existed to using it in a few hours and getting really good results Then I sank into the open source quagmire, which open source advocates like so much to talk about I seized to be a user Entirely and also became a contributor So at first it was the usual things their docs were Maybe a little lagging their development or something and I wanted to contribute to that and then I had some actual needs for hypothesis strategy as I'll mention later and I So and I enhanced that and so forth and so on and time went by And I gave this is now my third talk about this subject that I've given and sometime in 2017 I was invited by the hypothesis developer To become a co-maintainer I was invited by the hypothesis developer to become a co-maintainer and I was actually quite proud and happy to become this co-maintainer and From then on I really haven't contributed to hypothesis at all and that's Sad for me a bit because there's a lot of interesting work going on in hypothesis but the reason for that was because I was working much more in rust And with a little more optimism than formerly when I started working in rust I looked for a property-based testing library And I found one and it was pretty nice. It was called quick check and we used it and Got some benefit from it very clearly but the one I'm going to talk about is Prop test which is younger newer and which is Hypothesis alike explicitly hypothesis alike And from my point of view a much more friendly and easy thing to work with Okay Okay, so you know that I'm a hypothesis booster here The question might be do I actually use this stuff and the answer is yes, I do Okay, so I've written a bunch of these little python libraries here And with all of them I've deployed hypothesis and you can see over here That my code coverage statistics can be quite good And other places somewhat inexplicably to me They're not so good and um some places I don't know So payudev is something that I took over from a previous developer and um I seem never to have really followed up to get the coverage statistics I'll talk a little bit more about the influence of payudev And this one here the one below the double line Is in rust unlike all the others which are in python above the double line And getting coverage statistics in rust is necessarily harder than it is in python And so we just don't know that yet although I guarantee you that it is less than 100 um a point I want to make here is that Um in the open source world I often hear people talking about code coverage is just kind of a metric So people will say, you know, it goes up. It goes down. We feel better if the number is higher We feel worse if the number is lower Um I tend to think it's something more than a metric especially if you can have a well known number A well known coverage number 100 is the most convenient but others are good too And the reason for this is that when you're deploying And there's property-based testing that I'm going to tell you about It gets this coverage in a Very useful and interesting way so that if suddenly your coverage drops That doesn't just mean that your tests are maybe not so good That's a hint to you that right now you should look at that Find out why it's missing that spot that it used to hit You've introduced something new In your code and it could be a bug Or it could be a Nice event you've introduced an invariant that actually causes some code to be dead And if it is dead now is it's good to time as any to remove it in my opinion and simplify whatever you've got So I think that this coverage stuff is nice And useful not just as a metric But as a sort of debugging aid Okay, so what what am I going to give you here? I'm extremely simple code examples so that They will be so dead simple that you will hardly notice them so that I can give you So I can actually convey what I'm trying to convey with them so that the code examples shouldn't distract you at all I'm going to talk a little bit about things that I learned And also I'm going to refer to other things because this whole property-based testing world is Really big and getting bigger and I can't really Cover I can only cover a smidge of what this is all about in a time allotted here I'm an extremely simple example This is something that's going to convert take a natural number which is actually An integer which is non-zero in python and given a base. It's going to convert it to a String of digits that represent that number in that base Okay And so here's the old way the so-called example-based testing to contrast it with the property-based testing And what I do is I think of good examples and predict What they should do and I write all that down and then I run My tests and hopefully they all succeed So what you see here is checking that it does the right thing for zero Which I've defined as gives you the empty string and other things explained here And there's the ellipses here Which means and I would do a whole lot more To cover as much as possible. Okay, so that's example-based testing Doctrineer I think that when that's what you can write. That's what you should write. I think that that's better Than nothing By a margin which is considerable So What is good about these tests? Well, they catch bugs and that's good and we want to do that and they catch Surprises when you do something new somewhere else and that breaks Something that your old code previously relied on that's good too And they are Partial specification of the code they test They give some idea Of what the person who wrote this or the tester thinks it should do But they're implicit because you have to look at these examples I just went back to the previous side You have to look at these examples and try to figure out what the person who wrote the code was thinking their code should do So they're implicit Here's the property-based testing that I'm advocating And the idea with the property-based testing is you say something you express something that should be true About your function and what I've done is I've actually stated What should be true about my function as a theorem? So I happen to have a convert an extra function called convert to NAT And all I'm saying here is that if I run convert from NAT on my natural number with a base And then I converted that to convert to NAT. I'll come back to the same value that should happen Okay, so I happen to have this extra convert to NAT Function and these are logical symbols that say for every value b that's greater than or equal to zero That's all the natural numbers. I'm implicitly assuming here that they're whole numbers And for all bases that are greater than or equal to two because one base one is uninteresting This will hold true So that's my statement about this function And now here what I've done is I've copied it into hypothesis I've used the hypothesis Functions and decorators and some so a fourth to get this And what you should see here Is that it really just is a copy. I took a mathematical expression And I changed it Into an expression written in the code. It's more verbose. It's less nicely typeset, but it's almost the same thing We can see that here V and value are the same V and base are the same And I'm using this hypothesis strategy That gets python integers And exerting that I'm only interested in ones that are greater than zero for values or greater than two for base And then this is just a hypothesis function Which asserts Exactly what I'm saying up here Okay, it's as straightforward as that Okay, and here's another example, which is simply a repetition of the same ideas. So Just shows that you can turn them out pretty regularly in this one. I'm just saying that The strings I generate Will not have leading zeros. That's how I've defined it I like it that way It's nice and consistent and so I Didn't write out a mathematical expression for that But this is how I say it in python Okay, so just to show you that there's variety even in this extremely simple example And it's in a file and so forth And then the question is what benefit do I get From having written this thing And the benefit is very nice. We assume that I'm running this code under um in some Python testing framework either py test or unit test. Those are the ones that hypothesis is compatible And so hypothesis will go ahead and it will test That specification it will test that that specification holds true over and over and over again So hypothesis will do the work of selecting arguments for the value in the base It will make up those arguments randomly More or less It will run that function that test function. I had It will check whether it failed or it succeeded And if it succeeded it will keep on going Coming up with more possible inputs and it'll keep on going Until that assertion fails in which case it'll stop and report that it's found a counter example Or Alternatively It will stop whenever and whenever whenever your configuration parameters that you've said say it should Okay, it'll keep trying Until it's told to stop or it finds a counter example And then obligingly if something failed It'll hang on to that information Okay So as I said, I'm not doctrinaire. I think example based testing when that's all you've got is a good thing But I think the property based testing Is really great So if you write hypothesis tests, you still only have a partial specification of your code You may not have been able to say every single thing That your code should do But you've converted it from an implicit To an explicit statement About what you believe your code should do So the reader of your test doesn't have to intuit from a bunch of examples What you're thinking it's right there in the code and that's good So also you get more real tests With less code these are not the same they in a sense. They are the same tests over and over again But because they vary the inputs They do something different So you write less code To get the same amount of tests which are arguably just as good And for me personally, that's a nice benefit because I I'm one of those early testers I'm testing as I'm writing And with example based tests Maybe you've generated quite a few and then you realize that you want to Change the arguments to your function that you had somewhat misdesigned it and Practically speaking it's annoying to change 50 example based tests once you've just discovered this thing but with With property based testing if you've just written that one specification, then that's one thing to change Tidy and quick Another thing you get from this is you employ code reuse in your choice of examples And that's really important because Generally if things are going well, you can use a hypothesis strategy And those hypothesis strategies have been written by people who actually know about the values That they're fine that they're those random values that they're selecting so they know Often more than you So for for my first example, I don't know why I was using this but I was using um Python's decimal class And so the first time I ever used hypothesis The strategy started producing random instances of decimal And it turned out that that exposed A naive idea in my head about what decimal was and actually how it worked. So the The strategy since it was written by a person who knew more about decimal than I did tried certain varieties of decimal number that I didn't even know existed and Broke things all over the place, which was good, which was what I wanted So that kind of code reuse is very nice. You hand off all the work to some expert in Decimal or floating point or whatever and you don't have to Do that, okay And then the other thing is this database of failures that hypothesis keeps around That's really nice too because Now it has a property that it wants to check But the next time you run after you fail it goes straight to the thing that failed last time And tries that first So you can actually be Less clever than you would want and perhaps want to be with um, then you might have to be with example-based tasks where you probably Do something to make sure that the example that was interesting somehow got to the top hypothesis does that for you. So that's great And in fact hypothesis does a huge amount of other stuff, which I'm Going to sort of mention at the end and let you decide if there's any particular thing You want me to talk about more Okay, so so I've been talking about how great property-based testing is and the advantages you get But it's not actually easy To use it. So it's harder than example-based testing Sometimes it's difficult to come up With a useful specification You can't think of anything to say about the function you wrote And as a former academic I'd say that's a bad sign already so I would say you have no business writing that function if you don't know what it does or something like that And I I still stand by that But yeah, it does it does require some mental work to express things and And that makes property-based testing harder and you can always this is great. You can always cheat with the example-based tests And people are tempted to do that. So you write an example-based test. You think oh, these are some interesting values So you stick them in then you stick some Random value in the expected result and then you run it and then you see what the real value is and you're like Oh, yeah, that's what my function does and you plug that in and you have a test and My point here is that property-based testing Kind of makes it impossible for you to cheat in that way and that's good Some things are just going to be harder to express using hypothesis Um, the whole idea is that we have these sort of universally quantified statements And then we can say a lot of stuff because python is very expressive about what should happen but some things are difficult Or sometimes, um, you can sometimes just use hypothesis in a silly way And I I think I've done that a couple times because I was already using hypothesis for something And it was cheaper to use all the infrastructure of hypothesis to get a string That didn't even have to be random Because randomness didn't affect the quality of my test, but somehow it ended up there Sometimes time is a problem and I've certainly encountered that with more numerical things where you want to test something But you can really see that hypothesis is cranking along and it's taking a long time Um, to actually search even a few examples because it's just computationally expensive Or you might you might run into a practical difficulty Where you want random values and hypothesis Doesn't have a way to generate those particular random values for you hypothesis contains many strategies for generating typical python values And also, um, it contains many higher order strategies Which allow you to build An assemble strategies between those higher order strategies and its internal strategies You can build More powerful strategies and that works great and I know that I checked this it took me Closer to two years than to one year of Did well no it took me 18 months is one and a half years It took me 18 months to get from the point where I was using hypothesis I started using hypothesis regularly to the point where I needed to actually write My own from scratch Strategy before that I could always just synthesize strategies using hypothesis higher order strategies So, um That's another I think vote or a demonstration of the power of hypothesis as it is I'm also going to mention that I had a prejudice About property based testing which I lost due to the offices kind offices of a person I never actually met So because I was first introduced to property based testing by Haskell and anybody who's written in Haskell knows that Haskell is the the pure functional language and Usually if you're writing in Haskell the demonstrations involve abstract syntax trees and fun things like that and it's all kind of It's got a nice functional feel and so I I associated property based testing with pure functional stuff But I was really wrong about that and I found that out when I took over this high udev Library, which is just a python wrapper for libudet at this point Many of you could be forgiven if you're not storage people for not knowing what libudet is And so I'm just going to mention briefly What it is so that you understand what I mean by not really a functional thing You all know about devices we have devices on our computers and they're like the mouse and the keyboard and the hard drive And there's this device abstraction Which unix introduced and which is very useful And besides all those familiar devices that I talked to you about just now that we all think of when we think of device There are lots of subterranean devices that support those visible devices that we care about And also if you're a storage person like me, they're also Super devices that are built from the devices that people normally think about so there's lots of devices cooperating And udev That dev part is about devices udev is just essentially A database not in the sense of a relational database, but just a table udev is a table you identify the device and then there are some properties of the device What does two mean? two minutes okay And then questions all right so I'm actually going to stop talking about pi udev and you can ask me about that later But the point was it's all about devices and The original developer had actually had invented his own system of property based testing And I went ahead and hypothesized it up for various reasons and because of the combination of hypothesis And also And also the fact that I was running on funky machines I found more bugs bugs in my udev bugs in lib udev And a bug in the actual udev database it couldn't agree with itself and that bug. I think is permanently stuck there But I found it so There you go. Okay, python to rust. Do I have one minute? Okay, thanks Yeah, so as I mentioned before I did quick check at first because That was really what was available. Prop test is the newer one And the difference between quick check and prop test which you can find out about because they had many cordial discussions about this Is it's about how counter examples are minimized and you'll notice that in this short time I didn't even talk about counter examples and minimization But what I have to say is that prop test just works much more nicely for me Prop test is good, but it's not as mature as hypothesis And what I did is I just reworked The first example that was in python Into prop test and you notice there's a macro here Instead of a generator because rust supports macros. There's this little test annotation and then I've specified my values a little bit differently But it's really the same thing just in a completely different language And done in a completely different way, but the same thing and that Should get me to the end And I'm saying now you can ask me questions and you should feel free to contact me about this topic Because I would like to help you if I can get if you're interested in getting moving In many ways about this. Okay, so Questions Prefab I can't answer that definitely but they talk about numpy a lot Um, so that's that's not really my area, which is why I can't but um, they may Um and another thing that I didn't mention is that um There's And the hypothesis site there are external hypothesis related activity um, and Well, I've actually contributed to that to writing my own strategy and if you were absolutely desperate And this didn't exist You could write your own and put it up there and it would be it would be featured And can you just repeat though? Oh the question was about numerically heavy algebraically heavily specific things And um does hypothesis have strategies for that kind of stuff and I couldn't answer that directly But I do see numpy floating by so there's hope, but I can't be specific Yes Okay, here's how to run it says if it's for a random space Say sorry say that a little bit more You're running it like in continuous integration. Yes Failure that maybe didn't rip one. Maybe you're running something subtly differently. Is there some sort of output that Get so that you can instead So doing the whole space you can start with the thing that causes the problem in the other environment Right, so that's a good question. I think in the old longer slides I explained that that that's a problem Because it is hard To figure out how to reproduce precisely how you got there hypothesis It's not so bad because hypothesis will actually Tell you the failing examples as its output And so that could get you a reasonable amount of way there So so in fact, I should be more positive hypothesis will say these are the arguments to the test function Which caused this to fail and that will all be visible in the ci. Yeah How did you deal with the service inherent complexity sometimes in writing specifications for things do you? Learned the situations where the tests themselves have bugs because of the nature of writing that and how do you deal with that? Yes, um, so the question is well, what if your tests have bugs? And that's a good question because your tests will have bugs on occasion Right. Yeah. Um, and that's that's an interesting rabbit hole and I I do not Say I have any advice. I once told somebody you have to write your test You have to test your test, you know, you have to have this infinite regression But obviously that's not practical advice. Yeah, so, um There are two things that can happen one is your tests Let bad stuff through And that's unpleasant that can definitely happen. The other one is that your tests fail because They catch good stuff essentially you wrote them in a way such that they fail because they're They're perhaps They're either you either wrote them And they're a bad match for what you meant to write or else You were wrong and you wrote them and those those are those are two different things But both of those are less troubling Than the other one when you just let the bugs through Yes all right So sure Should I just start yanking clothes out? Oh, yeah, you can. Okay. I don't want to cause any trouble Uh, excuse me guys, it's just a couple of announcements. So, uh, if you are not aware we are having DevCon party in the evening if you wish to attend you can pick up the tickets at the registration desk and Tomorrow morning for the DevCon keynote. We are having the red hat CDO and the founder of code code Newbie speaking that will be at 9 30 in the big room. So please do attend that things And I Know that one is you guys this one is mine. Thank you Thanks Oh Oh, it's over here, do you want to use this one? Do you want to use this one? Yeah, I mean, this goes out of range. So if you plan to walk or even stand here? Yeah, I mean, it's very good. Yeah, it's good. Do you want this one specifically or? Yeah, I mean, this one also works. No, no, this is working. This one. So whoever plans to stand here can use this. If you're planning to stand away, then you should use that. Okay, so not this one. All three are active. All three are active. So yeah, you can use that if you're more confident. That's the clicker. Yeah, if you want to use the clicker, you can just put this in. Check, check. Cool. Awesome. Check, check. Check, check. Yeah. Right. Yeah. Can you close that? I think that's the issue. That's why we can't see the slides. No. Oh, it's on the next person. It's just not. Yeah. Go back. How can you point here? I wonder if I can just click it physically. It's there a power button. How's it off? It's off. Oh yeah, that would make sense. Oh, there we go. You guys have a minute. Yeah, you have a minute. Just come to the side. Okay. Do you want me to introduce you guys or? I can't, sure. I mean, I can just give like a welcome to Tom. Sure. Hello, everyone. Thank you for coming to the second day of the ensuring software quality track at DevCon to us. The next talk is open shift on open stack by Eric and Emilia and I'll hand them over to them. Hey, guys, my name is Eric. And my name is Emilio. This summer we worked in Boston on the open shift on open stack team and this is our presentation on overcoming development challenges. So show of hands who here is familiar with open shift? Cool. All right. So for those of you who aren't open shift is an application platform that runs at Kubernetes containers. And it can be used dynamically scale applications as well as update your back and on the fly. So it's a really cool tool. And then another show of hands who's familiar with open stack? Cool. So open stack is cloud infrastructure management platform and it's used to manage and lease resources and hardware and stuff. All right. Cool. So for our work this summer, we worked on the open shift on open stack team and open shift can be run on open stack and that's something that we are trying to accomplish. But why you might be wondering do we want to do this? So basically when you run open shift on open stack, it makes it a lot easier to deploy, manage, and scale your open shift cluster. And you can do this by taking advantage of the compute and the networking and storage services in open stack. Now developing in an environment that's sitting between two really large technological projects can be pretty complicated. And this is something that we've been taking on for a little while. But there are working versions of this. For example, if any of you are familiar with the MOC, that is an open shift on open stack environment. But anyway, so our team works on code related to deploying open shift on open stack and we are trying to standardize and optimize that process. So throughout the course of this summer, we faced several challenges. The first one being adjusting to a new development workflow. That's something that most developers will have when starting a new project. More specific to an integration project such as open shift on open stack was making technology to meet in the middle. And so we're going to dive into each of these one at a time and use our summer projects as examples for each. So first off, we're going to talk about getting adjusted to a new development workflow. And we're going to talk about how the tooling and the technology that you get exposed to are going to come into play here. So first of all, the first thing that you have to do before you get started on any project is obviously communicate with your team. And in terms of the open shift and open stack team, that can be a little bit more challenging than others because our team is actually distributed across the world. And so in order to get started, you're going to have to use a tool called IRC. And how many of you are familiar with IRC? Wow, that's shocking. Well, good for you guys. And then so the second fun part about that is since everyone's all over the world, they're not on your nine to five schedule. And so everything that you want to do has to be built around that time zone. And that's going to become increasingly more challenging as your problems get harder to solve. And so as you move along, obviously you're going to have to eventually hit a point where you ask for help. And the first thing that you have to learn about using IRC is you can't be afraid of the chat. And I think that's something that everybody inherently has. At first, I know for a fact I was terrified to post in all chat and look stupid for a long time. But eventually you reach a point where you realize, there's going to be someone who wants to help you. And if you post in all chat, you're more likely to get that help. So do that. Second of all, people might always be willing to help you with one problem. And that's fine. But don't just ask for help. Ask for resources. A lot of the times people will be able to link your documentation or things that you didn't know exist to help you solve your problems on your own. And that could just make the development process easier for everybody. Okay. So then there comes DevStack. And this is another part of developing for OpenShift on OpenStack. If you develop on OpenShift in general, or rather OpenStack, you're probably going to work with DevStack at some point in your life. And basically what DevStack does is it builds a little miniature version of OpenStack in your own local environment. And it's kind of neat because you can build from your own source code and deploy it however you like. But just as anything that's software related, there are obviously some complications that come along with that, which Eric's going to go over. So have any of you guys ever set out DevStack before? All right, for those of you who haven't, buckle in. First thing you're going to need to do to set up DevStack is stacking. So basically you just have to run this script. It takes about half an hour. Works fine. It's great. But if you want to include services such as Octavia, which is OpenStack's load balancer, you're going to have to set that up and you can fake file. So you're going to have to do that and then stack it again. That's going to take another half hour. So you add these services. You re-stack it. It should be fine. Now, something's not going to work. As me and Emilio experienced, it's like I don't want to exaggerate, but we spent six or seven years setting up DevStack. And so we spent a lot of time debugging. Then finally re-stacking again. And this process continued and continued and continued. But we got it. And DevStack is a great tool. All right. So we're going to do the sub-project within OpenShift and OpenStack that both of our projects were related to. And that is career. So first and foremost, the most important part of the slide is this guy. And that is the career platypus. He's unofficially named Carlton. Don't remember that. I didn't tell you that. So basically, what does career do? Career enables OpenShift to use the networking packages and services that are already in OpenStack in place of its own software defined network. And so, why do we want this? Basically, both OpenShift and OpenStack have their own software defined networking solutions. And the problem there is that both of them are going to inject their own checksums and their own headers into their networking packages. And that makes them very slow to decrypt. So by going around that, you can really speed up the networking for OpenShift on OpenStack. Now, career as a project can be configured to run either in VMs or in Kubernetes. And that allows it to be kind of isolated from your OpenStack environment. But it's not like isolated from the software itself, just physically isolated. And there's quite a bit of a tech stack that you're going to have to get used to to work with it. So my project in specific this summer had to do with scaling out the career controller. And so the career controller is a component of career that is responsible for listening for Kubernetes API events in OpenShift and then translating those into the appropriate OpenStack API calls in order to service different networking events that needed resources to be created, modified or destroyed. And so what that entails is the following. Basically what we have right now is a high availability mode and it's an active passive mode. So if anyone's familiar with what that means basically it means that one node is going to be active and then a series of nodes are going to be passive or idle and they're going to wait for that active node to fail and then one of them will take its place. And this is good. It's high availability, but the issue is that it doesn't scale out. And when you start having bigger data sets and bigger data centers it's going to start getting stressed under load and we don't want that. So we are working instead to develop an active active high availability solution which means that all of the components are able to handle jobs and pick up the slack of any servers that might have died. And overall the benefit here is that not only is it highly available but it also scales horizontally with it scales its performance when you scale horizontally which is, you know, obviously better. And in order to work with it, actually let me go back real quick. You see these guys are in little boxes that's because in order to run it in high availability mode you actually have to run it in Kubernetes and so you're going to have to run the career controller each in its own pod composed of two different containers and that's where you start running into fun stuff with Kubernetes. So first of all because both there are two containers running in your pod it makes debugging a little bit of an interesting situation and so first of all when you build a container I don't know if you guys have built containers before it's not the fastest process in the world so don't make the mistake I did at first you want to be pretty vigorous and test your code before you build because it will slow down your development cycle a ton. So first of all read your code and second of all take advantage of unit tests the third thing that I want to mention which is kind of maybe not for everybody because I know some people don't use Docker but a lot of people don't always realize that you can actually use the Docker tools as well as Kubernetes tools but if you're having a hard time with Kubernetes remember that you can use Docker's tools and they have a pretty rich toolset that can help you out a lot but anyway back to the unit test slide because that's pretty important if you work an open stack then you're going to have to work with docs and I'm sure at least everyone's going to test but just to go over them at a high level unit tests are really good for isolating different parts of your code and just making sure that they work the way that you expect them to and again what's nice about the unit tests is that you don't need to build your containers or your final product even to use them to test your code and so if you take advantage of them and you build unit tests to test your code along the way it's going to make your development process a lot faster there is however one part of talks that I think about unit tests in open stack that I think a lot of people won't be familiar with and that's Pep 8 so Pep 8 is this fun little test that tests your style basically tells you whether or not you suck at programming and News Flash the first time you use it you're going to suck at programming and I'm not talking about like this I'm talking like like that many errors so what is Pep 8 what does it do basically it enforces a really strict programming style and it's kind of used to just kind of keep everyone in check and make sure that all of your code within a project is uniform to some degree and it tries to make sure that it's all readable and at the end of the day it's going to make your code a lot easier to understand whether you like it or not and you know it's kind of a necessary evil now how many of you are familiar with github alright how many of you are familiar with Garrett alright right so uh same yeah so you're also going to have to learn Garrett if you want to work on open shift on open stack and the you know the fun part about this is Garrett you would think oh it's a git tool you know how different can it be actually it's like really different um so in Garrett first of all you don't follow the standard open source workflow like you do in github you don't fork it you don't you don't make your own branch you don't take out pull requests you actually just clone the main project and then you use something else called git review and then you push your code up to Garrett um and this could be a little tricky to learn for the first time uh and there's a lot of little things that go on underneath the hood that you don't really realize um once another thing you might not notice or you might not know at first is that um when you submit your code to check it into Garrett and you don't want a code review you actually have to go in and give yourself a bad code review and then basically give yourself a minus one to tell everybody don't review this I didn't know that I got code reviewed it didn't go well um but uh the upside of Garrett is actually it's kind of neat um to it's a really neat way to manage uh a lot of projects that are taking or a lot of people that are working on one project at a time and I think in comparison to git personally I like it better as a way to um see the various stages in your code and to see either the reviews that people left what changed and ultimately just to see what projects are going on in general and we'll just take a look here so this is what a standard um project uh commit would look like in Garrett and you can see here your history and these would be the changes that you've made along the way and each of those are expandable which is pretty nice um but when you look at the project view like this right you can see all of the projects that are being worked on under one large project and all of their progress which is I think a lot more organized than git either way though I can't speak for the open stack team I don't know why they chose to use Garrett over github but um whether you like it or not you're gonna have to use it so deal so anyway the grand takeaways here are um don't be afraid to ask for help um and if you do ask for help also remember ask for documentation every now and then um always try to take advantage of all the tools that you have at your disposal and you know ultimately you're gonna bump into some stuff you might not like and definitely won't be used to but there's probably a reason that it's there and it's usually there to help you so you know get used to it so now I want to talk to you guys about making technologies meet in the middle so being a member of the open stack team the obvious open shifts and open stack we had to make them meet in the middle but there's also services within them that we had to adjust to meet them so this summer my project was called watch endpoints as a service or the acronym WES for short um basically what it is is a service to speed up career by listening for networking events so I'm gonna talk about what that means in a sec so why do we want this tool so this is a diagram of the interaction between a bunch of services you only really need to look at that part right there between the career controller and the neutron which is open stack networking tool and basically you can see that the career controller creates a load balancer and then will continuously pull neutron using the open stack API to show the load balancer until it becomes active so the way I've been describing this through an analogy is imagine you're a chef in the kitchen and you want to take you're gonna have to preheat the oven the current system in place the chef is gonna be going back to that oven every 3 seconds until the oven has been preheated what my tool does is just have the oven alert the chef once ready so you can just see by that it's a lot less work and it's a lot more efficient so how do I do this basically there's this messaging queue between neutron and octavia which is the load balancer service and rabbitmq is getting a queue of all the messages as it is a messaging queue so when the career controller tries to create a load balancer eventually when it becomes active octavia will send a message to neutron saying that it hasn't become active my tool will sit and listen to the messaging queue and wait for that event once it sees that event happen it will tell the career controller so this makes too much sense this is the obvious solution so why isn't this currently in place basically the career guys ask the neutron guys to add this BAPI they said thou shalt not take events that came not from rabbitmq for that would be duplication and we're running shorter on maintenance so no so what's the end game if they already rejected this idea why am I doing this basically we're testing it with career communities and if it behaves as we expected to being faster and more efficient they're going to propose it to open stack as a career self project and if it goes well it will be integrated so this brings me to the challenge of making two technologies meet in the middle my listener would not hear any octavia load balancer events you would hear the four of events find from neutron but it wouldn't hear anything from octavia so naturally because it was working with with neutron and octavia I figured the problem must have been within my listener so my train of thoughts first of all I assume that neutron and octavia events behave the same way because they're both open stack services so why would they not secondly when I would use an open stack load balancer command it would generate events but they appeared to look like port events like the event type in the jsock return would be port dot something so at the time I assumed that these were octavia events because they were generated by a load balancer problem so based on these assumptions there was no way to distinguish these octavia events from the actual port events and so this was an issue because if this tool can't sort the events in that manner it kind of defeats the purpose of it because then you're going to be sending back hundreds upon hundreds of messages back to your career of control so the explanation of all this we figured out after talking to the load balancer guys that octavia doesn't actually emit events like neutron does so why is this and why do we assume this basically on the career side of things we just assumed that most open stack services emit events why wouldn't octavia on the octavia team they thought why make it emit events ironic which is open stack bare metal phishing tools didn't do it and it was never a requirement so it was never enabled which makes sense so both sides had reasonable expectations there was just this miscommunication so meeting in the middle how did we fix this issue temporarily over here we have career expectations capabilities in the middle there's load balancer as a service version 2 so if this satisfies, if this behaves how we want it to why is this the final solution why aren't we using this well for two reasons the current open stack deployment uses octavia and also lbos v2 is deprecated so right now we're going to talk to the octavia team to get them to reconfigure octavia to emit events but for now we're using this for test purposes and this isn't the first time ship stack teams have to meet in the middle in the past openshift requires that nodes, a node being an open stack server can talk to each other through their host names as opposed to VIP addresses whereas open stack doesn't need at least support this so the solution they found is called neutron DNS and it's actually a tool that they've had built into open stack for a while now nobody really knew about it though so just digging deeper help them find a solution to this so less of a gain sometimes you have to dig deep to find a common ground so throughout this summer we dealt with a lot of challenges but we really learned the teamwork and collaboration between teams is crucial there's so many different teams so many different resources out there especially via IRC that it really gives you to make use of them to take advantage of your resources and renad obviously is all about open source so open source means being part of the community so you really want to take advantage of the whole community because you have that support thank you you can keep the questions oh okay oh here do you want to take pass this around um what does he does he's asking about Garrett wait what did you ask about Garrett hang on microphone you can use this one one okay when you say Garrett it sounds like it's like a replacement of github but you I think when I use gyrus you need to clone the repo and then I can gyrus is like an extra tool for github that you can use on your computer so I'm just confused we'd be able to just talk more because I was confused about gyrus as well I think Garrett actually works with github you can host your code up on github and I think Garrett is really more for managing different commits and code review so instead of using like the traditional workflow like I think for most open source projects in general what people normally do is you'll fork github repo and then you'll work within your own fork you'll make a branch and then you'll take a pull request out against the upstream but when you use Garrett you don't do that basically what happens is you just clone down the I feel like I'm too close to this basically what happens is you just clone the upstream and you can make whatever changes you want and when you submit it you use a tool called git review and git review is going to I don't know all the details but essentially it tags your work with something called a change ID and then all of the changes that you make under that change ID get submitted into one general repository and then eventually if it's reviewed and accepted then those changes can be pushed back up to the master to the upstream github yeah definitely we use Garrett a lot in Red Hat to explain it to you in the simplest of terms when you use github and say you submit so you commit something and there are changes requested you send a v2 with a new commit ID so in your history you have a commit ID then a v2 commit ID or v3 commit ID when you use Garrett with git it allows you to edit a single commit so you can do because it uses something called change ID so say you commit something get it tags it with a change ID say your team request changes made a mistake you don't change the commit ID you keep the same commit ID you keep the same change ID and you can make changes to the same commit so when you look back you have a cleaner history and in one commit you can see all the comments v1, v2, v3 just for the same commit so get it allowed you to do that in github it's like just more of a keep sending commit sort of a thing it's granular but it's not the same it gives you more granular control yeah it's very true although it's it's fair to say that github is starting to develop more Garrett like workflow tools because they're recognizing that the workflow component of it is actually really important I had a question I apologize I missed the first few minutes did you all did you talk about what is the well two things the rationale for the work in the first place like what problem are you trying to solve if you talked about that then ignore me because I know I just want to make sure that everybody else heard it the other thing I'm curious about is what you see coming in the next few months for the shift on stack work in the next you know next version or whatever that is so man do you all feel about for the future let's enter the oh I don't want you to do it oh okay well in terms of the first part of the question we did go over a little bit just the concept of what the shift on stack team is trying to accomplish and how career is part of that goal that being said if you want to add to that feel free to if anyone has any questions about it please ask in terms of future though for my project at least we're going to containerize it for triple open stack on open stack and I'm not sure what the next steps after that will be yeah I can I'm pretty sure my project is not ready to get pushed on it was a lot more research focused that being said I think some of the changes that were done are going to be kept and I think some of the changes serve as an example of what we should not be doing in the future which is fine that's part of the process and yeah so I think yeah also I do know for a fact that in terms of just the active active I know that the open shift on open stack team has decided to kind of put that aside by now for right now it's going to be coming I don't know if it's going to be in the next release but it's on the menu just it's not a high priority since the money was worth saying that the open stack project there's a lot of directions that that the team is going in on one side is just supporting new features like making sure that we support things like staff and stuff like that another part of that is like standardizing the process so that other people can just play open shift on the stack easily and without having to do a lot of customizations the third part which I think is what you guys kind of concentrate on is optimization which is making sure that bits of the code are optimized in this case networking so that's what you guys were concentrating on but we have we're going in multiple directions with open shift on the stack if anyone is curious talk to me outdoors and also just to give you guys an idea of like what open shift on open stack like why we are interested in developing this we think of it as it's kind of our hybrid cloud offering because right now open shift can run on almost any cloud provider including open stack and we want customers who are thinking of building clouds on prem or in the cloud to be thinking about using open stack as their under cloud solution so yeah another question I know that this container module for open stack I forgot the name is it something where it's like it's like a container module for open stack but I just can't remember the name but what's the difference if you know what I'm talking about between using open shift and the one check that one so this is okay this is very ancient history there used to be a project called um project solemn S-O-L-U-M which was about essentially making using open stack as a container scheduling system that's long since been obsolete by Kubernetes um there is another sort of interesting effort in open stack land around called Cata containers Cata containers are actually a fully free and open source implementation of Intel's clear containers which are really VMs that you can spin up very, very quickly. They're super minified VMs that boot almost as fast as a container and there is some movement within open stack to make Cata containers sort of a first class citizen but you just at this point there's no competing with Kubernetes momentum in the container space. We don't think anyway that answers your question Any other questions? Alright, well thanks guys um so ladies and gentlemen next up we have Xiaofeng Wang senior QE engineer at Red Hat and he's going to talk about performance test methodology so welcoming him on stage Good afternoon everyone I'm very excited to be here at DevCon this year. My name is Xiaofeng Wang. I work in the virtualization QE team of Red Hat Today for my presentation I'm going to introduce you a new network performance test methodology I was running performance test earlier this year with widely used performance test methodology My job was to do some network performance comparison among three different hypervisors like KVM ESX from WMWARE and HyperRE from Microsoft so during my testing some issues blocked my test so I had to fix them to finish my performance test when all the issues were resolved I felt that the solution to the to the problem can be summarized into a new performance test methodology I will give you a detailed introduction on these issues and the corresponding solutions but first sorry that's not work I will show you how I run performance test before that means with old performance test methodology the first step is always run the test tools in my case I choose the hyper so you can use the hyper for some package generators and mouse so I run hyper dash s on the server side and hyper dash c on the client side with some hyper options when the test is finished the hyper will show you your performance result like this one so the final performance result will be in the last two lines so in this case the performance will be 21 gigabit the third step is run more tests and get an average value as final result so I guess we always do that the last step generate the performance test report based on the average result so this is the how I run the performance test before with the old performance methodology what's the problem here maybe you cannot find any problems I will show you the problem I just found the first one there's no step by step traffic just send as much traffic as possible so what's the meaning what's the step traffic like this chart it's a step step step step for example if I want to test 10 gigabit NSE network adapters I will use the traffic should be start from 1 gigabit to the 10 gigabit with 10 steps one step will increase 1 gigabit so it's called step by step traffic so what's the problem here without the step by step traffic because the IProof or netproof just send as much as much traffic as possible to against your DOT means the device and the test in my case it's a network adapters so sometimes the system will be broken instantly by exiting the max traffic at the beginning of the test so at this time I cannot get the performance result of the system so I have to do some setup or some things to make the test running sometimes it's not running because the system broken this is the first problem the second problem is about max value I find nevertheless the IProof netproof or something else they just give you the max value that means the max performance in my case the network adapter can support but it is just a max value so the network adapter in my case cannot work well at the max performance for a long time just the max not the working value have you heard about the Intel has a technology called Turbo Boost have you heard that it's a little bit like this concept like I just said the Turbo Boost you can use the Turbo Boost not for a long time just at some very special scenarios for example one minute or just a short time that's the best max performance of that CPU but at the normal scenarios you just use a lower frequency of the CPU the concept is the same as I just said this so all the performance test tools just provide you a max value so the max value is able to reach the performance but the stability cannot get granted with the max value that's the second issue the third issue the test tools like Apple 3 use mean mean to calculate the final result why not medium there might be a result different between the different algorithm like mean or medium so what's the best one I will give you an example and more explanation on this on the solution section so the mean and the medium is different algorithm to get the average value so you choose different algorithm you can get a different result this is the third one the fourth one is there's no way to find a spike from the current test result log have you heard the spike in the performance that means you can find in some scenarios performance drop to a deep or increase very high so if you just use the iproof you cannot find this you just got a final result the last issue the report is sorry the report is not user friendly as I expected just to give you a number no chart, no tables just a number so that's all of the problems so what's my solution for the first issues we need a step load that means a step by step load to avoid the max or let the iproof or netproof send traffic as much as possible we give it a limitation tell the iproof or netproof how many traffic you should send this time how many traffic you should send next time so in my solution in this case I will run the iproof multiple times each time I will give it a limitation so the iproof will send that traffic as I expected that's the step by step load so I use the step by step load to increase the load until the network or network adapter reaches a point where the performance is significantly significantly as the load increase the network will be able to keep up until it runs out of the resource that means when if your network adapters cannot support that traffic the chart will be drop or keep on that on a line not increase that make sense yeah how to do that is very simple because iproof has a dash v options which is able to set the bandwidth it just needs you run the iproof multi times and run it will be very easy because you can write some write a loop to set up a read about different bandwidth and run the iproof multi times so this is the solution of my first problem make sense? yeah for the second issues I use some mathematics mathematics to solve that issues I use correlation and r squared that's true mathematics algorithm so I will explain this chart later I will give you a brief introduction about what is correlation and r squared for the correlation correlation is a statistical management relationship between two variables possible correlations range from one to minus one and a zero correlation indicates that there's no relationship between variables and if the correlation of minus one negative one indicates a perfect negative correlation that means this one for what indicates there's a perfect positive correlation meaning both variables move in the same direction I have an example to explain this I have these tables about two variables temperature and ice cream shelves you can find when the temperature goes up the ice cream cells goes up as well so these two variables have relationship they have relationship but what's relationship you have to do more things I will do that later we can easily see the warmer weather and higher cells go together the correlation in this case is about 95% therefore the relationship is good but not perfect high positive correlation yes because I did some modification on it so what's the R squared the R squared is a statistical measure of how closely a train line matches the date so if in this example I got the point in the chart I can get a train line across the date so the R squared will tell you how close the date with the train line with the R squared I can find how the train line can represent this date train so if it's very closely the R squared will be 100% that means the higher R squared the better mode fits your date I will go back to this chart I will explain this this chart for for the X axis represent the load like I just this this is the load for the Y axis is the PPI means the packet that means how many packets the network adapter can process, can handle in one second so when the load increases the PPI should be increased as well right but when the network adapter cannot handle the light amount of traffic they will go to the Y line so the load and the PPI they will be two variables they have relationships so I can calculate the correlation value between these two variables so the blue dot line indicates the correlations so you can find the correlation is always 100% but at this at this time they will drop because they cannot increase the PPI cannot increase so I believe the correlation a number 90% is my accepted number so in this case I can find here there is a cross two lines cross here so I can find the value in the X for the load right and based on this the blue line I have got a liner the liner will be Y equals AX cross B so if I find an X value I can get the Y value right in my case the max value should be which one maybe this one the max value but in my case I don't want that value I just want a value that can work for a long time so the value will be here or there should be this one idea it's a little bit difficult for me to explain it very clearly but let me explain it again we have two variables load and PPS when the load increase when the load increase the PPS should be increased as well so this line the blue line indicate that these two variables relationship they always increase but at this time they always increase so the correlation value will drop here but I set an acceptable value for the correlations I set it to 90% so there will be a cross point two lines cross point B here this is the correlation the dog line indicate the correlation so I can get the cross line with 90% here so I can get the X one and I can get a liner based on the blue line Y equals AX plus B like this one you can get it by some in my case I use the google shape so you can get it it will be generated automatically so now you have the X value so you can get the Y value that means you get what you want it is not the max value that means the maybe lower than the max value but it can work for a long time just in my case further the third issue is about the mean and the median the mean is the average you have used where you add up all the numbers and then divided by the number of numbers in this example the mean will be 9.73 it is called average in the google shape google shape has a function called average to calculate the mean value and the google shape has another function called median it will calculate the median value median means the middle value of the list of numbers is 9 the difference between the mean and the median is the mean value is calculated this value you cannot find the value from the list that means in the performance result the mean value will not in your performance performance number generated by the iprof it is just calculated but the median is this value is in the array of your performance numbers so that's the difference which one is the best it totally depends on your requirement so your test result and your requirement I prefer the median in my case because median result always exists but mean result does not in this case maybe the difference is not too much but in some case it will be very large the fourth one is about how I can find the spike in my performance test I use another mathematics it's called sand deviations that means the sand deviation is a measure of the distribution of a set of data from it mean perfect use the tool to find the spike you know the deviation is how you can find a mean value the actual value how far the real value from the mean value that means if there is a spike if there is a spike the sand deviation result will be higher or very high or very low so you can find it because you can from your result array you can get the max sand deviation value and the median deviation value if the three value is totally different that means there must be a spike in your test it makes sense for the last one it's about the result demonstration for my perspective I like a test report with with a table it's a table report with some chart the table report should be clean and simple and easy to understand and the table report should include most important performance indicator it has to be some chart intended to supplement the table report I will give you examples this is an example I run the performance test on the ESX this is the report table I have the RXPPS the TXPPS throughput average latency max latency this is not the final one I will add more into this we can find the standard deviation with this value if there is a very high or very low value that means there must be a spike in my result it should include some CPU memory usage in this report so this is a table report for the chart you can find there is some chart for you to reference for example this is the chart do you familiar with this step by step yes this is the result from iprv I press the iprv result and put it into the Google Sheet will give you a line to indicate your performance so you can find give you overview of your performance result a lot of spikes yes a lot of spikes so yes could you please repeat the statement you made just a minute ago about the standard derivation if the number was too low or too high you didn't have the table chart oh yes you were mentioning you didn't use it or you threw it out oh the standard deviation this standard deviation indicated the slide right but you can now find a very low or very high value here or did you not include them here yes but this but this table is not the date behind this table is not the same as the chart so they are different I'm sorry because so if I draw the picture of the date behind this table it's very small no slide oh go back so this is the last problem and my my resolution so that's all about my presentations if you are very interested in my topic you can find it from the conference page but this page is just for the Red Hat internal so maybe you cannot get it but if you are interested you can find me and give you your email address I will make them to the pdf and send you right that's all ok questions is the expert tool is that available so is the expert tool is that available to people outside of Red Hat or is it only Red Hat employees available at this time or expert or is that sorry the expert is the name of my project I just call this message but just give it a name called expert it's just a name got you no other meanings sorry for confusing ok questions when you're doing your testing have you looked at percentiles instead of doing what you know average and mean median to be honest you might have I didn't find too much difference between the mean and the median but in some my colleague gave me their test result I found that but I really don't know what the scenarios what the scenarios they use so but I think the the median one is is more reasonable from my perspective but it's not the it's not a regulation here it just depends on your requirement or your test purpose but in terms of knowing what the throughput of what you're doing what you want to strive for is say 80% throughput or 80% of the time you're going to get through or 90% of the time this is where percentiles will come in handy versus just doing an average or a median hmm oh so from from all of the test test value the interval value is from the median and all of the numbers comes from the median I use median all through my test not part of mean and part of median median thank you just an announcement we'll be having a party in the evening on the lounge if you haven't collected your tickets you can collect it at the registration desk and we'll also have a keynote speech tomorrow so if you're if you check the schedule you should be there at Metcalf Large and thank you for being here I I I I I'm I'm I I I and to and testing framework for react so we simply use the Java with Selenium Web Driver to test react but if you're not thinking about end then you left right left right let's see if this actually works hey it does awesome all right cool wow more people showed up than I expected I wasn't sure anyone would want to talk and come to a talk that just focuses on what we do with the community so thank you for being here I appreciate it we do a lot and we're terrible about telling people about it so I'm gonna get up here and tell people about it all right I am showing that it is it is 430 so we'll get started hello everybody my name is Thomas Cameron I'm a senior principal cloud engineer in the engineering business unit at Red Hat my contact information is up here I am Thomas at Red Hat.com I've been here for long enough that yes I have the Thomas at Red Hat.com email address and you can follow me on Twitter at Thomas D Cameron what I want to talk about today like I said is we we don't Red Hat does not like to you know thump our chest and talk about how awesome we are and what we do and and so I think we don't talk enough about and explain enough about what we do with the various open source communities that we work with so I want to spend a little bit of time today doing just that I'll talk a little bit about where we came from where Red Hat came from where we are today I want to actually also interject something about what we do from a business perspective because a lot of people think of us as just another software company and I want to explain a little bit about why that's not the case seg into what our mission is because our mission actually calls out community our mission statement actually calls out community it is ingrained in our DNA I'll talk a little bit about some of the projects we work with I'll talk about some of the acquisitions and how that has affected communities and then I'll talk about some legal issues that Red Hat has been involved in on behalf of communities some of the community programs that we help sponsor the developer programs and why we do what we do so first off where we came from a lot of folks it's really funny I still read articles that talk about Red Hat as you know the scrappy newcomer to the IT industry and I'm like I I'm not kidding we got a new what was it we got a best you know young company or something like that like just a few years ago and I was like we started in 93 like we're old but yeah we were started in 93 by a hacker by the name of Mark Ewing and a businessman by the name of Bob young Bob sold he had a catalog business he sold computers and media with free software on it out of his bookstore and Mark built a Linux distribution called Red Hat Linux Bob sold so many copies of it that he approached Mark and said I want to buy your organization I want to buy your company and they adopted the name Red Hat software and went forward as a going concern and in 99 we did well enough that we were one of the most successful I think at that point were the seventh largest IPO to date so went from nothing to huge IPO in just a few years and 99 we really changed the way that we did business because previous to 99 we were just a distribution we we packaged up software from you know various projects upstream created the distribution sold it and we're successful at it but in 99 that really changed when we acquired Cygnus software Cygnus software it was a hardcore engineering company that did a lot of kernel stuff and did a lot of toolchain stuff that allowed open-source software to be ported to different hardware architectures that changed the way that Red Hat worked at that point we went from just being a distribution to actually doing a lot of hardcore engineering and contributing a lot of code upstream we had contributed before but with the Cygnus acquisition that really led to us being a powerhouse in the open-source community because we were able to actually do a lot of really good engineering work and contribute code upstream so from very very early in our days upstream was really important to us and I'll explain why in a little while in 2002 we changed from a consumer based model where we were selling box sets at you know Best Buy and CompUSA and praying that people would buy our software and augmenting our business or augmenting our revenues with like t-shirt and ball cap sales to an enterprise model with Red Hat Enterprise Linux and that actually was a pretty controversial move there were a lot of folks in the community who had gotten free distributions from us Red Hat Linux and when we turned around and said hey we're not gonna do free distributions anymore we're gonna sell subscriptions to software there was a lot of bad a lot of animus from some some parts of the community and so from 2002 through today there are still some people who are you know a little chapped at Red Hack is because we don't give our flagship distribution away for free we we sell subscriptions to it this was a massive change this was a huge change and companies like Dell and HP and IBM and even Oracle threw in their support behind us we were arguably the first to market an enterprise open source and I think it's pretty fair to say that we have been the most successful open source company to date today we're approaching three billion dollars in revenues we've got over 80 offices in something like 40 countries around the globe and we have about 12,000 people that would work at Red Hat so there are a ton of folks that are doing work around the software and the technology that we sell subscriptions to and also committing code upstream now as I said earlier a lot of people think oh Red Hat is just another software company but let's be real clear we don't sell software licenses the license under which you use the software that we that we support and sell subscriptions to is an open source license it's the GPL one of the versions of GPL or LGPL or patchy software license or MIT license or whatever there are a ton of licenses that are included in our technologies but we don't sell software licenses when you get a subscription to one of the Red Hat technologies that we sell subscriptions to it's not just about the bits the bits are certainly part of it because we deliver the bits through our content delivery network but it's about our technical leadership it's about accessing the bits from the from our network it's about knowledge base access and all the solutions we have there a big one is it's about a 10-year life cycle for open source technologies now if you work with open source technologies you know that in a lot of cases you know a six month old package is completely out of date but in an enterprise environment you don't want to be upgrading your infrastructure every six months so you know having that long life cycle is important product documentation how to tips and tricks things like that a big one in enterprise space is security we have a huge security team that's global that reacts to post-release security incidents and then also hardens our software to make sure at least to try to make sure that we don't have any security issues when it goes out of the gate it allows access the subscription allows access to the labs on our portal so you can do things like pre configure various services and pull down config files and it's access to the solution engine support solution engine is when you type in a problem it'll actually use AI to go and search through our knowledge base to see if there's something that matches it support up to an including 24-hour follow the Sun a big one software compatibility you know we test our software with our partners like you know SAP and Oracle and IBM and things like that so when you install that software it's a certified solution it's access to education and certification you know you've heard of folks who are Red Hat certified engineers for instance legal protection which I'll talk more about in a little while this is important to a lot of our enterprise customers and that partner ecosystem that partner partner ecosystem of ISVs and IH sorry independent software vendors and independent hardware vendors and vars and consulting groups and things like that and then certification on common and popular hardware platforms from you know Dell and HP and Lenovo and so on so it's not that we're selling bits the bits are only a tiny part of what the subscription brings so when you hear people talking about oh software you know Red Hat's just another software company company it's really not just it's not that if you look at Red Hat's mission statement online this is our published mission statement it says our mission is to be the catalyst in communities of customers contributors and partners creating better technology the open source way this is in our DNA you will hear me say that over and over and over again we are totally dedicated to working in communities whether it's communities of customers or upstream communities and I'll give some examples of that so what projects does Red Hat work with I think before I go any further what is the first thing you think of when you think of Red Hat Red Hat Linux right I mean for the most part people think of Red Hat as a Linux company which is fantastic because that's certainly where we came from that was where we started was in the Linux space but there's a whole lot of other stuff that goes on at Red Hat Red Hat Enterprise Linux is one product out of all of these technologies that Red Hat sells subscriptions to so there's Red Hat virtualization there's Red Hat open stack Red Hat storage which is Gluster and stuff Red Hat satellite for systems management Red Hat J Boss middleware which is like I'm going to show you some of the things that we do in middleware and it's like multiple pages of technologies Red Hat mobile application development management a lot of people don't think of that with Red Hat Red Hat open shift for container management Red Hat core OS core OS is now a Red Hat property manage IQ for cloud management Ansible is now a Red Hat property we acquired them not too long ago and for every one of these products that we sell subscriptions to there's an upstream project that we sponsor and maintain and contribute to so you know RDO and overt.org in the Fedora project Centos project Centos is also Red Hat technology now you know the upstream core OS open shift origin etc etc etc now that's a whole lot of communities right there but the argument can be made well you know that's just upstream for your products right you know that's not real community I disagree because we do everything in those upstream communities nothing makes it into Red Hat Enterprise Linux or almost nothing makes it into Red Hat Enterprise Linux I'll say nothing because there's always going to be some weird corner case but but generally nothing makes it into upstream into Red Hat Enterprise Linux that didn't come through the Fedora project we put everything out in the community freely distributable downloadable open source technologies the same is true for over.org the same is true for open shift origin the same is true for manage IQ Gluster.org stuff etc etc etc those are all communities but then there are I don't know about a million other projects out there that we are aware of and try to keep an eye on that we work with like at the Apache and OpenStack and things like that so there's a ton of work that we do around communities upstream but it goes a lot deeper than just what you see on the screen these are projects that we are involved in that are not productized they may be part of a product but these are these are things that we are involved with from a foundation perspective we're with the cloud native we're part of the cloud native computing foundation the Libre office foundation Linux foundation open NFV Python blah blah blah I mean you can see that these are all foundations free software foundations that we are members of to some level or another so we do a ton of work around standards that have that are not you know necessarily product related and let's be real clear the truth of the matter is by working with these upstream projects and making sure that there are standards which are repeatable across different communities and different technologies the truth of the matter is this makes it easier not harder to migrate off of our commercial offerings we make sure that things are standardized across communities so that they work no matter what now we don't want you to obviously but but the reality is by being members of these foundations and standardizing things we may get a lot easier for you to do other stuff from an operating system perspective I know this is kind of an I chart for those of you in the back I apologize but you know the Centos project Fedora Cygnus and upstream GNU and Linux projects and all of these utilities all of these technologies which are yes certainly part of red had enterprise Linux in a lot of cases but also part of Fedora and a lot of these technologies have been adopted by you know canonical and by Suza and by Slackware and and so on and that's part of the deal that's part of the deal we know that a strong vibrant upstream community you know rising tide raises all ships and so it's a good thing that the stuff that we're involved with like upstream Linux and SC Linux and system D and so on it's a good thing that there is a large and vibrant set of communities around these technologies if you look at containerization we're part of you know cockpit and cryo and Docker and flannel and Kubernetes and atomic and rocket and so on because again having a strong upstream community makes all technologies better including ours certainly there is a there is a component of we want to have the best technology in the market but we participate in those upstream communities by contributing code by contributing governance by contributing bodies and so on so and then from a desktop perspective again there's a ton of work that we do around GNOME there's a ton of work we do around Libre office and a whole bunch of other upstream technologies for the desktop like xorg and things like that so a lot of work that we do in those upstream communities to make the desktop better and everybody wins all the distributions use those technologies from a middleware and identity management standpoint got an interesting story about our identity management platform that I'll tell you in just a few minutes but if you look at what we do upstream with the Apache web server for instance what we do with the various jboss.org projects around application services and data virtualization and memory in memory caching of big applications and things like that there are a ton of projects and communities in upstream that we contribute to from an operations perspective you know again upstream over to an open shift and open stack and and things like this for cloud technologies all of these are open source projects that we work in the upstream with and anybody can have access to and from a storage perspective we acquired Gluster for NAS style storage software to find storage using the Linux operating system and then the storage layer on top of that and then we also acquired ink tank so Ceph is part of the Red Hat family as well for block storage so a ton of upstream work being done there and then some really cool stuff that we're doing like with the condor project for things like high performance computing or high throughput computing for computational clusters and things like that we contribute a bunch to upstream technologies like NFS and you know GFS or should say GFS too since that's the current version but there are a ton of projects that we work on there that again get adopted by a bunch of different distributions and then from a developer tool standpoint real quick how many folks are how many folks think of themselves as more developers than operations folks okay how many folks are more operations than developers okay good we got a really good mix okay so how many folks have used Eclipse yeah okay we do a ton of upstream development in the Eclipse with the Eclipse Foundation and and a bunch of other technologies on here there are technologies there are developer tools on here for you know sort of typical middleware or application development all the way down to kernel level and C programming so a ton of stuff there as well so did y'all know that Red Hat was contributor would contributed to all of those upstream projects couple of you did well you work here I don't you better know so Red Hat has acquired a bunch of companies just I've been I've been at Red Hat coming up on 13 years and just in my tenure here we've acquired a bunch of companies and it's been really fascinating to see what happens when we do acquire a company what we do around making that technology open source and building communities around it since we were founded we've acquired over 30 companies some of them have been very successful and frankly some of them have not we made acquisitions that turned out didn't work out real well and you know that that's just kind of the nature of doing business but in every case where we have acquired a company or a technology if it's been successful we have either created or at least maintained and invested in existing communities around those technologies and interestingly a lot of the acquisitions that we made were actually closed source so we had to acquire something that was closed source in a lot of cases we had to go through a bunch of legal you know rigmarole because of license technology a lot of technical rigmarole where we had to replace that license technology with open source technology and we have spent a ton of time effort energy and money in making sure that those technologies are accessible by upstream communities and then building communities around those technologies so you know if you look back at some of our early acquisitions Sustina was a closed source company that was doing HA clustering and a shared clustered file system for Linux we bought them we turned around we made it open source and and it's available out there on the internet for anybody to download we acquired a lot of the Netscape assets from AOL when you all remember when AOL bought Netscape our identity management platform is actually based on the old Netscape identity manager code now it's been so long that there's no code left anymore but that that was absolutely closed source we bought that from from AOL and had to jump through a lot of hoops to make that open source uh meta matrix for data virtualization kumranet okay no red headers are allowed to answer this does anyone know what kumranet is responsible for yes kvm kvm spice and spice that's right so kumranet was a little company that we acquired out of israel that uh that was the the the founders of and the inventors of kvm which is now used in open stack and you know red hat virtualization it's a it's a super popular hypervisor technology on the Linux kernel um that was closed source when we bought it the the kernel module the kvm kernel module is open source but like the management interfaces and uh red hat uh what they called at the time uh what we called at the time red hat enterprise virtualization hell when we first bought it it was based on windows that went over like a ton of ricks um but uh but yeah so we bought that turned around rewrote it made it so that it could be deployed in uh with open source technologies and now red hat virtualization for instance is awesome technology uh managed iq was uh cloud forms our our multi cloud management platform ansible when we acquired ansible tower was closed source it's not anymore so we've done a lot to make sure that uh that that we have made those technologies accessible by anybody who wants so just just out of curiosity how much do you guys think red hat has spent in acquiring technologies and then turning around and either maintaining them as open source or making them open source who said that you're close now it's it's we spent have you tallied all the two point the ones that we have records on oh okay but a lot of them were private transactions because they weren't publicly held companies uh and so there are some of them i mean there are a lot of them that we actually don't have dollar values on or that at least they're not publicly released so on the low end about two point four billion dollars on the low end for technologies that we bought either made open source or maintained as open source and made sure that the communities had access to them red hat has also been involved in a whole bunch of legal fights on behalf of the open source communities that we participate with um we've had red hat executives and evangelists who have testified uh at the US congress they've testified in front of the European Union Parliament uh they we have written amicus briefs for the United States Supreme Court uh and various lower courts around the world uh to fight things like patents uh software patents to fight things like patent trolls uh to fight things like anti competitive practices in the industry you know we spent a lot of again time effort energy and money making sure that we were in the front of those fights to protect the smaller communities that didn't have representation we've kicked a snot out of patent trolls and that's been awesome um there's been a ton of articles there have been a ton of things that we have done where we have either beaten them in court or we've even purchased patents when it was at at risk we was just like you know what we'll buy it we'll you win this round and then we turn around and make it available for free to the open source communities so we uh we have done a ton of work around protecting intellectual property and trying to fight patent software patents um we also offer a patent promise that says that if you're using our technology and that technology is found to infringe on a patent we will indemnify you and we will also write code to replace whatever patent and covered software there is so we're very very anti software patents uh we found it we were one of the founding members of the open innovation network which is a collection of companies like it's us uh Google IBM NEC Phillips Sony SUSE and Toyota of all things Toyota um and anybody who's working on open source uh any Linux related software can join Open Innovation Network uh and we have a shared defensive patent pool so if uh if somebody tries to mess with us it's almost guaranteed that we will have a patent to fight them back with yes what is Microsoft going to join that's a damn good question I don't I can't presume to speak for them yes sir very small question open invention network no open innovation did I fat finger it you did oh you didn't see that look away right after the statement dang it we haven't you haven't both ways yeah it's it's innovation right next to each other really yes wow okay I'm a moron it's been a long week this whatever today all right so we're also involved in a lot of community programs from a supporting perspective um we have a group inside of Red Hat called open source and standards or OSAS um I don't even know how many people are on OSAS now it's going to be a very very large team uh they're technical folks they're administrative and organizational folks and community management folks even like marketing and and artistic folks in OSAS who help communities organize so if you're a member of a community and you're struggling with community management or if you're struggling with like how to form the community and how to get it vibrant and active we can help go to community.redhat.com and we will absolutely provide assistance uh we have done things you know everything from helping with governance issues and organizational issues and and community management all the way up to the simple stuff like yeah we'll host your website or we'll we'll provide email services for your community so we're very invested in making that happen if you go to community.redhat.com you can see there's a ton of information there and and I just did this screenshot today you know dev.com for us is on there we talk about activities in the community we've got an events calendar uh we've got a knowledge base there's a ton of stuff there that are resources for community managers uh and and if you need additional help you can always email us and we'll help you there so redhat also has some developer programs that are that are absolutely focused on uh thank you on uh community members so redhat software is available at developers.redhat.com and this was actually something that I and a bunch of other people but but I was probably one of the most obnoxious about it um and I don't mind being obnoxious internally it happens all the time um but we fought really hard to have a developer program where folks could download redhat software and not do the 30-day eval or not do the $99 developer subscription that we had for years and years and years there were a bunch of us internally who said we cannot do this if we want developers to be able to participate in the redhat community not just upstream we've got to make it available for free and so they did so the business units made it available for free if you go to developers.redhat.com and click on linux um got a cool video about linux and scroll down a little bit you can get a subscription to redhat enterprise linux the developer version which is the exact same version but we also enable all the add-ons for like ha clustering and uh clustered file system and uh gfs I mean a bunch of stuff we add on to it and it's zero dollars so if we get the box too no unfortunately we don't ship physical media anymore 1999 called they want their technology back I know I know right I would love to have that I saw that I was like oh no okay dang it but uh but and what's funny is every once in a while someone will email and say hey I've got this whole box set of redhat enterprise linux 4 and I'm like I'll take that just ship it up my wife's like please don't so but anyway so and over and above that if you go to the middleware side you can do the same thing with the jboss stack too so all of our all of our je middleware stuff um and and the entire developer suite you can get started with the middleware developers kit as well zero dollars hello so we're knocking on the door don't tell him we're in here okay thank you so why why does red hat do this we're a commercial company we're publicly traded I'm not gonna even pretend to say that it's not very important to us to be revenue positive right we got 12 000 people that we gotta feed and and shelter and you know make their families happy and stuff like that uh but when I came here in 2005 I came to red hat because I am an open source person um I am a red hat I'm an open source advocate you know I uh my background my background is kind of bizarre if you already know the answer to this please don't answer but I'm just curious does anybody want to guess what I did before I got into IT are you really it's locked I mean there was a life I know it's like let me in I know we can't it's locked anyway so so anyone want to guess what I did before I transfer before I changed into IT back in 1993 no and no car salesman oh come on damn shots fired shots fired no what's that professional water polo player I don't know where that came from no I was a police officer what I was a police officer when I was when I was 16 years old I volunteered through the police explorer program at my local police department when I was 18 I became a corrections officer when I was 21 and can get commissioned I became a police officer when I was 24 I was like I can't afford to do this but one of the things that drew me to open source because I started back in 93 worked on you know novel network to date myself and then on windows because that was the next big thing when I when I got into Linux when I got into open source and I realized that there was this incredible community where you could get on IRC or you could get on at the time it was a use net you know comp dot os dot Linux dot blah blah blah you know you get on these things and you could ask questions and people would answer and it's like that's the guy who wrote the code like holy cow and then I was like hey that guy's asking a question or that gal's asking a question and I know the answer I can help them and that was kind of my introduction to open source was I can participate in this community like I can help other people that's why I became a cop and I learned that I can do it in the community as well and I'm pretty typical of red hatters most of us who work here are doing this because the communities that we participate in are awesome and we get help and we can give help and it's it's a really cool sort of karma thing it's amazing when I came to red hat we had 1200 employees now we have over 12,000 employees but I have I have worried over the years like how do we keep the culture how do we keep that spirit alive but myself and several other folks are actually involved in new hire and new manager training programs where we teach the culture and we teach the history of red hat so that as folks come on board maybe they didn't come from the community maybe they don't understand but by the time they go through our new hire orientation by the time they get into the trenches with us they do understand how critically important it is that we participate in these communities. Most of us are very passionate about open source sure there are some who just you know it's a job right but for the most part we're really good about hiring folks who are passionate about open source we're passionate about helping folks we're passionate about enabling people to do things they never could have done if they had to rely on proprietary expensive closed source software you know when we did the one laptop per child program you know that was one of the things that I was most proud of up to that point and I keep getting more and more proud of the things we're involved in like we put technology into the hands of people who could never ever afford it if if it weren't for the free nature the free is in beer nature free software that's cool man I'm a tiny tiny tiny little part of it but I am part of it and that is incredible to me and everybody at Red Hat most everyone at Red Hat is the same way and over and above that we recognize that we are beholden to those communities that that we get technologies from that go into our products you know we are responsible to to make sure that those communities are vibrant you know we wouldn't be successful if it weren't for you being here today and participating so we're grateful for that we recognize that we owe a debt of gratitude to the communities we would not be successful if it weren't for what the upstream maintainers and developers have done and it's important for us to give because we have been given so much you know I got I got a wife and two kids at home and if it weren't for open source if it weren't for these communities I wouldn't be able to put food on the table or I'd have a job that I hated you know but instead I get to do this I get to come and talk to you I get to participate in this incredible thing and that's that's a pretty pervasive attitude in Red Hat and you know I love it I when I came to Red Hat I thought I was going to be there for two years I thought I'm going to go two years I'm going to get some street cred I'm going to be a consultant I'm going to that was 13 years ago you know I love this place and I love that we are geared to being of service to upstream communities and and I'm not unique in that at all so we are honored we are absolutely honored to be a part of your community and we will we will do our best to be good stewards and to contribute and we are thrilled to participate with you so thank you very much if there's any questions feel free to ask yes so I just want to say I feel like Red Hat needs to do a much better job explaining the technical aspects of upstream first how so I found a blog post from Dave Neary in 2015 on community at redhat.com blog upstream first turning open stack into an NFV platform where it contrasted upstream first with the vendor branch model okay at all the other vendors are following and it explains how it's harder to get all your patches accepted upstream and then backport them to your product your product basically being like a legacy branch and how other companies say oh it's usually just to develop all the features on our branch yeah and and how that provides temptation to go proprietary in the future and similarly like I contributed the X to go project and X to go product for like a decade relying no machines open core and uh for for a while it looks like and and then showing off the next version of no machine was all proprietary and now we're you have to develop the open core I feel like Red Hat needs to focus on the technical aspects of upstream first I will I will do so the next time I present this thank you any other questions so it's all well and good that you are such a heavy participant in these uh communities and it's very clearly felt in a lot of different places but sometimes it also feels like um when Red Hat is participating in communities uh it feels a little weird because like the reasonings or when the discussions are happening it's like we get half answers or things like that so I'm involved in Fedora and a number of other communities as a community person and I interact with Red Hat there's a lot um and a lot of times it winds up being where I feel like I'm getting half of the answer or a quarter of the answer and with the part I'm given I don't think it's good or smart or anything like that but like I feel like if I had the whole answer we'd have a better discussion and maybe have a better result at the end of it yeah yeah that's a fair criticism um I'm not justifying it because I don't know the specifics of of you know what you've experienced um I will say it is an unfortunate reality that you know like I said we are we are a commercial company there are things that we have to do from a product perspective you know that that are going to um dictate the way that we do some upstream work and there are going to be times where we've got things that are going on that we can't disclose for financial reasons or for you know disclosure legal disclosure I mean there's a ton of things like that maybe that's what happened I don't know because I don't know the particulars what you're what you're doing or what you're talking about I will say though you know if that happens call it out and I mean don't be a jerk about it and I don't I know you well enough know that you wouldn't but call it out you know call the person out privately and say I know I'm getting a half answer here like this doesn't make sense is there something going on internally that you can't talk about like is there you know what's going on um we will do our best and there is an internal policy I mean and it's not even a policy it's an ethos of transparency and disclosure where we can okay and you know I mean if you run across something like that again I don't have any juice really but tell you know you've got Thomas at redhead.com man send me an email I'll help out if I can yes sir hold on wait for the mic so that we get you on video you're being recorded for posterity you talked about red hat being good with upstreams and I'm wondering if this is in opposition to being good with downstreams as a person who is a free software advocate and a person who works independently on free software projects I find red hat projects to be the most difficult to cooperate with in terms of as an individual coming into a project okay and as a downstream so like if I'm a user of say gtk right interacting with gtk folks is actually kind of hard and it gets harder the more red hat they are so like the more employee they are it gets harder they get harder to talk to harder to contact and so on and I don't think this is necessarily a malicious thing or like a problem but it definitely does seem like my experience with the debon 2 community back in the golden days they focused a lot more heavily on downstreams maybe the to the to their detriment on their upstreams but I'm wondering if there is a balance to be struck between upstream focus and downstream for focus yeah actually Steven wants to answer that would you hand him the mic so as Thomas mentioned some of this red hat has grown very fast in a fairly short amount of time and there has been a certain amount of we have seats that need filling and sometimes they end up with people who are looking at red hat as just a job and less of less of the passion that some of us old timers tend to have and like Thomas said there with with osas and with several internal programs we're trying to express to individuals more about why the community is so valuable and yeah there are times and there are some projects that are very heavily red hat employee-based that need still need help getting it that that is what they should be doing so there yeah there there are times when we don't do a great job of this call it out when you see it if you need to talk to me escalagetredhat.com I usually try to take those people and have a meeting and you know explain the light to them so make it quick I really literally have to go to the airport like right this minute okay this can answer can be done by any of the red hatters in this room or perhaps not in this room I am an ISV VAR partner of red hat how do I work to make sure that my business is noticed by red hat in that we do not have a cloud product and we're not the buzzword thing but if it wasn't important then I wouldn't be in this business thank you yeah we do have to go