 Hi, everyone. This is the COA and why you should take it. If you're not familiar with the COA is, that is the Certified OpenStack Administrator Test, which is authorized by the OpenStack Foundation. My name is Amy Marish. I'm an OpenStack course author with the Linux Academy and we have a special guest with us today, Ann Bartuccio from the OpenStack Foundation. Hi, nice to meet you all. My name's Ann Bartuccio and I'm the COA program manager at the OpenStack Foundation. Before we get into the actual panel, I wanted to go over some of the numbers of the tests. So we have, as of May 10th, 2017, so these are yesterday's numbers, 855 people have taken the COA. Of that, 855, 495 people have passed the exam and the passing rate is approximately 60%. Does anyone have any questions about these numbers before we go into the panel itself? 76 out of 100. Anyone else? Alrighty. So I'd like to introduce the members of our panel. We have Matt McEwen from AT&T. Hi, Matt McEwen. I am a manager of one of our upstream development organizations at AT&T that focuses on OpenStack development in the community and then also downstream in AIC. Jason Barfield from Ultimate Software. Hi, I'm Jason Verbelied. I am a cloud engineer at Ultimate Software working on the operations team. We do everything with our OpenStack clouds from the administration side. We build, expand, do contendents, quotas, everything like that. And last but not least is Rick Bartra, also from AT&T. Hello, I'm Rick Bartra. I'm an OpenStack developer from AT&T. I've been working in OpenStack now for about three years. I wanna thank you all for joining us today. So I know Matt has not taken the exam. So I'm gonna address this to Rick and Jason. So you took the exam. Did you pass it on the first attempt? So the first time I took the exam was actually at the Austin Summit. And so the first time I unfortunately did not pass the exam. I could talk a little bit as to why. I believe the reason was mostly because of the version of OpenStack that I was used to at AT&T. We were running on a much older version. And on the exam, the tooling was a little bit different. For example, for that version of OpenStack, the OpenStack client was introduced and I wasn't at all familiar with that. Then also the identity version we used at AT&T was a little bit different than what was on the exam. So I guess the tooling was just slightly different from what I was used to. Now Jason, I know you just recently took the exam. Yeah, I took the exam about three weeks ago. I did pass on my first attempt. It was, for me, I've been using OpenStack for several years now since the Grizzly release. I still used all the time on the exam. It's not an easy exam, but I was able to successfully pass it on the first try. Now Matt, in regards to Rick's experience, what do you feel is the relationship of the version of OpenStack versus what you're using in production? Well, the version that we're using, I guess I can answer that two ways, right? One way is that the version that we're using in production, we are really eager to get it up to the current version. So that's something, the skill set is what we're interested in. We're interested not just in the skills that run our current cloud, we're also interested in our folks learning the skills that are going to make them successful on the next version. So from that perspective, even if we're, even if the skills, some of the skills that Rick had to learn to be successful didn't currently apply in our cloud, we're really eager for him to learn it so he's prepared when we upgrade. And then at the same time, we're really eager to have our engineering staff be sort of fluent in the state of the art of OpenStack, right? We don't want them to be, in their own skills, a version or two behind. We want them to be up to date because we also want to be contributing code upstream to the community. Great. Now this is, for the two of you once more, without breaking the NDA that you agreed to when you took the test, how do you think the COA has influenced your work? Jason, I'll start with you and give you the mic. Directly with my work, the COA didn't really, hasn't really affected me, but what I see its value at is in our position, in our company, we have a lot of employees that are gonna be transitioning off of our legacy platforms which are very VMware and Windows-based into OpenStack and Linux. And if they're able to complete the COA and become certified, it'll give us a good gauge of where they're at and if they're ready to come in and start getting access and helping us out. And for me, I believe what the exam did for me in my current role and just the role is that OpenStack developers, it forces you to get exposure to all components of OpenStack as the exam covers x percentage of certain components. You have to learn the APIs. You have to know how they work. You have to at least know the basic networking. You have to learn all that. So it really forced me to study those components and just be prepared for the exam. And then also, I would say, just in troubleshooting as well, like we get production defects sometimes when they go down to the developers. Having that taken the exam and having a more broad view of OpenStack, I believe the exam prepared me for that. Great, thank you, Rick. Now a follow-up for you, Matt, is how do you see the opportunity being tested on the whole OpenStack affects your employees? Oh, immensely, especially on the development side. It might be a little counterintuitive. Why would you want your developers to have this broad-ranging operational background? But in practice, that's exactly what they need to be successful is what we found in practice. Because when you're getting software engineers, let's say straight out of college, they may not have deep cloud backgrounds. And to really understand how the code that they're writing fits into the big picture, they need to have hands-on experience with the whole stack. And the certification and the preparation for it is a really good opportunity to get the skills that they need to be successful as a well-rounded engineer. And those are some really good points. So going back to Rick from the development side, do you think the COA is valuable, and if so, how? I do believe it's very valuable, just so I'm going back to what Matt was saying, just having the whole operational understanding of how OpenStack works. And myself, like Matt said, coming out of college, not having a lot of cloud experience, the preparation that goes into taking the exam and passing it, that's what's really helped me be successful in my job. And Jason, as an operator, how do you feel about the COA? Is it valuable for you? Overall, for an operator, it's very valuable. It really shows you how to get into OpenStack day-to-day, how to create tenants, how to manage your users, how to manage your quotas. It's almost an invaluable resource to know who is ready and who isn't ready. Great, that's nice information. If I can add on to that too. I don't work in the operations side at AT&T, but I know peers who do, and in our operations team, they are starting to really encourage their staff to take certifications for OpenStack. And in fact, on our Tier 4 support team, there's a goal for all the team members to get the certification by end of year, so pretty ambitious. They can do it. All right, now, Matt, this is gonna be for you. So how do you think a test like the COA impacts current employees, and what impact does it have in the hiring process? So yeah, two very different questions. For folks that are interested in a job, I would say that it really sets you apart, because one of the things that is most difficult to understand from an interview, anyone who's done an interview on the giving or the receiving end probably agrees, it's really hard to understand someone's drive and motivation just from a series of questions about what they know or what they've learned. A certification like this shows that you are willing to go above and beyond, and even beyond the skills that it proves that someone has, which are incredibly valuable in the employees that we're looking for. It also shows that the candidate is really driven to figure out what they need to and be self-motivated, which is even more valuable. Yeah, that's great information. And with the new performance-based tests like the COA, you either do it, you can't guess, you have to have the knowledge to back it up. Now, there's been a lot of talk about the COA test itself and some issues. So, I wanted to ask Jason and then Rick, what issues did you experience or have you heard from others? In my experience when I was taking the test, I did run into some issues with copying and pasting. This was difficult when I had to enter network IDs. I found myself actually going to Horizon more than I normally would in my day-to-day work for creating VMs and things like that. Also, the link, I couldn't find the link to the docs. There was a couple questions on the test that I would have benefited from quickly going to the docs and finding the answer. Rick, how was your experience? So, I guess my primary issue that I had the first time I took the exam was being able to use the CLI. Basically, I downloaded the OpenRC from the Horizon dashboard and for whatever reason when I sourced it in the CLI, it wasn't working properly. There's other ways around it. I eventually figured out how to get that working in the CLI, but as this is a time test, if you waste so much time doing that, unfortunately, you're not going to get through all the questions and the reality is you can't do everything in Horizon. So, that was my main issue. Another thing that I believe was kind of something that slowed me down during the test was the fact that if you skip a question, you have to go through every question to return back to the question you skipped. But I believe that's something that's being fixed in the new test, so. Now, Ann, as Rick already mentioned, we're in the process of improving the exam. Would you like to go into some of the new improvements based on just what is being changed as well as addressing some of their issues? Yeah, absolutely. So, like anything in OpenStack that we build, we're always looking to take that community feedback, put it into improvement, and create a new and better product, and that's exactly what we're doing. So, the COA, because right now it's about a year old. And in approximately four to six weeks, we hope to have a new version out. And that new version is going to address some of these things exactly. Actually, I brought my notebook because there's so many new features coming. I wanted to make sure I didn't forget any of them. Great. So, let me open this up here. Okay, one of the things we do here often is in the, so right now the exam is on liberty and it'll be going to Newton. There has been questions about where are those OpenRC files? They were hiding in your root directory for a lot of people, and that's kind of very confusing. So, all the resources you need for the exam are going into a new directory that will be clearly marked. And the questions are all being rewritten in a way that when you need to call something, it'll say, this is in this directory. I think that's gonna be a great improvement because it is a time test, and we're testing your ability to use OpenStack, not the ability to flip through your files and find things hidden. That's not what we're there for. So, we don't want people using time. We wanna make sure all the time is going towards do you know OpenStack. The issue about thumbing through questions will also be fixed, which I'm equally excited about. So, right now you can jump one question forward, one question back at a time. And if you realize, oh, I've made a mistake, six questions back, you do currently have to sit there and tab through. And so, we're gonna create a menu in which you can get directly where you wanna go. And as well, to speak to docs, we are improving, we've heard that people are, so let me back up. One of the other things, and I think maybe Matt, you alluded to it, is that this is about using OpenStack and being able to navigate what your resources are is a huge part of that. So, that's why during the exam you have access to docs and you also can use the OpenStack client and call OpenStack help. Because that's part of the day-to-day, is do I have a problem, I don't know how to solve it, let me figure out how to solve this problem. Not everyone knows everything. So, we're gonna make docs very clear. They're hopefully not in those Newton update but then very shortly thereafter, we're hoping to roll out a third panel that has, so you have a horizon, your command line, and then there'll be a third panel that has docs fully right there. So, we wanna make it as easy as possible for people to get the resources that they would normally have in their day-to-day use of OpenStack. Let me make sure I got all the big, I think those are really, those are the things that we hear the most. Oh, one other thing that is gonna be a great improvement. So, right now the exam is only offered in English and obviously for our community that's, can be more troublesome for some than others. And so, we are rewriting the questions for ease of readability and comprehension. So, it'll be very clear what the task that we're asking you is. It's not like we're test, we're not here to test your English comprehension, we're here to test OpenStack. So, it'll be very clear in the update what exactly we're asking you to do. Did I miss any other things that you guys felt like were big issues that hopefully we have on our list? Copy and paste. Oh, let me talk to the copy and paste issue. Yes, so in the exam, you cannot copy and paste large amounts of text. You can copy and paste about the size of a project name. And that trips people up right now currently because they're either thumbing back to get something that they were looking for and then they wanna copy paste that forward for reference. Gets a little tricky. So, in the rewriting of the questions without giving too much away, they're redone in such a way that the thing you might need to potentially copy paste is right there and very obvious. So, that will be a big help as well. One of our engineers at the OpenStack Foundation who worked on this exam, Chris Hodge, he couldn't be with us today, I really wanted him to be here. He wanted me to remind everyone this is a full Linux environment. So, while you can't work outside the environment, you could do something like install a text editor inside the exam environment. And it would be super, super clear because the proctor will end your exam if you were to pull up your own text editor outside the environment. But if you feel like you need it inside the environment, you could do something like install a text editor. It would take up some of your time, but you might find that very helpful. And as a full Linux environment, it will have either Nano and Vim already in there. But what Anne is saying is like, say you'd like something else or you want a visual one, you can go ahead and install it. One thing Anne did not touch on, and I just wanted to mention with the move to Newton, the OpenStack CLI is very intuitive. So, it's a whole lot easier to use than the former project clients and even more easily used than the Liberty version. Something else that I would possibly suggest as an improvement to the test, I would say possibly giving back not just a score, but maybe a percentage pass per component or possibly say, hey, these are the questions you missed. The second time I took it, I felt very, very confident about my test. I felt like I almost got a hundred. But unfortunately I didn't, but I was, for myself, improvement. I just wanted to know what areas did I do bad on on the test. So. Yeah, to speak to that, because we do hear that often. So, when you take the COA, you will get back a pass fail with a total score. So, unfortunately the problem is currently, if you get back a 50 to Rick's point, you don't know, do I know absolutely everything and I just don't know Swift very well or what are some of the issues. And there are some patterns that we have given back to people. You know, people struggle on troubleshooting. People tend to struggle a little bit because Swift networking as well tends to be a trickier area for some folks. We've heard requests to divvy the results back up that way. Haven't really made a decision yet and I think I'd be really curious to get community input and employer input about, you know, if community members would want that for their own good, if they would want that shared with their employer or not and those types of things. But. So, from a show of hands, who would be interested in personally getting that type of feedback in your results? I think that's a community response. That is definitely the community response. I think one of the things that has maybe held us up is if you took the exam and you passed everything but say you failed the networking part, would you want your employer to know you failed that part? I think maybe would they ask and then would you be in an uncomfortable situation if you just said yes. That's a good input. We do have the microphones and our next step is question answers. Let's just put up our chaos and answers. Thank you, Matt. So obviously this would be something that would have to be built in but have you considered making it a thing where the person can choose to release it or not via the platform? Like it's a thing where maybe there's a box that you can check that involves like, and I don't mean to add more work on, it's just that that seems like it might be a viable option in that if you want your results kept private or if you want- How kind of like the SATs? Do you want them sent to the college? That was my thought. That's a really good idea. We have some people I dragged from the foundation here to take notes because I definitely do want to capture all these ideas and feedback and capture what people's questions are too. I think that really gives us some good guidance as to the areas we need to communicate better on and what brought you here today and what you're interested in learning. You know, and that's the one advantage of this being the foundation's test is because it's our community, we do have a say. Members of the community are helping with the test, members of the community design the platform underneath the test, et cetera. So getting in touch with Ann, you know, if you have a question or something is super simple and she's very responsive. Oh, thank you. My email's just ann with an E at openstack.org. So you can always email me if you have questions you don't want to ask here about the exam or ideas or comments you want to share. Ann? My question is, is there any kind of some state statistics for who took this HEO exam from not in the United States and also America? For example, European countries, Japanese people or Chinese people, Korean people and so on. I want to ask, is there a kind of some state statistics? Are you asking like where you can take it outside the United States? Yes. So the exam is online. It's a virtual exam. So you can take it anywhere that you have a stable internet connection. If you want to do training, there are some training providers that do provide in-person training. There's also a lot of training providers that provide completely online training. So you can scroll through the open-stack marketplace slash training and you'll see those people. So I'm asking this. I'm running user group in South Korea and I want to emphasize that yeah, open-stack HEO is great. So please take the exam, but I'm not sure what is the advantage for Korean people to take this kind of exam. So in this case, what would be the best advantage for Korean people or Japanese people and so on? So yeah, this is my main motivation. Ian, I think it's the same reason that we would take it in the United States. Having worked for NTT at one point, you still have your skills and you still want to verify with your employer that I have a certain level, I have passed the certification test. I don't think it's national. I think it is international that you're passing something because you're using the same version of open-stack as the people in the United States, the people in Russia, the people in Poland. So it would be much better if I want to see some kind of some statistics, for example, 20% of Japanese people took this exam or and so on. I do not have the information on me, but I could do my best to look it up and get back to you. Yeah, okay, thanks a lot. Thanks, Ian. So, prefacing this, I said I had some comments and some feedback for you guys. I run the open-stack meetup in Colorado and we were donated by the foundation certificate to take or a voucher to take the test and we gave it, had a drawing and it just so happens that one of my coworkers won the drawing, totally random, I promise. And he didn't talk much about it so I actually interviewed him last week because he had put it off and put it off and finally he scheduled the test. And in my interest as an operator of a private cloud was as a hiring manager, not that I am a hiring manager, but as I'm definitely on all the panels, what value would I get out of it? And his feedback was very positive. He said, you will not learn if somebody is actually an operator or not. But if they pass the test, they can come in and do the job because they were able to do the research on the fly if they didn't know it. You don't know whether they knew it when they got there or they learned it on the fly. Either one is good enough to be an actual operator in the end. And so his comment was very positive in my perspective because I wasn't going to really chatted up at my meetups unless I got that kind of feedback. And he obviously had nothing to lose. He may not even continue working in an open stack. So it was a very honest feedback and so thanks for that. He said it was difficult. It was not, like you said, it's not multiple choice. It's actually go-do actions. And he felt that that was very valid. He did run into the cut and paste type issues. He thinks he misnamed one of the VMs just from finger speed. And I talked to some of the COA folks earlier. They say he probably got partial credit at least for starting up the VM in the right way if not the right name. So that's positive feedback. And I only have one question just because I am signed up to take it, I'm taking it next week. So you need to let people know that you can't sign up and take it. You sign up and you schedule a time to take it. It's gonna at least be 24 hours ahead of when you pay for it. I didn't know that. But I thought also that the exam was expiring but it sounds like it's been extended to like four weeks. So I wanted to take the Liberty one because I know Liberty better than I know Newton. The only question I have is when you take it, is there an advantage to taking it on a large monitor? Are you in a fixed window the whole time? Do you guys wanna take this one? I know, because there's also been, I think people have preferences about screen size and what they take it on. Yeah, I took it on a 13 inch screen. But I usually have my letters really small so I didn't have an issue. But yeah, there is a lot of feedback that the screens aren't big enough. I think Jason and Rick wanna address that? Yeah, I took it on a 15 inch laptop and I had some issues with the screen size. I had to scroll the browser window side to side. Okay, so if you use a bigger screen, it's gonna work, it sounds like. Yeah, it's a web-based web browser so it should expand. And you adjust your visuals so you can have the whole thing, yeah. I will, to just some of your earlier points throughout there, one of the things that's also improving in Newton, why I should have read directly from the notebook, is the speed of the exam. So I think people's stress about typing and I'm typing furiously because that clock is in the corner ticking at me. We're hoping to give people a little more room to breathe by speeding up the exam environment as well. So I'll admit I didn't finish a question and a half because I was being very methodical to make sure everything matched because I heard about the cut and paste issues. Yeah, I'll give you back. I did use all my time, but just because I had to make sure I passed this or else Amy would be very mad at me because she wanted me to have my CLA for the panel. Is there any other questions? All right, we'll hang out for a little bit at least outside. I know Anne has to run to a meeting at 4.30. Thank you all for joining us this late in the summit. We really do appreciate it. And I wanna thank Anne, Matt, Jason, and Rick for participating. Thank you.