 Is that better? Okay. There we go. So thank you all for joining. This has actually been a very interesting talk to think through and discuss and the only talk I've ever given at a an event where I've had press requests for interview to explain what I'm going to talk about beforehand. Which is strange because I thought the title was fairly self-explanatory, but so my name's Mark Baker. I'm a product manager at Canonical and with a sales engineering and marketing kind of background I've been to I I couldn't quite remember seven some it's something like that, but many of them participant in the win the enterprise working group Which is now just called the enterprise working group as of yesterday and do various things, but mainly my father now, so I really I really alert and the gentleman here at the front who just asked this question was that I'm aware that having a talk selected that says the second selection process is broken has a certain sort of circular irony about it and kind of disproves one of the major points I was trying to make when I submitted the talk so So I'm well aware of that and I already had alert number two is That after submitting the talk I was then approached and requested or asked if I'd like to be a track chair Which answers a whole number of the questions that I was originally raising when I submitted This talk, so but the good thing is it gave me a great insight into how the how the process works and you know what what works and and where the areas for improvement are so Motivations for this talk were You know I work for a vendor who works for a vendor in here that's involved in the ecosystem okay, and so I work for a vendor and There is pressure on vendors that want to be Participating in this ecosystem would be seen to be participating to try and get talks accepted right It's some it's part of how we represent ourselves to out to the open-stack community as active Participants is by standing up on a stage here and and saying this is what we're thinking or here's what we're contributing or these Are the things that we want to do And so there's pressure and that's not understanding how the system worked and actually tripping and failing to get talks Expected was one of the motivations for doing this right so Point number three on there was was says when other people were getting talks accepted There was sort of natural suspicions that well, what were they doing right? Were they giving the backhanders to people or or were they gaming the selection system? Do they have special inside knowledge that we don't everyone's shaking their heads to know backhanders so But there was but there was Definitely, you know, how come they're getting it is they just writing better talks they better presents as you know How's it working and of course we never want to think that right as when you're submitting a talk My talk is gonna be the best talk ever and if somebody else's talk gets selected Clearly they have unfair advantage. So it was suspicions around that I'm fairly to understand why some got selected but also to try and improve it because The way that you know having seen this over the certainly the last three summits It You know it felt the point that we need to try and improve things We need to try and change things and that was you know Probably the biggest motivation for for doing this talk so that we don't have to have the same discussion again next time so the great news This is kind of a luxury problem, right? And this is your slide a fine ocean people So thank you for sending me this graphic or giving the access to it But the luxury problem is is that the the summit has grown from you know a mere 75 people Although the number of people I meet that say that they were at that first summit You can add a zero onto that easily But but the it's a luxury problem. So it's gone from 75 people all the way up to wherever we are today It's like 7,000 something like that here today Last couple of days. So it's a big big problem. It's right. It's a big growth huge growth over over four years and that growth in the number of attendees means that vendors like us and Community members that are all very excited about open stack want to Participate want to submit talks want to be seen want to be contributing And so those numbers And again, thanks to people that provided this you look at the the numbers of total sessions But there have been since Since it started You know grew from San Diego were roughly a hundred to today in Vancouver, and this is just Somebody's already disputed these numbers, but so you can tell me if they're right or wrong 400 then is that for the summit including design summit call for paper. Okay. Good. Good. Good. Good Alrighty, thank you for that. So But I think I mean the big jump there between Paris and Vancouver was we have an extra day, right? Which is point number one point of view is I think we just seem to have more rooms I don't know that's true or not. So there were more slots available But the total number of sessions, you know, it's grown a big big jump with Paris Vancouver But it hasn't grown at the same extent if we go back the slide, you know from Austin all the way up To where we are today, right? So many more attendees many more vendors participating and proportionally less slots available If you map that against the number of submissions, well didn't get that data for Up until Paris, but 1100 or so and the same number again for Vancouver. So the talk selection process, you know, you're looking at sort of 10% 20% of the talks being submitted Are gonna actually gonna get selected right 80% of the talks get thrown thrown away and I've Spoken to several people who have said they don't bother submitting talks anymore because they don't the chances of it being selected And that's a sad state. I think right. We need to try and ensure that people have confidence that they're contributing Good content that it stands a good chance of being selected So the the four steps for those I know foundation people know this but for those who don't the way that this process Currently works is that that's four step process, right? There's a call for papers That goes out. I believe the call for papers for Tokyo is gonna go out pretty soon, right? Okay, so not very far away. We're only just recovering from one summit and then having to think about the next so it goes out Good sort of five months in advance of the summit happening there is in a process of voting and so Voting takes place online. It's changed a little over the over the last couple of years and we'll talk about a bit about that But community members will go and vote for talks that they see presented online and say, yes I would attend that and and that's you know, all goes into the the process Step three track chairs are appointed so for each of the tracks as enterprise and storage and networking And I can't remember them all but there's whatever eight nine tracks something like that there are four track chairs who are appointed and Who get to determine essentially which of the talks that have been submitted into that track make it onto the schedule and then Finally the selected talks are put up on the schedule That's some scheduling magic that happens by smart people with spreadsheets or whatever they do and they get put on the schedule And then you as community members or us as community members I just say can go and view the schedule and say I'm gonna attend this talk And you know, you'll see as you've seen many times this week There may be 180 seats, but 300 people say they're gonna attend the talk But only 60 turn up. So it's all those kind of various dynamics of confidence magic that happens So we look at step one This is a very open process, which I think is great in terms of the call for papers. Who submitted a talk in here Okay, give you a hand over if you had it accepted Okay, so less so there's there's no limit to the number of submissions from an individual Not that I'm aware of I haven't hit it and I've submitted 12. I think this last cycle, so There shouldn't be a limit. It'll be less than 12. So So there's no limit a number I Actually, I landed three which has turned turned out to be a bit of a curse, but There we go, I know we've only got thought of that one. So there's no limits a number to the number of submissions from an organization and Even though I had don't have that data. I would love to see that Rumor has it HP who's from HP anyone here before I've anything rude about them room has it the HP defense 700 people this show Is it okay? It's still a big number. Yeah, so how many talks did they submit right? That's gotta be very many And there are no rules regarding Submitting talks into multiple categories and I know that people do this right so they'll tweak the title of tweak the abstract and Submit it into multiple categories and so there were no sort of limits about that But I think that's a good thing right because you want diversity and you want to be able to get Lots of different perspectives lots of different angles on it. There's then voting right and So this is a very poor Analogy but voting, you know is often seen as being the sort of the one of the cornerstones of democracy, right? if it's if it's things is that the result of a vote is all fair and equal and of course, it's not as The bar boy is one example, but there's very many different examples around the world where they have voting systems that are just completely abused and so they are Not an indication of talking popularity of talk popularity on their own right So vendors in this room you can confirm or deny right, but most vendors and I've had this on authority, right? From from a couple of our competitors and partners and others that this process goes on inside organizations, right? Which is you know high internal in internal mailing list? Voting's begun. Here's a list of our sessions his links to our sessions on the schedule go and vote them up now Right, is any vendor you don't need to say where you're from and you can put you know, is this does this resonate? Notting going on that right and so This is essentially direct ballot stuffing, right and the bigger the company you are The more effective you're going to be able to do this. We don't really do this at canonical We have tried it, but we're not big enough to really make it then anyway But I know from talking to other track chairs you can see very big companies I won't pick on any but big companies that employ hundreds of thousands of people that you know Will have two or three hundred votes from their organization to go and and get a talk voted up Then there's indirect vote stuffing and and you know some people will call this marketing but people Blogposting vote for our talks. We've got really interesting things to talk about We think the key issues are networking or storage or scalability or whatever and we've got these talks about it So go vote them up. We do that There we go, so so yeah, we do that and social media campaigns as well. So we'll We'll tweet about it, right? We've got these talks. Please go and vote them and other but other people do that I see that all the time or ants are great at that go vote for our stuff, you know, and then use letters and all that So I'm not sure I should aspire to that, but they're very good at it And there's a lot of mutual back scratching that goes on so community members of You know, I'll vote for you if you vote for me and why don't we submit a talk together? And then we can both vote each other up and feel good about it so But luckily so there's a lot of abuse of that voting process goes on But luckily it's only one of the the inputs. There's tracks the track chair selection then and this Is you know the process of selecting the selectors selecting the people that are going to select the talk Typically chosen by the foundation. That's how I believe it works. I don't know if it's really true The criteria for selection seem to be based around people that know stuff, right? So I'm an expert in storage or networking or Nova or whatever it may be People that have shown that they are good members of the community and have been a good track chair before People who are active members of the community in some way haven't been a track chair before may not necessarily be a technical expert But have contributed to the community in other ways, whether it's documentation or joining working groups or whatever It may be running meetups or something like that um, I Wanted to call out this being a an active member of the track to select some process. So This is my first time being a track chair in involved in this summit And it's actually quite a lot of work to do it. Well, I think it's quite a lot of work and that you have to Coordinate with the other three track chairs and you have to take have a lease. What do we have? I think it's four kind of hour-long meeting discussions where we're stepping through the process and doing work offline Well, we're reviewing all the different talks and and for the putting our case forward so it's a time-consuming role it requires quite a bit of work and It's not a thing that you can take it on lightly so I can understand How the foundation will choose Pick James for example, not that you've ever been a track chair as far as aware But if James had been a really good track chair previously that he would get to do it again, right? I Can understand that process, but I still think there's a definite room for Improvement there how the teams work together is that again my experience it may be different in some of the other Streams, but there's typically four persons per track voting the relevancy of the topic How interestingly the synopsis was written or the title I'm very interested to see different types of talks being submitted so those Controversial ones or attention-grabbing ones. You could say that this was one of those But but you know you do see those the open stack is doomed and those kind of talks But also some very technical ones that you had to go and sort of Google to find out what it was they were actually talking about So all of these things the relevancy of topic technical detail How interesting the speaker was or how you thought they're gonna be just very difficult to gate We're all taken into account. What was very interesting is we quickly worked out that you could categorize these talks, okay? these are kind of inflammatory attention-grabbing talks these ones are sort of technical meaty content these are Trying to deal with people type of issues, right? So to categorize them into themes and it was very good to work in that basis The way that we certainly worked is that we all went our separate ways wrote up our list of the ones that we thought were good Made some notes as to why we thought they were good then circle back Well, you know the four of us or actually was the three of us because one of us didn't really do very much And the three of us where we all said yep, we like that talk that made it onto the initial list Very interesting. I think I'll go this point somewhere else, but oh Say here was that I'm very interesting is that there wasn't as much commonality as you would expect Right, we had something like a hundred and twenty odd talks in our track, which was the enterprise Stream and enterprise track and of those the initial list where we all said yet We want that was nine, right? That was under 10% which was I surprised me, right? I would have thought that Given that we're all kind of similar backgrounds and thinking about the same stuff as on an enterprise track The fact that it was under 10% I thought was was interesting So the initial late initial list drawn up We have a debate as to which ones are going to make it on which not people state their cases, etc They get put on the list some of them get moved out and put into other tracks, right? That's either they were just put in plainly the wrong track or we think it's a more relevant subject to go other so Some of the observations from this was I saw that as the number of submissions has grown There seems to be less weight placed upon voting. I don't know because I wasn't a track chair previously but Talking to some of the other track chairs is that certainly in the early days. There was much more weight placed upon the voting Now there is not it's only one of the inputs and Indeed we selected talks on our track We selected talks that had very few votes, but we thought that they were interesting and relevant topics, right? And it's just maybe they hadn't stuffed the ballot directly or indirectly or all marketed their talk at all So the flip side of that is that reliance on the track chairs has increased, right? So track chairs have to do quite a lot of work and they have to do that selection process But the track chairs see the see the talks very differently there's a lot of very interesting discussion some of it Quite exciting about which talks we think should get through Um There was very little guidance track chairs have an awful lot of scope right to determine what they think the issues are in that track and There was no Certainly not that I'm aware any sort of guidance given from tech committee or The working group relevant working groups in our case though when the enterprise group All the foundation for that matter on we think these are the key themes that you should look at I think that's both a good and a bad thing so There wasn't a great deal of cross-track collaboration and that may because we were a non technical track Maybe technical tracks have greater cross-track collaboration in terms of ensuring that themes or Talks to deal with in that way And there wasn't any kind of session feedback to take into account so Whilst content content is king and should be king. I think also trying to understand Whether that topic is relevant and whether that presenter very hard to say this diplomatically But whether the presenter is going to be engaging for people or not, right? and There was no sort of feedback to be able to look at in that respect So move on to some of the improvements. I think so And this is discussion right so I want to be able to see do you think? We should start to limit Influence the big organizations can have in terms of trying to Submit talks into this process or individuals, right? So take my example an uncle submitted about 40 talks 36 something like that of which I submitted About a third of them That's because I was tasked with making sure we get talks, right? But should should there be a hard limit on either the organization or the number? Should there be a hard limit on anything in that respect? What do people think? Yeah no No, they can be so as a as a sponsor you can buy a track or if you So, you know, we've done that HP have done that. I know a number of other vendors have done that That's clearly indicated as a sponsor track, right? It has this brown color on your gender And the Yeah, we did all right yesterday so The and as a sponsor Yeah, I know you'll keep me honest here But as a different levels of sponsor get one or two talks or something like that as part of your sponsorship But again, they're indicators being sponsored talks So there's no process by which oh you're a sponsor and therefore we're going to be more favorable to the fact that you've submitted Talks through the call for papers process Yes, correct, right So Well, and here's the thing so in our case, I'm submitting lots of talks in but I'm I'm putting other people to be the speakers Right, so I'm the submitter but not necessarily the speaker But in multiple cases we had you know, James was down to do about three talks, I think but So there we go. Yeah, but that's a very valid point and I wasn't aware of that gentlemen's agreement Can we use the microphone, please sorry should I should have said that no it's wired So I I don't want to rip it out of them. So I'll give you a practical example for another Conference for which I am I was on the selection committee now that I say this probably I won't be next year Two years ago, we tried to limit the number of talks by company and We found two things happened as the gentleman there said there's game playing which happens We've we've seen some cases where we tried to limit the number of talks by an individual and we just picked 17 people in a group John Jim whatever there's a whole bunch of names. Yeah, we picked talks by four different people And it turns out the three of them were mysteriously unavailable and all four talks landed on the same person Right. So when you try to gameplay this when you try to put limits You will find people will gameplay the system because the motivations you're talking about and I'm a vendor So I'm driven by the same things But I'm also an ATC on a project and I'd like to talk about my project, right? I think honestly what we have here, which is an open process is a better process agree and so my When I wrote this I I that the limit anything Comment is because I don't think we can limit anything I don't think we should limit anything because exactly with the moment We create a system then we're just gonna create the system that games it anyway Yeah, and so we're much better to work downstream in the selection process than we are upstream at this level So I think you've highlighted an important problem Or an interesting problem at least I I would like to see what the numbers are a third of the talks being accepted It's really a very very good metric If you say that there's 196 stocks being accepted or 200 top. What was the number the same 296? Close to 300 right If you have 296 out of 1100 honestly, that's not a problem You know, you can't it is not a situation where everybody can be given a spot But there are places where they have gone down that route where they've said and courage more Participation and for all the people who have not gotten a 40-minute slot Tell them will give you a 15-minute slot Right. Let me get the numbers. Yeah, 295 honestly 295 out of 1094 honestly. This is not a problem So I would say that trying to do things which seem simplistic like limits are Probably the wrong solution, but if there's places to do improvements. Yeah, absolutely. I think there are More diversity in talks like more different subjects That would be interesting, but limiting. No, thank you. I think the other problem you'd run into trying to limit by Company or by entity would be that then you are You're letting the corporate politics of each entity determine the talks late here. So for instance, I work at Cisco Yep, I've got I don't know a hundred and forty thousand co-workers or something And amongst those hundred and forty thousand I'm basically a nobody and I'm not saying that I should give a talk I should give a talk but I would never get to propose one, right if there was a corporate limit, right they would throw the marquee speakers at it and I think that the summit could potentially miss out Yeah, I'm feeling less recognizable, but dynamic speakers some of the brightest gems are deep in the mind, right? So I know at my own company HP and several other companies people put a lot of talking just to come to the summit like HP will send somebody who gets a talk accepted. So there's a lot of Relatively repetitive safe topic talks going in just to get people here and I've noticed I mean this summit There's so many Seth talk. It hurts My good thing is like I want I want to find a good Seth update But I don't know which one is going to be any good because everybody's talking about Seth And I've spoken to a lot of people who's ending up. Well, we had to present on something and there's always people interested in Seth So it's almost like the the the corporate motivation That's a good I think that's a great topic. That was raised that I think it was probably Clara Lauren raised that actually is Exactly a lot of people submit talk so that they are guaranteed a means of coming to that thing. Yeah, thank you Long-time listener first-time caller So it sounds like Like, you know people don't necessarily want to put a hard limit on the submission process and you mentioned kind of downstream in the selection process Yeah, there may be more opportunities Do you think that for track chairs? Getting data about this would be useful input Yes, so I think I mean there's two areas We don't let it but just at least track it like that this person submitted 50 talks or this is from you know, this is one of 200 that I think would be useful. Absolutely. So I mean I talked generally as I think there's transparency Period is going to help, you know, sunlight being the greatest disaffected and all those good things So and I remember there being an incident you'll probably correct me on this They're an incident where somebody got voted on to I can't remember the tech committee or the board and Then somebody did the analysis on where their votes came from and it was 95% from that individual's company, right and 85% and so I think that would be you know if voting data is available Especially for track chairs to be able to make decisions. I think that would be extremely helpful. I Think it's also voting is extremely hard I know as a community member who voted on on talks. Yeah, when there's when there's over a thousand You don't have time to view all the talks to be able to make comparative decision, right? and so You keep skipping through to find talks that you're interested in. Oh, maybe I'll vote for that one But after a while you get bored of finger ache or whatever and that's it. So I Think theme based voting would be very interesting, right to be able to say these are the themes, you know Seth I want more seven one less sir, right to be able to have Whatever neutron scalability or whatever it is themes so that to help track chairs be able to select talks based upon people's interest in those themes Sorry, so you had another point. I don't want to monopolize the mics. I'll when she go ahead and I'll So so my point was I love the idea of team based Talks so that chairs are given, you know, these are the themes that we want to Have talks around so that's great the second is we do need to also allocate or look for a new entrance into open stack whether it's women or underrepresented minorities or you know, absolutely new members or New people getting engaged in open stack. So we do need to encourage those types of players also to be able to speak Maybe first time speaker that was be a long time. So I think that's a perfect segue to what I was going to ask You started saying it would be good if the track chairs had information about Where the votes came from from what the young lady said It would be good for track chairs to know demographic information about the submitter I Personally think that's a horrible idea. I Think if you want to have a selection process, which is a meritocracy You should have as little information about who the presenter is and try and gauge the talk as a pure meritocracy Right and I know that this is a very very hard thing to do because Some people will go to the extent of getting a professional writer to write the abstract when the person is a shitty speaker Okay, come on. Yeah, there's there's a motivation here as a vendor to get a talk accepted people We've seen people go to extremes. That's an extreme. It's very hot I know as a track chair again We were trying to make a conscious effort not to look at the company that the speaker that the proposal was coming from Right because I cannot help but be influenced by that Yeah, so I'll just end by saying I think if you do get that information As a person submitting a talk now you have you have given me the strong motivation good or bad to include a woman on my presenter list Get a person who's a first-time contributor on my speaker list. I would rather Have it be known in the public That your selection process is entirely based on right only the stuff Which is in the abstract and everything else is ripped off before you get to see it You don't know the name of the presenter. It has to be blind He has to be a quote try to be a blind process though I know that a really blind process is a hard thing to do it puts more of a You know a weight on you as That's very interesting. I hadn't thought about it being blind in that respect So instead of name company just having some whatever unique ID That's correct And even if you were to do the voting that way where we as the public have to vote for talks strip the name of the speaker out so I'll mention the name yesterday I went to a talk just because I noticed the names of two people on the panel and I said those two people It's going to be an interesting talk Right, but I want to take that away and just leave it for a talk to say the talk is about Replication and trope. I don't know who's giving the top, but I want to know about the subject. Okay. Thanks. Thank you So I can say actually I think encouraging new contributors and actively Discriminating if you want on new contributors women in open slag center is a great idea It's something we need to do to rebalance the the situation we're in So maybe have every track will say we will pick one based on on the co the contrary reason Yeah, we will pick a minority or an underrepresented or whatever for one talk So, you know, that's actually happening and then do the bulk of them blind Because otherwise you get a lot of if you go purely on abstracts, you're gonna get a lot of group thing and a lot of Yeah, the same things and people who can write good abstracts are people who've done it before so you're fundamentally there Discriminating into new people perhaps who might have something interesting to say but can't polish the abstracts far enough so Actively split your your selection and say we'll do one or two talks that are Actively trying to encourage a certain thing and do the bulk of them on a on a blind vote instead So you get the best of both worlds Okay. Thank you. Good. Good feedback. So I think I Mean some of the suggestions I thought through, you know, obviously the role of the tractors crucial I think they still need to be appointed by a board or committee or some organizers I don't think we want to have any process of voting for tractors or anything like that because we'll just Extend this process forever of you know, how do we select the people that select the selectors so I Think there should be some level of rotation so I don't know whether I mean granted you want experienced tractors who are active in the process, but I don't know I don't know if anybody is always a tractor, right? Maybe there's a role for that, but I Think there should be a degree of rotation in them and I think that you should then this probably happens already But there should be some structure around the backgrounds from which tractors are selected so that you know There's Limits the difference that vendors have that we're getting users, you know, maybe a scale user, maybe a new user Because they have different requirements and different interests those kind of things Track chairs it would say not unusual for them to disagree in doing a little bit of research and saw that this is a Common problem and in talk selection and there's a nipped experiment that showed how they made To to select one set of talks use two different selection committees and then uses some fuzzy scoring and noisy scoring and stuff to be able to To get to this process. It was quite interesting What they saw is the two different committees, you know Had a 57% rejection rate of papers and so very hard to get people to agree on that So this is a common problem, but they're influencing their sorry They're implementing that system of noisy scoring to try and Fix it moving forward, but We can't really do that because of the scale of the problem that we're dealing with So I do think that we need to get feedback From talks so that we know As a track chair in the future if you'll be able to say This talk this topic got high scored very highly and we could see especially over time These themes are being scored very highly. I think that would be great input, you know for future tracks That's probably a bigger problem in terms of how we get feedback and whether that's app delivered or paper delivered or the incentives that people get Etc. Etc. But I think getting feedback on previous talks having Previous successful failure as an input into that process. I think would be vital and I also think feedback from the the board to tech committee and the number of working groups now that there was the session here Just previously was from the product group. I think They are looking at the roadmap and the themes that are coming in the future with open stack I think getting input from them as to these are the things that we want to deal with would be extremely valuable as the track chair What time is it? We're supposed to finish Now yeah, I thought so. Thank you. So that so that was it. So suggestions, you know As to how anybody other suggestions how we think we can improve your comments No Well, this is my first time so I can't say yes or no yet, but But I don't believe so I'd be very surprised that happened so But it's yeah, it's a good idea I mean in the absence of any sort of scoring system and feedback system that is one that could be used, right? Yeah, that's a double thumbs up from the back there anonymous scoring We'll be good. Yes, sir. Does that mean time's up? All right, thank you