 Good morning. Thank you very very much for joining us here today. My name is Linda Levy Grossman. I am the president of Theatre Washington and the reason why I am here today is because about two years ago Clay Lord from Theatre Bay Area and Brad Erickson reached out to me to share an idea that they had and that they were working on with Alan Brown and others about new ways to measure audience engagement and something that would be fairly revolutionary and they invited Washington to be a part of this national study. At this time our Theatre Washington cake was still baking in the oven and our organizational evolution from the Helen Hayes Awards to Theatre Washington was underway and we knew immediately that this was a key project and something that we very much wanted to be involved with especially as we were forming our new identity as well. I very much look forward on behalf of my colleagues at Theatre Washington to working with you as we go forward after today after we learn and understand this exceptional learn about more about this exceptional study and how we as a theatre community can work together to implement it to understand it and to really impact our audiences and to impact our community. As I said this idea was came from our friends at Theatre Bay Area our sister service organization who not only represent their own constituency in San Francisco but have become a major force in profession in advancing our profession nationally. I have enormous admiration for them I have enormous gratitude for them for what they are doing on behalf of all of us in professional theatre. I especially want to thank obviously arena stage for opening their wonderful cradle to us today and very much to play Lord who is not only the project manager of the intrinsic impact study he's also the director of communications and audience development for Theatre Bay Area and their wonderful executive director whom I have the privilege of introducing and that is Brad Erex. Thank you Linda I just want to say a few words and first of all of the the kind of participation and support that we've gotten from Linda and what is now Theatre Washington from the very get go with this project has just been amazing and I'm really thrilled that Washington and the surrounding area has been part of the six communities that we've been working in over the last couple of years with this project. As I was coming in the door Linda was you know saying to me that she was really hoping that the information that we would be hearing about today would be able to live on and would really be something that would be useful and and you would find ways to implement the ideas that we're going to be talking about today and I just want to say that very much that is the case. As a service organization we're you know and we tend to be kind of wonky clay and I ourselves we like that but data alone for us as a service organization really isn't all that compelling we want data that's going to be able to be usable by the field to help you do what you do better to make a deeper impact on your audiences to make that impression stick and last longer and that's exactly what this study and the information that you'll be hearing today will be leading you to we'll be hearing from we'll be hearing about the 18 theaters that we've studied across the country 18 theaters six cities 58 productions hundreds performances and 19,000 surveys were turned in that's the basis of this study and of course there were three theaters here in the Washington area that were involved in the project the arena stage where we are today will be mammoth and metro stage over in Virginia and they have been incredibly important in the study as well so you will be coming away I hope with ideas that you will be able to follow up on in your own theaters and at your own companies today and we're finding ways to take the information that we're hearing here and use it with advocacy and policy development as well we were in New York yesterday and heard from Jenny Luludis who runs art in New York there and she had just used information from the study in the book that you'll be seeing here as she was presenting just this week before the city council in New York trying to make new arguments for why New York should be spending fifty eight million dollars to support the arts and it was a very impactful message because they had been used to the economic impact numbers and they sort of told her we've heard those we know about them you have anything new to tell us and so she was talking to them about the way that the arts are really having a profound impact on communities and on people's lives and it's that kind of information that we'll be hearing about today so I'm really thrilled that you're here I'm thrilled for the participation that we've had in Washington and I do want to let you know you'll be hearing more about this today but as Linda and I were talking this is not just about a study that you know some of you were part of and some of you were not and that's too bad this is a study that is ongoing we have continued support from a number of generous funders that will allow us to continue this this kind of study going forward and we've also been working to make this this study really accessible and affordable so that every theater company across the country can do this kind of work themselves so with that in mind and with lots more to come I'm going to hand the microphone over to Clay Lord oh I should say many many thank you so thank you to our many funders that's part of my job up here and I'm forgetting it so we need to we need to thank our funders including the Doris Duterteble Foundation the Andrew Mellon Foundation enormous support from the National Endowment for the Arts and from the Cue Center for Arts and Heritage a number of funders as well around the country who have made this project possible I certainly want to thank Alan Brown and Rebecca Ratskiff everyone at Wolf Brown and Clay Lord who has been the project manager for this over the last two years so Clay and Alan. Thank you, hello my name is Clay Lord and I am the Director of Communications and Audience Development at Theater Bay Area we're a regional arts service organization that specializes in theater and dance in the San Francisco Bay Area we support both individual artists and companies which makes us a bit of an outfit to be the ones leading a national study but hopefully in the next few minutes and few hours you'll come to understand why we got so passionate about this that we we felt it would impact our particular members more if we were able to engage in a national conversation about this before we get too far though I do want to add my thanks to both Theater Washington and Arena Stage for hosting us and also to the other theaters Willie Mammoth and Metro Stage for being great partners in this work you'll see a slide in a couple minutes that shows you that we had somewhere close to 20 to 30 partners in this work total and it's it's really that's the only way we could have done it and so without them this wouldn't have been possible and I also want to thank the folks at HowlRound and Pound and Play we are live streaming today which is a very exciting and new experience for me so hopefully I don't mess it up and so thank you to them and welcome so agenda this whole thing is three hours long but we do give you breaks in the middle and there are moments throughout to ask questions but we also encourage you to raise your hand anytime you have a question the lights are a bit right on us so if we don't see you immediately wave your hand a little and hopefully that'll jog the eyes so we're in welcome and introductions right now and then we're going to go on to what's intrinsic impact and why should you care why I care by area cares and why we think it's a really important conversation to be having nationally and then Alan Brown who was the chief researcher on measuring the intrinsic impact of live theater will be presenting some of the top line results of his report they're very interesting exciting results and battle those two portions together with the Q&A will take about 75 minutes and then we'll have a break for 10 minutes in the second section we're going to be demonstrating one of the most exciting outcomes of this work which is an online dashboard will actually be showing you screenshots but the dashboard now exists and allows organizations to do this work for a much cheaper amount of much lower amount of money than than you could have done it before it actually integrates an online survey mechanism that then automatically transfers data into a database that people can look at immediately and see results with interpretation so that they can immediately within 24 hours have feedback from their audiences and then be able to possibly augment what they're doing will then speak to representatives from each of the three companies talking through some of their graphs having them tell you a little bit about their experience and then we'll have a kind of group discussion about what the benefits and limitations of impact assessment and any type of assessment are and why they're equally important to do anyway and then we're going to kind of look to the future we're going to talk about why engaging audiences is important why engaging them in getting feedback is important this concept of a long arc of impact assessment which is basically just looking at both what we're doing in the future and also what it does to people over time to be able to assess and tell you about the impacts that they're having and then practical implications of the impact assessment question and answer engaging the field at large future work that we're doing and then we'll conclude so that's the agenda some quick housekeeping so the executive summary is available online for free along with some other excerpts from the book it's available at theater they area that work slash intrinsic impact and I encourage you to take a look at that the books are available down at the gift shop and if you want a book please go ahead and buy one but don't feel like you are being pressured to we think it's pretty interesting and it's got a bunch of interviews with artistic directors from around the country as well that kind of augment the work or add to the research that Alan Brown did my email address is important only if you are interested in doing this work but I mean you can email me anyway but if you're interested in doing this work it's really the best way to do it is to contact me and we will be later in the presentation I'll be talking about some of the future work that we're doing we do have some subsidy available for 30 arts organizations who want to take part in this work and we encourage you to get in contact with me if you're interested in that and then the two websites here intrinsic impact org is a free resource that has a lot of kind of the background and history of this work of bibliography things like that and there's a hashtag which is pound new beans we encourage you to tweet and if you do use that hashtag so we know you did so who am I so I told you that I'm the director of communications and audience development in the area but I hope that that's not the most interesting thing about me and I wanted to start from a different place which is max from where the wild things are when I was four and a half years old my first theatrical experience was getting cast as max in where the wild things are at my preschool and I didn't know quite who max was or that he was important and my teacher kept telling me trees are important shrubs are important max is important so I kept telling my parents trees are important and they didn't realize that I was anything special in the play and so like my mom thought I was gonna be a tree she didn't bring a camera she got there she was very disappointed that I hadn't told her I was max because of course she would have gotten it anyway it was it was the most amazing experience for me it's it's one of the few memories that I have from being so young and it's it's kind of kickstarted a life of artistic experience for me and a lot of what we're talking about today is how on an individual level your artistic life leads you to where you're going so for example for me this eventually led me to work for theater Bay Area which is an art service organization and I particularly started working for theater Bay Area because of their mission which is to unite strength and promote and advance the theater community of the Bay Area and the second part is what really happened which is working on behalf of our conviction that the performing arts are an essential public good critical to a healthy and truly democratic society and invaluable as a source of personal enrichment and growth I was particularly interested in these parts essential public good democratic society personal enrichment growth I thought that art could change the world and I thought I found an organization that believed in that too and indeed we do and as director of communications I was like oh great well now we can talk about that and then we started trying to and we realized that for the people we were trying to talk to we didn't really have the right language to speak to them about these things in terms of what art does so we started asking our constituents a bunch of different questions and this is the audience participation part so here's question one why artists arts administrators do you make or consume art raise your hand and I'm not going to pass this until someone answers yes because it takes in different places and allows you to meet different people good to know that you and your experiences are not a lot good yes it helps you to express your feelings as an art maker or an art watcher as an art maker cool anyone else because you have to we hear that a lot yeah allows me to share my culture both as a member of the community and as an individual it allows you to share your culture as a member of the community and as an individual absolutely so and over gives an interview and spoke and what she says is that for her making making theater going to theater the best theater should be like Jim for the soul which basically works out to you don't want it to be so hard that people can't complete it but you want them to feel like they made an effort in the end I think that's a really interesting way to think about it and if you've ever seen a city company production you know how accurate that description is of what what angle guard on her fellow artists do really amazing stuff so what does the best art do like think about the best artistic experience you have what did it do to you talked about it for weeks stuck in your head yeah so provoking new questions for you and also encouraging you to engage with other people who might not be in your spirit yeah to go experience more and feel more deeply yeah connects you to other people yeah it challenges your core beliefs absolutely so this is a patron named Barry Levine he's a normal patron except that he and his wife go to a hundred productions a year so he's kind of what you might call a super patron and they've been going to a hundred productions a year for years they have all of their play bills I saw them they're in boxes it's amazing and he says he's talking about journeys and here which I don't know how many of you know that show it's a show about some soldiers in over one and it's basically leading up to them at the end of the show going over the brink and I'm real cheerful but for Barry three years later this is his response to being asked about that memory the experience is just amazing the emotional impact is bringing tears to my eyes right now he actually started crying you would just walk out of there like wow you got bam you got hit like that it's it's it's less articulated than it might be but maybe that's the point it's so hard to describe what that is how do you know that your words doing that to people and you know that it's resonating so people talking at an emission and afterwards yeah yeah and sometimes they're really resonating they don't even talk they just thank you so sometimes when it's resonating a lot they can't formulate they just thank you so sometimes it actually renders people in articulate they can't actually describe what happened to them yeah it reignites a love in the medium yeah who is an artistic person here artistic director okay so do you ever sit in your audiences why to hear their responses happening in what what does that mean so you hear them gasp you get them clap you have them laugh or you hear them silent and is that good or bad or depends depends yeah there's a way of the group breathing together so this is Oscar uses he's a artistic director of the public and there were four other artistic directors in the book who basically said the same thing which is that I sit in the audience and I listen and I just feel it and as an artistic director who's lived my life in the theater that's enough yeah and on the other side of the corner you've got a patron like Mrs. Sean McKenna he's another super patron in the Bay Area and he says the way you can tell that it's a moving production is that it sends a tingle through his whole body and that it doesn't happen very often but that it happens often enough that he knows it's gonna happen again which to me sounds like nothing so much as a drug hit so so that's what we do so then we get to this question how do you convey your organization's success at doing that to your funders to your government officials to your board members so anecdotal stories from your audience you show them you sold tickets yeah you use numbers and anecdote absolutely and we're gonna talk a little bit about anecdote versus numbers it turns out that anecdote is where we feel comfortable and numbers are where we have to go so this is Diane Ragsdale this is a very long quote but the basic gist of it is that if you keep ranking things the way you've always been ranking them which is often through numbers then you keep getting the same people at the top the same organizations at the top but if you start looking at some other things like who reaches everyone who's not going to those big organizations or who's actually generating a community-wide discourse who is changing people's lives and you get different rankings it's an interesting question this is Nicki Minaj Nana winter Nicki Minaj as you can tell cracked her outfit from Michaels this is at a fashion show and I can't believe they were actually seated together but this is my favorite picture Nicky's anecdote she's free spirited she's kind of non-specific but she's bright she's interesting she's where we're comfortable and then Anna winter over here with her body necklace she's the numbers but here's the problem they shouldn't really be together at least they kind of look strange together and no matter whether you're talking about advocacy or internal communication or fundraising or board oversight Anna Winters always gonna win and the reason is because Anna Winters the one who has the money so all of these people that we're talking to unfortunately are the people who have control over kind of the financial fate of a lot of our income and so then it becomes a problem for us to only speak in anecdote when what they really want is something harder and faster and more consistent so this is David Kilpatrick he's the executive director of a small community theater in La Crosse Wisconsin and he is the the inspiration for the title of the book and he says as artists and arts administrators we've turned ourselves into being counters because the people we deal with what they count as beans so this is where we are right now but wouldn't be nice if we could get here and we could make what was unmeasurable measure that's where we were at Thurga area we were looking for a way to do this looking for a way to take all of these inspirational things that we have found out from our community and turn them into something that would resonate the same way that we had 80% capacity on the show and we generated X thousand dollars for the restaurant down the street resonated with our funders our government official and then Brown heard this in a presentation that Alan Brown was giving at the National Arts Marketing Conference in Miami and he heard Alan say this and then he heard Alan go on to talk about what this meant to him if you can describe something you can measure it and Alan first heard this from Ed Pauley he's the Evaluation Director at the Wallace Foundation and this is the cornerstone of all of the thinking that has gone into this work at Wolf Brown for the last five years and it's led to what's what we're calling intrinsic impact which is the intellectual social emotional and empathetic impact of art on an individual and the important part is the end words measured using a standard metric and a common vocabulary so that everyone is speaking the same language but you're talking about the right things this was a big exciting light bulb moment for our organization and so we immediately went out and we talked to all of our thunder friends that we could find the Doris Duke Foundation the National Endowment for the Arts the Andrew W. Mellon Foundation the Pew Center for Arts and Heritage in Philadelphia the San Francisco Arts Commission City of San Jose Theater Development Fund we also made partnerships with LA State Alliance the theater Alliance of Greater Philadelphia Arden York theater Washington Arts Midwest we've now added Arts Boston and the League of Chicago theaters we did an open application process and had over a hundred theaters respond to us asking to be a part of this work and unfortunately we only have funding for 18 of them but we have 18 of the most diverse theaters we could find in terms of size type of programming geography it spans from a community theater in a town of 35,000 people to round about theater one of the largest nonprofits in the world we did interviews with eight other artistic directors who were not actually involved in the study as well as 12 artistic directors that were compiled them all around this question of what is the role of audience feedback in the creation of art when do you start thinking about your audiences in your artistic process and the conversations are amazing and they've really informed a lot of the thinking about how this work is applicable and in the end two years of research later and honestly this isn't supposed to be part of my job but it certainly is exciting to do two years of research later we've had 11 months of surveying 26 theaters 12 cities 24 original interviews with four patrons included there four original essays from people like Diane Bragg sale Arlene Goldberg Rebecca Nova 58 production 60,000 surveys went out placed by hand by very intrepid usher's marketing directors amazing people who then went back afterwards and counted all the ones that were left over so that we could figure out a response rate of over 40% it's an amazing amount of work that a lot of people did and it's culminated in this book and this conversation and we hope that it generates a whole future of thinking differently about how we talk about the value of what we do so the book is available in person today as I said it's also online and without further ado I'm going to pass it over to Alan so that he can actually speak through the results of the research thank you for coming thank you clay and good morning before I start I just like to thank you all for taking such a big chunk of time out of your day to be here I'm so honored that you would do that and honored to be on this incredible stage in this amazing building today at arena stage we're having quite the excellent adventure it started in Chicago Minneapolis Boston New York here today and tomorrow in Philadelphia and then San Francisco and Los Angeles next week so it's a bit of a blur but there are some really wonderful treasured colleagues here and first of all I'd like to ask just so I get a sense of who's here are there any students or interns here in attendance today wonderful thank you so much for coming I'm wooly mammoth folks just so I know where you are great thank you Metro Stage Caroline thank you for coming and arena stage staff many of you thank you for coming I think we have some friends from the Kennedy Center here a number of you wow didn't mean to close down the Kennedy Center that's fantastic and we have also some friends from the National Endowments for the arts who are here with us today a couple of you and Jonathan NASA wonderful I'm so honored that you're here I'm going to go quickly through some of the research findings really just sort of surf the highlights of the findings hopefully provoke you a little bit to think about impact assessment I I hope that you go home from this session with a more nuanced understanding of impact assessment in general the different approaches to it the the pluses and minuses of undertaking impact assessment why a theater or another art group would why would you get involved with this and really able to kind of lead a conversation within your organization about impact in general so as Chloe mentioned our methodology involved paper surveying and it's very intentional because we needed to gather data from as random as possible across section of audiences when you email ticket buyers you're only hearing from decision-makers mostly in ticket people who acquire tickets you want to hear from all the other people who come with the ticket buyers you actually have to go into the venue with a paper form and that is what we did there are benefits of course of going with online surveying which which relate to the economics and timeliness of reporting but in this case we took the road less the more difficult road and we kept we counted the forms that were left behind so we could calculate what we call a pick-up rate which was 65% in your courage theaters to reuse their surveys that were left over to distribute them in another performance just to get more a higher pick-up rate and our response rate as a function of the pickup was 45% on average okay so 45% is not the percentage of all the surveys we printed it's this percentage of surveys that were taken home from the venue and that ranged from you know quite a bit and there were some interesting regional differences in response rates people in the Midwest take surveys confirming a long line of market research assumptions God bless the people in La Crosse Wisconsin they were just waiting for I have to say that it's very much you know you kind of get out of this what you put into it in terms of in terms of the effort level of messaging to the audience that this is important to us we ask you to fill out this form for a reason so so in some sites curtain announcements we're done lobby signage and other response enhancement efforts and the response rate very much had to do with that at the end of the day people fill out surveys because they want to support you there was no incentive involved here and because of that in a moment there's a bias we call loyalty bias in all art surveys really is that people who are who want to support you are more likely to fill out a survey we also surveyed staff at the 18 theaters about the impacts they anticipated their audience would report and we asked them to do that before the show opened before they actually saw the audience react to the show because we wanted to try to capture what staff was thinking about impact and it went right into the other staff surveys right into the dashboard and were displayed right next to the audience's actual response and you'll see some of that data as we go through the dashboard today it's a great conversation starter so in terms of questionnaire design we produced a template with some mandatory questions and many optional questions and each theater could customize their survey to some extent based on their own sensibility their own sort of artistic philosophy so as we go through here I'll kind of point out mandatory questions and optional questions but this allowed the theater you know our bottom line is three pages of questions in 11 point type no squeezing so you know in a hall you have light issues you have people who have visual issues so we really can't go smaller than 11 point type I feel in a venue before any discussion of data from any researcher you always need to ask about limitations and disclose bias there's always some bias and in this case it's very very important before we get into this that you understand some of these limitations and the really the main message here is that impact results across science across works of art are fundamentally not comparable okay you really can't compare emotional resonance of a production of cats at a big musical theater venue in Los Angeles with a production of the agony and ecstasy of Steve Jobs and Willie mammoth here for different audience different work of art different venue we really have to bear this in mind as we go through here because there's a lot of contextual benefit to seeing impact score is kind of comparatively but it's not a contest a work of art does not necessarily should not necessarily produce every impact you know they're not that is never the artist's intention necessarily we and we shouldn't second-guess the the creator's intention so I guess that's the that's the main thing is what we do encourage is within an institution look at your impact results because it is the basically the similar audience in the same venue and the work of art is different and I think there you've got to let more much more of a like to stand out in terms of comparability so as with any survey there's some openness to interpretation we tried hard to use simple language in our survey protocol but ultimately there were some words and phrases that were subject to interpretation my first this is merely for your entertainment is age patterns with subscription package and you know that's not a clear picture there never is one and I'm not going to dwell on be a whole nother conference but our data certainly showed and by the way we did not wait our data for anything subscribers responded at a much higher rate than single-ticket buyers but we didn't have anything to wait to so we have lots of single-ticket buyers in the sample so we can look at them but we didn't wait the data to try to adjust the representativeness for just presenting you a kind of raw data so one and this is really one of the things we're going to talk about this morning is single-ticket buyers you know the distinction between a single-ticket buyer and a subscriber is kind of blurry because there are lots of single-ticket buyers who use to subscribe the don't anymore there are lots of very knowledgeable single-ticket buyers subscribers certainly don't have the corner on that and you can see here the you know the age so let's first look at motivations we asked we provided respondents with a list of 11 motivations and asked them to take three that best represent their reason for being in the theater that night and this graph displays seven of them the uppermost blue line is is you know I haven't forbid to relax or escape you know I think sometimes we kind of forget that people just go out to actually have a good time and actually what's so interesting is that goes up with age and then kind of eases a little bit at the 55 and over level and right below that is to be emotionally moved or inspired you know as a motivation you know some people do go out really wanting to be moved and other people maybe don't think about it so much but that also goes up a lot and then eases the green line you know to spend quality time with family members thank heavens goes up for the child rearing years and then really want to do a focus group with those people towards the right hand side of that scale we're just other priorities kick in to revisit a familiar and this is really interesting it is a correlation with age now I think this has a lot to do with the musicals that were in our sample there were 12 or 15 musicals in our sample and there's a very important phenomenon happening in the impact system of people valuing familiar art it's the Christmas Carol phenomenon the nutcracker even with opera people going back to the same operas musicals this we have been very focused over the years on aesthetic growth you know stretching people the odd you know introducing something new because that's a that's a value that's prized by many artistic folks want to stretch their audience but there's another side to that which is the value very legitimate value people get from reconnecting with art they know and love and and those two things coexist and so it's interesting as the age people go into the higher age cohorts we find the revisiting familiar art it becomes a more prominent motivation okay the black line is very intuitive because someone invited you you know younger folks are much more likely to come because someone else invited them which really just speaks to the the social importance social of the importance of social context in driving arts attendance for young adults is is huge and then the yellow line is interesting is for work or educational purposes you know we wouldn't expect that to be high but for the youngest cohorts of adults there is a you know some of whom are students there is a personal connection to the art form I believe I know this from some other studies we've done that that the the very younger adults are actors themselves or in school or study theater or you know have some personal connection and therefore so the backstory and motivations is that if you look because this data set is so large those of you who are researchers will hopefully relish this this is almost 20,000 cases almost any statistically running this data set is statistically significant because there's so many cases so so it really allows us to dig deeply into the relationships between variables because there's so much data and we have to look not just at statistical stability but at what we call them as researchers effect size the amount of the variance explain between between variables so I'm going to talk a little bit because you know I can kind of distort the results and tell you that this is statistically significant but it might be a really weak relationship and you have to be careful here so sure enough the people who show up to discover something new that's not a misgraph it's one of the motivation reported higher intellectual stimulation the people who show up to be emotionally moved reported higher emotional resonance the people who show up to learn about or celebrate their cultural heritage were more likely to report social bonding all right so there's a very subtle but significant relationship between the intentions the motivations people have and the impacts they report just fascinating if you think about it is that intentionality creates outcome like I suppose anything else in life right so setting up you know if I'm your marketer I think this really means setting up accurate expectations is really important we asked all the fears to report what percentage of the house was sold on the night of the performance and this allowed us to run a regression analysis between anticipation levels reported by audience members and the percent capacity sold of the house you see there there's a discernible relationship as the house gets fuller anticipation levels get higher there's a statistically significant relationship there it's not a huge effect size but I can tell you confidently that people who are in fuller houses report higher levels of anticipation makes sense right now there may be other co-factors there like the there's a star in the production or if it's a more popular production therefore it might be sold better you know there there are other reasons for this but I think this speaks to using appropriately sized houses for the work to kind of so the there's a natural relationship between the venue and the art because when the venue is full people report higher anticipation as you'll see this actually drives impact by the way if you don't have questions or comments just stick your hand up or holler at me yes can you unpack what anticipation means in this context because you're saying that people report directly following a performance that they were anticipating it so good question good question the question was for the viewing audience can you unpack what anticipation is yes in the original study of impact from 2006 we actually surveyed people before and after and we using a control number we paired their surveys and that allowed us to in the in the pre-performance survey explore what we call readiness to receive familiarity you know are they normally do they normally go out or is this a new and different experience for them and their level of anticipation the specific wording of the question is how excited were you before the performance started but because we couldn't afford to do pre-performance and post performance survey we had to ask people afterwards to speculate as to their level of anticipation before it started and that's a tricky cognitive exercise and we just have to believe people well and there there is I guess there's equal possibility that there the size of the fullness of the house is affecting anticipation or that people who anticipate show more and more likely to buy tickets or more people are buying tickets so the house is full you can't really you can't really the correlation can run either direction is that correct right yeah you can't prove causality but we'll get into anticipation a little more so a number of our questions related to pre and post performance engagement and I'll share with you some of those results here and this this graph charts a couple of pre-performance indicators by age cohort that red line in the middle there orange line is the percentage of people across all these productions who said they did anything to prepare about 25% so so stop right there and ask yourself you know is 25% an acceptable level of preparation you know of course the question is well what do you mean by prepare right well we actually asked the follow-up question in many of the surveys what did you do to prepare and it's very very interesting what people consider preparation some people who separately reported reading a review did not report preparation and other people who read a review did report preparation so there's ambiguity some people think reading a review or preview is preparation other people don't searching for information online interesting a lot of people cited Wikipedia which I thought is so interesting given how much effort we all put into producing our own synopses and for some of the productions especially the Shakespeare work and the work based based on this literature a lot of people are going back and referring to this source literature and in some cases reading a book or a play before attending it which is amazing but look at the relationship between reading a preview reading a review by professional critic that's the green line look at the relationship between that and age cohort this really tells a story you know contrast that to the percentages to say to read comments from written by friends or family you know on faith mostly on Facebook I imagine you know this is this generational differences going on here abundantly clear there's a big shift going on in terms of who is reading criticism and sort of who they who they believe there is another story just before we go on we also ask people how familiar are you with the story with the cast and with the playwright and in in looking at familiarity against anticipation it was familiarity with the story was much more highly correlated with anticipation than familiarity with the cast for the playwright so the story kind of knowing the story tends to lead to higher levels of anticipation yes yeah the plot yeah uh-huh there was another study that came out around movies I think the media nicknamed it the spoiler study that basically gave the same finding which is that people who actually know the whole plot of a movie say that they enjoy it more than people who don't which is fascinating I'm not sure that it leads to reveal every plot twist in your show but um one of the things that we've been advising is perhaps in your marketing materials um try to consider a less ambiguous image for your art than you might otherwise um if you feel like it's gonna it's gonna be a show that people might not know very much about so that they at least get a hint about the era or some major plot piece or you know some structural thing yes yes 10 percent of the variance in anticipation is explained by familiarity with the story cast no company so the producing oh no that was not an item yeah so messaging about story that's what I take away is the importance of messaging about story all right let's look at post-performance engagement um you know just generally a downward trend here uh by age which is very interesting is that there's there's a tendency you know except with regard to reading the program book afterwards which is really interesting provocative we I tend to think of program books as preparatory in nature mostly and what we're learning through this study is that the program book for a lot of people especially older folks is a meaning-making instrument something people go back to afterwards and I'm not sure how much thinking we give the program book uh if we look at it through this lens uh if we might add some other kinds of content to our program books that would help people discuss a play afterwards perhaps ask questions of each other on the way home yes yeah we're surveying people within 24 hours um so at the top of that is that red line email or speak to a friend you know that's word of mouth basically and and this won't go into detail but other studies you know the dominant mode of post-performance engagement is talking about the work on the way home like how much did you pay a consultant to figure that out it's like of course it is you know and but I think a lot of theaters spend a lot of energy producing in-venue discussions when the larger win is eliciting conversation between patrons outside of the venue and I I don't know that we as a field have really applied ourselves to that challenge as best we can so other items here you can see the green line is search for information online the blue line react online through source through social media it's kind of age driven at the very bottom is attend to post-reforms discussion and that's merely a reflection of the availability limited availability of these discussions at the theaters that were surveyed okay one of my favorite questions probably my favorite question the whole survey did you leave the performance with questions you would have liked to have asked the actors director or playwright we call this our unanswered questions question and 35 percent of all audiences on average said yes I left with questions and 98 percent of those people actually told us what their questions are and they phrased them as questions and this is so rich we're not asking people did you like it or hate it we're saying what questions did you leave with it and and and you can read these questions and really understand how the work is resonating and we categorize the productions by type from experimental all the way over to contemporary musicals and on average of course different kinds of works generate more or less questions you know look you know this is the percent who left without answer questions the glow is it's a wonderful life of radio play I've seen lower mom of me mom of me which is a good thing yeah because you don't want a bunch of unanswered questions to a high of almost 70 percent for theater at Boston courts production of El Camino Real and what's interesting is there were two pairs of productions in our sample same play different location of the green dots there the two productions have let me down easy one of which was produced by arena stage very similar proportion of audience members leaving with questions they'd like to ask Anna DeVere Smith um two different productions of ruin one of which was here in the arena stage another was at Berkeley Rep very different very different levels of unanswered questions and this is fascinating to me we can't really understand this why you know what might explain the difference it was at the production was it that the way the theater engaged their audience is that their history of engaging audiences you know is it good or bad that people leave with unanswered questions depends what they do with it she spends on the questions absolutely uh-huh I mean we consider if you leave with unanswered questions we take that as an indicator of intellectual stimulation it's like the wheels are turning you have been provoked to the extent that you have questions now that your questions might be you know how did how did you sew that costume or they might be why did the plot take that turn or in many cases why did the play write title the play that title it's amazing if we really looked at we really looked at the types of questions people were answering and the categories there's a mountain of qualitative data in this and there's at least 10 pieces papers I'm convinced those of you were students um questions you know about inspiration yes which production of ruined represents which I'm not supposed to say because it's not a contest but in this community where I mean I think a lot of people are familiar with one over the other um so actually actually I'm not sure so what I will say is that um my hypothesis is I don't actually know my hypothesis that Berkeley rep is the one that had more questions and um I think that that's basically just because they have a very concerted culture I'm not saying arena doesn't but for many many years they have had a concerted culture of engaging their audience and having them do um post performance stuff related to kind of questioning um but it's also not it is not a good or bad thing I think that's really important to say and the other side of that is perhaps there are fewer unanswered questions because the post performance engagement was in some ways more successful um arena for example I believe until I don't know if I'm right or wrong did post performance discussions after almost every performance of the room and so that means that a higher percentage of those people had an opportunity to have a conversation about that work and then they filled out the survey so it's totally possible that they just had some of their questions answered what's more interesting is actually looking at the content of the questions um we've been experimenting with word clouds and you'll see some of them for some other questions in a second but drawing those questions into the word cloud program is really fascinating because then you can basically see themes you can see um moments that are jumping out to people um and you can then craft materials that answer those questions or you can leave them unanswered but maybe provoke that question in more people for example by passing out a sheet of paper with questions after the show and encouraging them to go home and talk about so the question was what's the correlation between post performance discussions and um an answered questions yeah that's a really great question um uh I don't have a an answer uh I had to go back into the data set and actually see if we can answer that but I think it's a it's an interesting hypothesis um and I'm just generally aware that that attendance at post performance discussions is an increasing in general more people are staying but that's very anecdotal you know I'd love to hear from you all kind of the big the big trend yeah yes let me down easy is both arena and yeah well and what's interesting about ruined as well is that if you look at the actual impact in like the summary impact indicators for the two productions they're almost identical so even though they're two separate productions at as a footprint an impact footprint they're almost the same which is why this kind of gave us so much pause and discussion is because these are 20 off from each other yeah couldn't you also look at that though as the difference of the community that's being performed in absolutely could be the difference of the community and and um you know it could be the difference in um percentage of different demographics within that community um so the community could actually so so kind of the the bottom line here is for me is looking at your seasons and your artists can you anticipate which artists which works of art will generate the most unanswered questions and what can you do to essentially anticipate what those questions will be and just catalyze more discussion allow people more opportunities to verbalize their questions perhaps discuss them with others perhaps discuss them with artists or or dramaturgs and essentially or hear other people ask questions because when people get answers to their questions they're making meaning from the art and as we'll see magnifying the impact of the art so moving on uh whoops yep yes yes yes we'll see that's my last slide um so bless you woolly mammoth folks we're showing you're hanging out all of your laundry here um these are the three productions uh at woolly mammoth the green area on this radar chart is um the agony and ecstasy of Steve Jobs been in the news lately really interesting twist of events uh a production of edifice l.ray is the sort of brownish red area and the blue area is production of a play a delicious play called booty canyon um and since the woolly people are here you're you're going to get a little um background on all three of these productions after the break um but as you can see just visually here you've really got very different impact levels for these productions um and particularly the uh one man show agony and ecstasy jobs really resonated uh with the audiences in terms of the emotional impact uh but you know the overall strength of the emotional response the feelings of empathy here's an indicator for you did you leave the performance resolved to make a change in your life well woolly mammoths elected that indicator because they felt strongly that was an outcome they were looking for from their work and you can see how high that Steve Jobs played score people left that play wanting to make a different wanting to actually get involved uh I guess it's about labor abuses in China right um and gaining you know our our lead indicator of intellectual stimulation on the bottom there is did you gain you insight or learning you know very different profile then this other sort of more um I don't know what you want to call it um didactic outcome did you think about the structure of the play during you know during the play uh and that was actually highest for booty candy and maybe we'll come back and someone from woolly mammoth will explain why that might be true um aesthetic growth much higher for edifice array and booty candy and much lower for agony you know and actually it's just in terms of being exposed to something new right so this just I just want to give you a flavor for kind of what it looks like when you put a lot of this data into one graph and you're looking at sort of what we call a footprint an impact footprint of different productions and you can imagine what this would feel like if this these were your productions yes exactly that is precisely what will happen after the break roundabout different theater different market anything goes big musical production really resonated emotionally um which is interesting that you have you have the added elements of music and dancing as well as drama um the importance of being earnest is the blue area very sort of classic production except what was different here is that brian betford played lady bracknell um which um clearly had an influence on the impact generated a lot of questions what was it like playing a woman are you looking for other roles where you could play a woman yeah it's very interesting join us yes we're just about to get there um we did ask frequency in reference to that theater not theater in general theater going in general but that theater um plays in musicals so we have 58 production we classified plays versus musicals now i can't i'm not going to sit here and tell you that all musicals categorically have different impacts on average than all plays there were 12 musicals which may or may not be representative of the world of musicals but you do you know nevertheless we wanted to do at least kind of an exploratory analysis here and yes plays and musicals do have different impact footprints at least once we we had a little higher captivation levels for musicals a little higher emotional feeling emotionally charged but much higher intellectual impacts for plays uh which in you know if you look at the nature of the works there's more experimental work there are a new plenty of new plays in here and you really see um the higher intellectual impacts um a much higher aesthetic validation for musicals social the sense of connectedness with others in the audience was higher for musicals right yes reconnecting with comfortable art you know in life yeah those are those are average ratings very large data sets so they're very significant statistically but again almost everything is is significant the the thing about this graph is you can visually see the distance between the observations and in this case you're looking at almost a full point on a five point scale so yeah um i'd have to go you know kind of frame that help you frame that but you know on average respondents reported higher levels of aesthetic validation for musicals by a factor of say 20 percent and i'm looking at one to five as if it were zero to 100 roughly yeah and joanne i think this gets your question we looked at um decision makers and the next line after this you're gonna be about frequency and you know sure enough people who said they were decision makers reported categorically higher impact well of course they do they got it together they sought out information they convinced other people to go with them and they're more they report higher impacts now your question joiner brings up an interesting issue which is sort of what's the difference between impact and satisfaction and we explore that in the original meps in the original major university presenter study and it really impact is a um you don't need to ask satisfaction questions when you have impact data because they're largely the same thing people who report higher impacts are more satisfied i think satisfaction is great to ask in reference to extrinsic aspects of the performance the toilets the lobby the parking satisfaction is great but if you're going to ask about the art we really don't need to know how satisfied were you we need to know what impact that it had on you it's more germane it's better uh because it's actually and it actually is the same almost the same thing so um i think this just asks questions about decision makers and how can we you how can we empower decision makers because they're more deeply staked in the experience as sort of ambassadors to the other people in their party and possibly the most provocative finding of the whole study here is this is essentially a story about frequency single ticket buyers reported categorically higher impacts than subscribers and not just by a little why are more frequent attendees less satisfied on average yes uh the decision maker aspect with some of this we decided to go to that show when you're a subscriber you're making a decision to see every show they wouldn't necessarily intentionally so more decision makers are choosing to be there correct right subscribers by a basket of risk honestly they right that's that's essentially and and you know it's really kind of amazing is a lot of subscribers actually forget what they've bought and it's like honey we're going to the theater tonight well what's up you know what's on i'm not sure we'll talk about it on the way and i think you're right i think there are other factors as well yes it's a blend exactly right i think most of the subscriptions across these theaters were fixed series but there were some choose your own yes but they were asked about variations on questions to try and get at how strongly they felt about the company and they universally responded very highly in terms of loyalty to the company which is possibly another thing to talk about here which is that subscribers are subscribing to the company and the single ticket buyers are buying the show and i also want to just point out that single ticket buyers here could be high frequency single ticket buyers who have chosen not to subscribe to you so it's not just one of people although it is mostly very low frequency buyers at your company yeah so this interesting conversation with Ben Cameron about this this sort of emergence of the super straining of single ticket buyers you know are very knowledgeable have the resources maybe don't have the time can't commit um so and this is consistent with other other research audience segmentation studies i've done opera theater classical music but there seems to be a segment of the you know the most active buyers there's a segment that's loyal to the institution and we'll basically go do and you know go here you read the phone book and there's another segment of buyers whose loyalty is to the art form not the institution and it's very subtle but important distinction uh it's just a big deal i think especially in opera this is you know there's people they're very knowledgeable they're going to pick exactly what they want to go to and it's a challenge you know because they're what they're basically saying is we you know we love you we love your work we're having very impactful experiences but we're not coming back frequently as people as you like and i think that's a question for you to take take home and really think about this is a little counterintuitive to me uh if infrequent attenders are having more impactful experiences you know what will it take to get them to come more frequently or is that even a valid proposition well and for us as an art service organization um we're taking this as as a positive we're i mean there's a there's a bit of a theoretical jump but you know we're looking at our list co-op data right now which in san francisco has 1.6 million households and for example we're trying to identify if there are clusters of arts organizations that might using actual data have more in common in terms of crossover rates and types of programming and might be interested in doing a multi-org version of a subscription that then would allow people who were curating their own experience to curate it across organizations and and therefore attend more frequently just not always with you which of course doesn't bring money in your door but ideally increases the amount of arts attendance over time in a whole community which ultimately does raise attendance okay i'm going to wrap up here as quickly as i can because we need to break um and we will have a q and a before we break so sure um we had some fun with open ends we asked people what emotions were you feeling when you walked out of the theater and it just just just unleashed this tidal wave of qualitative data um and so we generated some word clouds and uh i just i just love word clouds uh because they just sum up so much information uh the juxtaposition of anger and hope here the sadness horrified disgusted despair sorrow is a heavy emotional impact and you have to look at this and basically in reference to the work and the the intention of the playwright and the goal of the theater and presenting and asked did you achieve your artistic outcome totally different impact here you know almost all kind of joyful positive stuff there's a little bit of sadness here not a lot and then woolly here's your booty candy so and it's delicious i mean dramedy has said the the the bifurcation of emotions here is extraordinary and you know what's really interesting here is confused and disappointed for me and i i you know don't take that as bad because from the playwright standpoint that could mean success and i believe actually in some in some sense it did is because the player i was hoping to provoke people uh to question some of their own values and beliefs so to see confused you know is actually the outcome the desired outcome here okay we're gonna hit this a little more so you know this is data you know would you potentially include a word cloud in a grant report i'm hearing from funders who are interested in impact assessment because they wish their grantees would talk about something besides ticket sales in their grant reports and critical reviews so this is as far i'm going to wrap up with the slide this is as far as i've been able to take impact assessment because we have this enormous body of data now we can really test these relationships so if you start with familiarity and preparation right is all the stuff you can do to set up the experience particularly you know promoting familiarity with the story we talked about but also just helping people feel welcome in your space and whatever preparation you can encourage people to do all of that has a positive effect on anticipation how much are you looking forward to this right the r-square this is a measure of effect size is uh these two things in their totality explain 13 percent of the variance in anticipation anticipation is like the floor of impact it's the baseline condition going into the experience anticipation has a big effect on captivation how absorbed were you captivation is the gateway is the linchpin in the whole system for me it's just the degree to which you are fully absorbed and drawn into the work because if you're snoozing all the other impacts can't happen really so we have a correlation of 0.34 an effect size of 16 percent between anticipation and captivation this is why marketing is so strategic because you're setting up anticipation levels which actually when you look at anticipation against impact you'd see a skipped captivate right you have a 40 percent correlation a 0.4 correlation a perfect correlation is one positive right so our indicator of summative impact is a year from now how much will you remember how much of an impression will be left that turned out to be our best indicator of impact overall a year from now so there's a strong relationship between anticipation and the rest of the episode that's one of the key takeaways here right captivation is critical how absorb are you because the relationship between captivation and this indicator of summative impact is 0.7 it's and in a way captivation actually is is an impact right I think that this is one of the main reasons people go out to arts programs is to just get in that state of consciousness we call flow where they're fully absorbed they'd let go of everything else in their mind and they're completely absorbed in your art and just achieving that state of consciousness is in and of itself an impact and it also allows a lot of other stuff to happen so on the other side of this the post-performance engagement stuff that you do the making meaning afterwards has a high correlation to summative impact all right this was a question earlier right there's a there's a strong statistical relationship between post-performance engagement and reporting impacts overall impact so both on the front end and on the back end what we did really ask about was that you know all of the engagement stuff that happened during you know like super titles broadcasting to mobile mobile apps that you might look at during there's there's a wonderful controversy we won't get into around that fortunately there's a strong positive relationship between summative impact and loyalty measured by a likelihood of recommending this theater to a friend so people who report higher impacts are more likely to recommend your theater to report loyalty which is great but what where it sort of all breaks down is is is I can't say that higher loyalty actually leads to repeat attendance because we have all these infrequent people reporting high impacts we don't come back as frequently as we'd like them to so I just challenge you to think you know what can we offer in addition to an excellent artistic experience that will encourage people to come back the social atmosphere you know really what additional value can we offer people and how can we collaborate across the ecosystem to encourage people to just take in more theater in our art in general let's just take a few minutes for any questions before we break yes versus the actual talk back opportunity I think maybe sometimes the questions come up you know after you sort of digested it for a little while and that might be a little bit why people don't necessarily come to the talk back but instead those unanswered questions come later yeah one of the things where we are wondering about is when the optimal time is to do this type of survey because we don't quite know when when impact sets in the other thing to say is that the talk back percentage is varied a lot so there are companies that had up to 40% of their audience attending their talk backs and the format of those talk backs also varied a lot so you know there's everything from the artist as experts sitting on stage and taking very personal questions from the audience about their own experience and then there's something as as dramatic as kind of story circle style conversations that are happening where people are relating how they are personally impacted by the work and there's an interview in the book with Dudley Koch who is the artistic director of roadside theater in Virginia who pioneered this thing called story circles and and he talks a lot about story circles and talk backs as as kind of a really important part of them the mission of any piece of art is to kind of ask people to engage with that art very personally by by bringing that art inside them and then talking about what it brings out from them and you know I don't know what the actual data says but the the artistic directors at this book who certainly believe that that type of interaction is proving to be more useful as an audience engagement tool than authoritarian or authoritative I guess not authoritarian that's a different but authoritative expert talking to novice and answering the novices questions given that right now the research shows or looks like that the people who are getting the most artistically out of the performances are not people that are coming back consistently has there been a larger sort of amongst the people you've been talking to a question about whether or not to start moving away from the whole idea of looking at company-based audiences into specifically a show content-based audience and to start switching like models literally from the idea that oh our company has an audience to the shows that we specifically do about this this and this has one audience and this this and this and that's how we should be thinking entirely. That's a really interesting question about a shift in thinking away from company-based audiences to show specific and I've just been doing some research for the San Francisco Symphony into musical tastes construction of musical tastes and there's a wonderful kind of new construct in the research literature called taste communities which is used in reference to social media particularly young people identifying with a popular artist like Lady Gaga and the sort of overnight form formation of taste communities which I think is actually different than just liking someone or saying you're a fan I think it's there's a social dimension to it but in in classical music and perhaps in theater we need to think more in terms of taste communities not racially defined not defined by age but by tastes really and that this might be another way to think about your community and your audience instead of just classifying them in terms of the package they bought. I think also that there's um there are certain theaters that are that are already doing that I mean I think flex subscriptions in a way are kind of a first small step towards something like that but that one of the things that's really interesting in the artistic director interviews that we conducted in conjunction with this work is that depending on the artistic director there is a very strong I mean it seems that every artistic director makes the decision about whether their organization is trying to get people to buy the organization or trying to get people to buy the show and that that there's a fundamentally different way of programming a season if you're trying to if you're really trying to talk to people in in terms of shows so Jack Ruler at mixed blood theater in Minneapolis they have their mission is very strongly about speaking to the people who don't normally come and so for each show that they do they build an entirely new audience for that show and they take it as a point of pride that you know they had 12 people overlap from one show to another now as a marketing person that makes me panicky but apparently for them that that is a real point of pride that they were able to so diversify the work that they were doing and so effectively go after the people that were supposed to see it in quotes that they um that they were able to create those two separate audiences so it is already happening it's it's a big conversation and it's an artistic conversation as much as it's a marketing conversation well okay we'll take one more question and then we really should break because we're a little bit behind so you too can duke it out and one of you can ask us a question um yeah we're going to talk a bit about advocacy labor it's a great question the question was about um where discounted tickets fit into the kind of subscriber versus single ticket buyer scheme um I have I mean we've done some research in San Francisco on uh the differences between patrons who buy half price and the difference in patrons who buy full price um half price people tend to be people who are trying things out for the first time at the company um at least according to our research I also one of the things that the graph for subscribers versus single ticket buyers really says to me is that we might need to change our idea of what loyalty means and what frequency means um I know that anecdotally especially in San Francisco there are people who attend every single show that a company is putting on that they buy all of their tickets through gold star and and whether that's good or bad those people consider themselves to be incredibly loyal to that company they expect to be treated as a loyal member of that company they're willing to recommend that company in the same way that someone who subscribed is except that and in some ways they think they have subscribed they've just subscribed through gold star which makes a lot of people insane but it also means that you can't no longer are we in a universe where you can simply put the half price people at the back of the house because those people are as as important as taste makers and as word of mouth kind of representatives as the people who've been coming for 20 years and paying you the subscription fees okay let's take a 10 minute break and then we'll come back thank you so much really But Allison has been so receptive and so gracious in receiving, like, you know, a lot of people in the marketing world, so it's very, very long time ago. I also speak regularly at other conferences in Australia. You know, a lot of the words, I'm talking in the next few months. But your answer might want to work with some of the companies that you work with on this. I'm just going to start by surveying all the people who attended the conference. Not so much the conference. So he talks about the remembrance of the experience of coaching people around, you know, all that stuff. I'd love to see it. She's been very responsive. So it's a... Wear that. No, that's kind of design. No, no, no. We need to place a site thing to just protect us. There's plenty of stuff about it in order to start that site. And that's where things like pre-90s and trial-based programs, I think, are really helpful. It's the hardest. No. Oh, I know. Oh, I'm so, so interested. Okay. That's very sweet of you. Sure, sure. Yeah. Well, as a researcher, I think everybody... That's my, that's my feeling is that people who are impacted more are really like the teachers. Learning happens to me. They're not. But we didn't do that much. We need to think. We do know. We can do variations on that. We need to tap into vast body knowledge. In the hands of the paper and cards of the practitioners, two of them are so important. They're all in the same city. They're never tired of each other. And they're doing exactly the same work. We're talking about... Yeah. Are both some of the most regular... Right. Right. Interviewing men... Yeah, but the challenge I thought you had, especially if you want to spend as much as you could ever use for the show, is the idea of what they're doing basically, going to get whatever they can out of a show, but hopefully it's the best show they can get. And this kind of faith that eventually they will find something that's as wonderful as whatever they are. I mean, Sean, the equivalent show of Journey to End, if you remember, was the production of Six Carolings that you're an author from. Well, and also, and for him to be exactly the short-term kind of focus not so much on marketing, but on audience and audience development, audience engagement, and so of artistic... I think audience development is not a marketing problem. It's an artistic issue. It's much more than it is a marketing issue. Then it doesn't matter if they're friends. They're just not doing their own service by just talking about the question. That's important. It's totally... It's half of the conversation. Yes. Oh, good. Sure. Thank you very much. Yes. Oh, nice to meet you. That's what I'm really after. Oh, thank you so much for coming by. Good. Excellent. Fantastic. So, honored you came by. Congratulations you get on the possible circuit. I filled out your FBI papers. Oh, my gosh. So that's basically it. Yeah, I know. Take care. All right. No, no. Hey, I have a question for you. So... If you email me. Yeah. Well, yeah. Yes. The theater selected performance is the pre-surveyed. We just encourage them to make that cross-sectional performance. Like they don't present it. Generally, we advise people to stay away from pre-due. Yeah. Because they can very draw... But if they do that, maybe it's a big person to throw in a metanode because often, you know, how different the audience can be. Okay. Hello. Uh-huh. Yeah. It's hard to say. This study really wasn't really about how different the audience could come from. You know, it's just a blend. Yeah. Yeah. Well, some of these theaters actually surveyed the entire audience. And I stole it. In small venues, in the big venues, some of them only surveyed three or four performances during 40. So it totally varied, you know, by size and venue. But for the smaller venues, when you should ask Carol at the mental stage, she was just about to speak. She has a tiny little venue, you know, and she got a great response. Thank you. Yeah. Well, thank you for coming. Not like me. I get that. So she's actually an orchestra. Oh! Right. Well, yeah. Right. Well, and actually, you know, the issue for orchestras is basically most people are not concerned and that really is, just philosophically, how much context do you insist that your audience have before you play? Right. And the study said, if you've got a little bit of an orchestra, symphony, to go show that like 80% of the audience wants at least short interval. That's like 80% of the audience. I just think that number is just going to grow as fewer people have on education. You know, that the people are going to like the insight into why did you choose this work? What do you do? What should I listen for? You know, right. You know, so give me a little something, a little insight, and then, you know, it's like Martha Graham company, they produce an introductory video of their dance and they have created a five minute introductory video. The audience is supposed to be in the same sense of what they're seeing. I think the same principle applies in more sense that when people have some insight they can appreciate it. And then there are very local people in the audience who hate talking in the video. And this is the other thing that is getting harder and harder to serve everyone in the experience. So, you're starting to see the sort of fit for the experimentation of the audience of the program. I and I have more talking in the context. The same program as Saturday night is getting harder and harder to serve everyone in the experience. It's like, for God's sake, that's not about themselves. You know, just communicate with your audience of the choices they have. You know, and immediately at some level you do some sort of an answer to that where there is a primary, you know, a three minute absolute introductory video. The truth is that if they wanted to engage them, as opposed to say putting them together. Well, most of it's proprietary to the program, but I can put it into you. If you don't know me, I'm still working on art itself. Especially in relation to good energy and culture. I think that's a secretory part of the program. Luckily, we're going to make our work more participatory. It really isn't easy to find to get to know each other. Right. Okay, nice to meet you. Actually, I don't know. That's hysterical. But I'll send you the So, we're now into part two. Part two is dashboard demonstration. So this is a demonstration of the online tool that we created to essentially make this information more accessible and make more people be able to use the information. And afford to do this research. And so Alan is going to start by walking you very briefly through how to get into the dashboard if, say, you were a company who had set up this work. And then we're going to talk to each of our three representatives from each of our three companies about the results that they've had. So I want to introduce them quickly. So this is Chad Bowman, who hopefully you all know. He's the director of communications here at Arena Stage until tomorrow. This is Carolyn Griffin, who is the artistic director of Metro Stage. And this is Jeff Perman, who's the managing director at Willi Mamma. So thank them all for coming. Please give them a round of applause before we start. They all really, they and their stats have worked incredibly hard to gather this data. And so it's wonderful to see them engaging with it and finding useful tidbits to pull out of it. So here's Alan. Great. Very quick tour. We created a website called MusicImpact.org where you can go to get a range of information about impact assessment. There's a button called references. If you click on that button you'll get a little bibliography of studies of impact assessment studies around the world. And the top item is downloading this study. Which is the same thing that's in the book except it has the appendices which includes the survey protocol. The actual questions. So if you want to see the questions download that report. That's where you'll find it under the references section in the intrinsic impact. So the theater is to see the red button up at the upper right. It says client login. Clicked on that and this is what they see. Jeff we're using your dashboard here. For demonstration purposes there's seven buttons to press corresponding to sort of different sections of the survey protocol. If you scroll down on that homepage you just see the data that's in your dashboard in terms of what productions. And of course here you can see the staff. The staff results are loaded in the dashboard right along with the audience results. And then if you click on any of the buttons you get a couple of drop down menus. First just which shows do you want to display. And you can click those on or off or choose all of them. You can choose to filter your results by any of a number of demographic or other filters. It's just like cross tabs. Ticket type, decision roll, age, gender. And we've got a couple of you know whether or not someone did anything to prepare their captivation how much they were captivated a lot a little. And we can customize that for you. And there's also another drop down over there you can choose to display all the results by show or in aggregate. So if you want to look at all of your shows you can aggregate them that way. So it's just simple results. Percent Willie Mammoth said they did anything to prepare. Here's your free productions Oedipus L. Agony and Acid Steve Jobs and Booty Candy and you see pretty consistent results. When you apply a filter to that through the dashboard by ticket type you get figures for subscribers, single ticket buyers and other types of buyers that would be comps and groups and it's typically very small numbers so the N is small so you're going to want to ignore that and just look at subscribers and single ticket buyers across the three productions and just that just demonstrates that the whole purpose of the dashboard is to allow the theater to interrogate their results. Ask questions of the data choose filters, choose shows and actually get your questions answered about your own data. Alright, with that very brief introduction let's move on to our panels and our first panelist is Jeff from Willie Mammoth and Jeff could you give us a little background on your three productions. Sure. Hi everyone. So the three productions that we had surveyed were part of our 2010-2011 season last year was Edipsel Ray the Agony and XCFC Jobs and Booty Candy. These were three productions right after one another that started in the winter of 2011. Edipsel Ray was part of a rolling world premiere through our friends at NMPN written by Luis Alfaro. There were three productions of that show and it was a retelling of Edipsel told through the lens of an Hispanic culture in San Navarro in Los Angeles in modern day and in the prison. The Agony and XCFC Jobs for anyone who doesn't already know this is Mike Daisy's show about a trip he took to China to explore sort of labor practices in the factories of our Apple products are made and that trip was sort of intertwined with his telling of Steve Jobs history and Mike's own history as sort of an Apple fan boy. And then the final production Booty Candy was a new play by Robert O'Hara an African-American playwright who we've worked with a couple of times in the past. And this was sort of a semi-autobiographical piece about growing up gay and black. It was a series of sort of short comic sketches that I would say had sort of a loose sort of connective thread and I think took some chances in terms of its presentational profile which is I think why we saw such a high measure in terms of sort of the aesthetics of that particular show. So I mean what was great is I think these were three very wide ranging shows in terms of audience and style and type and I think really reflected the kind of work that we do at Willie very well. Great. So let's take a look at a couple of your graphs and I apologize you're going to have to crane a little. So this is this was looking at your free shows and it's split up by age cohort the question is after the performance did you email or speak with a friend about the performance after you got home. It's always interesting to see the variations here it's not surprising of course to see that as people get older they are less likely to do this activity I wanted to particularly point out the higher numbers for steep jobs which is again not terribly surprising but interesting and also to point out that for booty candy admittedly with a fairly small group of 65 plus people that they actually emailed and spoke with a friend at a higher rate. Now in the book Howard tells the story about how Robert wanted to try and reach out to African-American churchgoers as part of this play about African-American gay culture. I don't know if that's related at all but I find that fascinating and I think it explains a lot of what happened in that word cloud but I'm wondering if you can talk both about the experience of steep jobs and whether you made extra efforts as a staff to encourage this type of communication in that show and also with booty candy if there's any indicator there. Yeah booty candy that's a real mystery for me. Either people are totally puzzled or like the other folks are really excited about it it's hard to know. In terms of steep jobs I think Mike created a piece with the intent of trying to get people to act I mean this is very much his intention behind creating a piece and his folks were leaving the theater after the show you know we had a piece of paper that we were putting in people's hands to sort of say this is how you can act this is how you can take action this is how you can get feedback to Apple. So the show itself was constructed as a call to arms which I think is reflected there in the data. Great. This next one is is the three shows again also split up by age cohort after the performance did you reflect privately on the meaning of the work and one of the findings of this research that's been so interesting is that younger people seem to have more of a need to reflect privately and you can see that here I'm wondering and maybe this goes to some of the kind of booty candy confusion and audience structure there's more precipitous decline with booty candy than with the other two shows you William Hammett Historica has a very young audience is that to say that I mean do you want to kind of hypothesize a little bit about you know I think the feedback that we were hearing from audiences from booty candy definitely skewed with regard to how old you are and it was sort of it was sort of a chopped up narrative you know not exactly in a straight line kind of wove this way and that more like Saturday Night Live in a way and I think that we were hearing confusion from older audiences about that and excitement from our younger audiences about that which in a way it makes me think well maybe this actually should be shifted like if that's the case then older folks maybe they should have been reflecting more about it but maybe what this is telling us is that the aesthetics of the piece sort of turned off a switch for them so it's interesting and it is interesting how sort of deeply it declines yeah do you have so we're going to talk more later after we talk to the other two here but do you have any other general comments about your experience of doing this work as a theater company you know I there's so much data it's so rich I feel like we've only just scratched the surface I think you know I was kind of flipping through it the last couple of days just kind of prepared for all of this I think the thing that struck me the most was sort of comparing the staff's response to each of the criteria with what we were actually hearing from the audience and I was shocked to hear that we I think we really underestimated our audiences in many ways I was really shocked to see that I just kind of jotted down a couple of things and we really underestimated how much audiences were would be thinking about the structure of the shows which is something that's always a concern for us especially with more experimental work the response that people were going to have to the work and we overestimated how much discomfort people were going to have how connected they were going to feel to characters, their social bonding how many questions they were going to have so it really I think it really prompted some self-reflection on how well do we really know our audiences and actually maybe they are maybe they are hungry for stuff that's even further out on the limb than maybe we had thought they were you know that is it's very interesting to hear that has been the experience of some other companies in your vein throughout the country that there's a it's not necessarily an underestimation as much as it's just hedging a little I think there's one company in San Jose that participated where the staff really expected people to be offended by some of the shows that they did and there's a question where you can ask were you offended by anything you saw and kind of their measurement of stretching people is whether they've offended by the percentage of their audience within a given show and they didn't and what was more amazing is that it was their oldest patrons that were the least offended by anything negative people on stage who cares and so I think that that's a larger conversation to have is there if your goal is stretching and if your goal is really moving people to a place that they're uncomfortable going really how far do you have to go and also why do you have to do that like what is it about what's going on that's making people less reactive of that stuff cool, okay let's move on to Carolyn so Carolyn can you tell us a little bit about Metro Stage and the three productions that you did sure, thank you many of you probably know Metro Stage we're out in Alexandria so we are not it's so interesting to me that there are so many different professional theaters and they all have such different aesthetics and different missions and different audiences and we share some of our audiences and some of them we don't for Metro Stage we're in a suburban area close in but suburban and that does have to inform a little bit of how we choose a season we started with this round with these surveys we started with our holiday show Broadway Christmas Carol well because we were doing Broadway Christmas Carol because it's a spoof on Charles Dickens Christmas Carol and using Broadway show tunes as parodies and we've done very successful work with another musical musical musical musical that parodies Broadway composers and so why not bring back Broadway parodies because people love Broadway parodies and now we've done it two years in a row we'll continue but the point is rather than really looking for something thematic that is the umbrella over which we have done in certain years now it's more we know what audience is like we don't have we have some crossover audiences we have some that don't crossover but we know what they like and we bring it to them so Broadway Christmas Carol being Broadway parodies was one we did followed by our traditional January February March African American musical his eyes on the sparrow it was about Ethel Waters it was a one woman show with maybe just a piano and very popular we have a very big following for this time of year for this these musicals we've done everything from Mahalia to Alberta Hunter to many many many just finished Josephine Baker Ethel Waters very popular but there were times when you know our and often times an audience would be 98% African American so we serve that population during that period of time in our season with those shows followed by the real Inspector Hound the Spoof on Agatha Christie by Tom Stockard the reason we chose that this year that year is that we had done the year before Tom Stockard very different type of show he had adapted a French play called upon heroes and it had three really renowned actors here from Washington it was a brilliant beautiful piece was really recognized critically and people are still talking about it it had tremendous emotional resonance this show we brought to Metro stage as part of our season because we had roles for those same three actors plus others the terrific cast many woolly companies and so we brought that back it's a curious piece because it didn't have the emotional resonance that people might have come expecting because they had seen heroes the year before and it just doesn't have the intellectual element that one comes to expect from Stockard so it's a little bit of an anomaly and I think the results that we will see will support that right, right also the first result here so this is the three shows split out by ticket type and likelihood to recommend future Metro stage programs you overall have incredibly good numbers for people and I know Caroline that this was something that you took pride in over a year ago just kind of your pride in your audience and their loyalty to you is well founded it turns out I was interested to see here the kind of variations between your single ticket buyers and your subscribers again it's not terribly surprising but what you just referenced about Real Inspector Hound is interesting here if you take a look at your single ticket buyer numbers and kind of the shift down in the percentage of people who were extremely likely or very likely to recommend I'm wondering not trying to ask you anything very specific about the show but can you talk about generally speaking people's reactions I know you stand at the door after most performances and talk to people so why do you think there was that shift to a slightly lower percentage of people who were highly likely to recommend that work I think with Real Inspector Hound as clever and wonderful production of the piece people that's kind of confused they weren't really sure who murdered the guy you know I'm still a little confused so it just it was not as satisfying I guess the satisfaction would be for the real stopper fan for the really sophisticated theater goer who understood that there were all kinds of crazy things kind of we had the two critics on the stage watching the play that we were watching from the other side and it all could have been just and then it would stop in the middle and they'd be talking it was very stopper very very clever but for maybe your average theater goer a little too clever do you feel like that had you known this amount of response was going to happen is there anything you could have done differently to prepare people you talked about a lot of people came in thinking this was going to be heroes and heroes is an incredibly emotional kind of invention of love style stopper as opposed to this style stopper so is there variation in kind of pre-engagement stuff that you might have done differently or not really? I don't know because I think variation stuff is so vital and so important that's come out of all of this I don't know for this particular show my inclination is should have done a different show you know I mean I just don't know how much you can prepare people for the abstract nature and the curious nature of the way stopper structured this piece and as really as it's still fabulous word play fabulous stopper but I just don't think it's to everybody's taste and maybe it's to people's taste who aren't coming to metro stage but to reference heroes a lot of the things we do I think are chosen because this is what I'm attracted to are things with really really incredible emotional resonance like a heroes even the effo waters piece these pieces critics may not always like bio musicals but there's some stories that are really really important to be told and some amazing people to be told about and they do have incredible emotional resonance and real inspector hand didn't and actually the emotional resonance I imagine that that's a factor here so this is looking at the three shows across ticket type again and if you look there you've got this great bump for sparrow and I know that you were specifically interested in getting a different set of audience members in this case African American to come see that show that it was a different type of show I those the amount of people who reflected privately on the work having jumped so much there I'm wondering if you have and also you've got a very large number of single ticket buyers in relation to your subscribers is that what is your subscription like and what was the difference in the audiences that you think might have encouraged this response well we have a very small subscription base to begin with with our musicals our winter musicals we get by house buyouts we get groups it's a very significant jump in their single ticket buyers but many many many of those single ticket buyers are actually coming as groups but they are not subscribers so the single ticket buyer for a real inspector hound is very different from the single ticket buyer for sparrow great so we're going to move on to Chad but do you have any last bits and then we'll come back to you to kind of talk about more general feelings about the work well I guess all I can say right now is that this is so fascinating and for theater like metro stage which is so small and so understaffed this is so inspiring to me to see this kind of these results because it really does tell me what I need to be doing in the future and I think when you read the essays by the other artistic directors you'll see some really interesting continuing themes across all artistic directors and I think this data maybe we can be more specific later very tells me exactly what I need to be doing next wow that's great thank you Chan could you give us you know we covered a little of this already just a little background in your three pieces sure when we were actually asked to select plays for this survey we were opening this building in the same season and immediately I thought well we can't do the first two shows because I'm still going to be trying to figure out how this building operates but we tried to look over a scope of what it is that we do and we have several pillars we have a presentation pillar we have a production pillar we do musicals we do non musicals we do world premieres we do non world premieres and in these three specific shows we tried to capture all of that in hindsight I know in the past couple weeks there's been a lot of discussion about the roles musicals play in the theater community and I wish we would have put Oklahoma in but we didn't so on these three shows let me down easy is Anna DeVere Smith's show about healthcare it started in a second stage this was our first entry into actually launching a national tour so we knew that we were going to be basically presenting work from second state but then sending it out into the world we knew that it had a very strong sociopolitical message similar to probably what Mike Daisy was trying to do but this was a pretty much a presentation and it was in again we have three separate spaces this was in our proscenium space so we actually wanted to look in what happens and impacts between our three theaters as well the second production there was a tour of Ruin going on and I believe that the Berkeley Rep was the tour our production wasn't the tour because we chose to set it in the round and because there's only maybe five theaters in the world that are in the round I know that's an understatement but there's very few in the round spaces it wasn't a tour it was a production and it involves again a sociopolitical message about some themes that are in the Congo and it came to us from a very good New York pedigree and then not just work is very well known so it was one of these senses that we were doing our own work but again it came to us with a pedigree the third and final production was one of these things that it was a world premiere it was in the Krieger it was the first time John Grisham's novel had been adapted but it was a theatrical adaptation we knew at the time that we didn't exactly know what we were going to get because the story is very powerful but again it was part of a greater well-known work and so I wanted to throw something in about specifically since we were asking staff what we think was going to happen versus what actually happened with our audiences on new work that span can be pretty significant because while you're creating new work you never know what you're going to get and so that's why we threw that in great so first thing motivations and we looked at some motivations earlier but these were the motivations reported by your ticket buyers for the three productions there were really some interesting differences okay so to discover something new was much higher for ruined right to revisit a familiar work for artists was three was twice as high for a time to kill right so presumably people were familiar with the source work to see the work of a specific actor director or artist was highest obviously for Annette Beer Smith and then to be emotionally moved actually was significantly higher for ruined as a motivation for going now we are asking about this afterwards so it's an interesting idea you know even though we're asking people retrospectively about their motivations going in you know if there's some coloration of this as a result of having seen the work already so I guess the question to you is motivation on this one more to relax or as a scape actually you know less we forget was highest for a time to kill if this you know as the marketer you know if you were thinking about kind of the very different complexion of these works in terms of motivation yeah and I'll respond from my personal belief here because there's a lot of discussion in the blogosphere world about mission-based work I actually think that large regional theaters have a responsibility to present a balanced meal approach there are people that want escapism there are people that want to be challenged there are people that want to be the various motivations that you covered to me from a marketing perspective I'm really happy with these numbers why because we presented a really balanced approach of course people that are coming to ruin they're going to become they want to see something new they want to be emotionally challenged people that are coming for let me down easy wanted to hear more about probably healthcare they wanted but they also came because they were interested in unfairness work it's no surprise to me that a time to kill people wanted to come and relax and escape why it's a best-selling novel so to me this this is a strength of the company and did your marketing materials your consciously or not reflect these motivations do you feel yeah I think you made a really good point earlier in the day about how as marketers we set up specific expectations in fact I was talking yesterday to Peter at the Washington Post about how you can do a musical but if you're doing a musical and Peter Sellers is directing that it's a completely different thing and so my job is to set up expectations and so here we pretty much guessed well on what these were going to be and the way we marketed those shows were marketed to those specific motivations that said we don't always do that perfectly particularly with new work because it's hard it's evolving great did you do anything to prepare about a third of let me down easy patrons said they had prepared in advance which was really quite a bit higher than the patrons for time to kill you asked people what did you do to prepare and these are some examples how does your patrons' reliance on criticism in the post affect your communication strategy Nelson you can't watch this oh sorry do we have the critic with us we have civil critics actually not I'm sorry you can take a pass on this I mean criticism as you know is a big issue in the theater feel particularly in markets where there is none and you know our research on audience shows there's a segment of ticket buyers who basically won't go unless they read something saying you can't miss this production and it's a big challenge in the field and generally you don't have that problem here you have high quality criticism and actually it really came through in your data as a form of preparation I mean I think the best theater cities in the world are actually significantly supported by the best criticism in the world I think the DC metro area has phenomenal criticism I think New York has phenomenal criticism I read Chris Jones' work in Chicago and I read Frank Rich's work in Hartford I've never been to Hartford but I read Frank's work so you know theaters love criticism when they're stroking you and theaters hate criticism and they're criticizing you that's just the nature of the business but God bless the fact that people are writing about us and this does not surprise me at all particularly as what does surprise me is that the 44 plus contingent I've reviewed and the 44 and younger contingent pretty much didn't that in itself is not surprising but it means that we're serving two separate distinct groups and there is one group that we're going to get with probably professional criticism there's another group that we're going to get with word of mouth strategies and probably what I consider pro-am criticism is like the upside so just to promote the book a little bit we did ask at the last minute we included a question on the survey like did you read a review by a professional critic or a preview because there was a hypothesis that people who'd read a bad review were going to report lower impacts and conversely people would read a positive review but we didn't code in our data whether the review is positive or negative but there is information in the book about the relationship between having read a review or preview and the effect on anticipation so I'll just encourage you alright Chad, one last thing staff results versus audience results for did anything about the performance offend you or make you uncomfortable and what you see here is the staff generally some people for every one of your productions anticipated a lot more people would be offended than were by a point or more honestly on average and I guess the really the question here is kind of are your audiences more resilient to challenging content than you think they are well obviously I think I think it goes to to Jeff's point I think this is something that marketing folks have to be aware of I think that we think that our audiences scare more than they actually do and it can lead to cautious marketing when in fact actually that our audiences are looking for are things that we're trying to steer away from in messaging because they could be looking for these things they're looking to be challenged but yet we're scared of challenging them in some senses because it could turn away a ticket buyer but for those people that are looking for that then you turn them away by not messaging to them and actually this was pretty common throughout I remember us thinking in our data that part of what marketing people do is guess on a reaction of their communities what this shows is I'm really bad at that and not only am I really bad at that our entire staff is bad or that you're just sensitive to the people who complain and that's not necessarily representative of the total audience can I ask a question yes because I'm a little bit confused by one aspect of this isn't a fence almost like a pre-selective quality it's whether you see a show or not like if a show is effectively marketed and I know that I'm probably not going to like it it's going to piss me off and I don't have any money to see it so isn't this a little bit skewed like in terms of people who already have elected to see the show by and large are going to be willing to go along with the content sure so is there self-selection going on before so like I'm wondering how much of that skews the actual numbers of that's a good question I think obviously there is some self-selection going on and honestly to be perfectly candid with you this question is problematic because people can be offended at the injustices represented in the play of women injustices against labor you know or people can be offended at nudity or offended in vulgar language very different things to be offended very different things and we blended them together here and in the future I think we can't do that so we learned alright I'd like to kind of open this up a little bit now and have just maybe 10 minutes of conversation with you all and the panelists here about kind of taking stock of this work and I put together just a few questions here ask you to reflect on this the first one is about qualitative versus quantitative data if you you know we'll start with you Jeff if you have any kind of sense does the value of the qualitative data versus the quantitative data I did find the qualitative data much more interesting because all we ever look at is quantitative data I'm gonna take a look at quantitative data and so this was really some fresh data points for us and you know one of the values of William Adams is to ignite an explosive engagement between artists and the community and so the way this whole survey in many ways played into the things that are important in our mission statement that we didn't really have metrics before and so that was the stuff I was most fixated on and most interested in and I think it's told us sort of the most about where we might want to go moving forward interesting other comments what I think what I found most interesting is how much the qualitative and the quantitative meshed and supported each other and when you get to the back where you read the artistic director essays and the one example I want to give because it's so supported by the quantitative data that having the anticipation by having done the research and made the decision and gotten into it a little bit before you even come to the theater that and then the post show opportunities for discussion all of which enhance the ultimate impact so many of the examples in this book have artistic directors talking about post show discussions and things that really are something that any theater can incorporate they do a post show discussion after every performance I mean there's something kind of amazing about that expanding the experience beyond what they see on the stage and giving people the opportunity to do some interviews you did some interviews didn't you my assistant did I don't have direct information about that but what I will say because I think this is so relevant for me with the shows I have coming up I have a show coming up in a week it's a one man show about his family escaping Cuba in 1964 so we will definitely have more post show discussions than we've ever programmed for that and then the show after that is Lonely Planet which is a classic AIDS play and we will incorporate some post show discussions and we haven't traditionally done that and it's really an inspiration for me and motivation for me to really get it together and do more of that particularly with the kinds of plays we have coming up in the next two shows so the timing is perfect and I can really incorporate what you found and I'll let you know how it works any of you did you get any positive or negative feedback from audience members on the survey on the actual survey itself the most interesting thing that I actually found on the qualitative feedback that we got on the survey was we spend a lot of time at arena stage trying to prep audiences we do blogs and videos and we have this great virtual drama thing where we write a whole bunch of articles and it's a lot of time and energy and I know that David our associate director was really interested in if that was for me people we actually got feedback from a fair amount of people that said listen you could have the best articles and the best preparation and I just don't care and the reason why I didn't think about this the reason why is that they this subsection of people want the work to stand on its own completely on its own and so I wanted us to be mindful of that's not a failure for us that there is a good subsection of people that want that crap and they want to discussion they want it and then there's another subsection that just don't this weekend I went to signature theater in New York and saw the lady from Dubuque and I left Sobbing it was a beautiful production I wanted to talk about a lot but when I'm Sobbing I'm not going to talk to anybody and that's perfectly fine so I thought that was interesting feedback did you have any feedback from audiences I'm going to have to defer Millie Harker our connectivity assistant is here who is really involved in pulling a lot of the stuff together and you were sort of nodding your head to me I think what was what was interesting in terms of the feedback from the survey is that we really recognize that the survey was super helpful and that we wanted to take some of the questions and extrapolate them and put them into a sort of miniature version of the survey which we continued and are continuing to gather more data this season what we found is that the wordsmithing of very specific ways that you put together the questions when we surveyed in the spring is so important because a lot of folks kind of shut down and responded on our open-ended parts how I can't even answer this question because I don't really care what it means I don't care what you're actually asking me which was harmful but also a good learning moment because we needed to recognize how can we frame these questions for these people who want to reach out and give us the feedback so how can we how can we get the answers that we're looking for but also sort of probe these people and allow for them to feel not diminished by answering our questions that's great I'll say one thing that we did that we did not do that we should have done we have to deal with in the future is allowing audience members the instant they finish a survey to see some results and get some context on their own answers and really kind of fulfilling the contract and so we have to figure out how to do that in a way that is both transparent and also doesn't disclose too much too much information so that's something we're going to work with because when you articulate your feelings about an artistic work and it just goes into a black hole at least we can do in some newsletter or something thank people and perhaps give them three or four bullet points of something you learn because if we're going to keep surveying people over and over again there has to be accountability and I think otherwise we'll see the sort of fatigue the survey fatigue that some of you already are seeing any others of you want to address any of these questions please take the mic that last question about will impact data ever inform programming choices I think that's a fascinating question and having just been reading what artistic directors have written almost consistently they say we program we make choices because these are things we believe we want to present we need to present audiences need to see the actual impact of it sometimes it works sometimes it doesn't there was no questioning as to actually the choice it's just sometimes in anything in any enterprise sometimes it works and sometimes it doesn't sometimes people get it they don't but what this does do is tell us how better to prepare prepare I guess inform is a dangerous word in advance of as we're looking at how anyway there's a lot we can do I think based on this these results to really enhance the experience and maybe make those things that only you and the director apparently understand and the critics didn't and the audience maybe doesn't maybe there are things coming from this that suggest things that can be done so I think it's extremely valuable but I don't think we're not out there doing surveys of what people want to see and never will thank you that's a great distinction those of you who are here are there any of these questions you'd like to speak to yes can you just clarify a little by the would you ever see intrinsic impact as a donor program oh sure well I kind of have a theory that there's a group of donors who would be very interested and good at providing feedback on a regular basis and might pay or increase their donations if they were engaged in that way so that's just a theory yes aside from learning about my future for unknown deep love of word charts I thought that that was amazing because I think for me I'd love to be able to pick a word or two that I want people to leave and figure out literally how big that word left an impression because for me that's the visceral component rather than the intellectual and I almost would rather have people leaving our shows with a really visceral feeling and I know which one or two words I think I almost always want that to be so here's where we're going with this and several of us have been chatting about this is trying to actually allow audience members right after performance to walk out in the lobby and on their telephone text the words they're feeling and have a dynamic word cloud in real time forming as basically crowdsourced emotional map of what people just experienced as a form of audience engagement we're just looking for a site or two to test that could be really fun other questions yes I would just come back to your donor program and we might consider the idea that we're already doing that in a sense on a certain level certainly from a as a development staffer the people who have the loudest voice in giving you feedback are your board members who are typically your biggest donors and your highest level donor programs and it's a tricky thing to try to promise that that voice to them in the context of a donor program and creating a basically a pay for play scenario sure but it's something that this context of thinking about these ideas could be really useful when you're talking about building donor programs that aren't just based on dollar values but when you start looking at young donors programs or specific special interest donor programs some of those ideas into that I think you could just as easily build a cadre of younger people who I might call citizen critics to as another group that would perhaps provide a different perspective than your donors but I agree there's a slippery slope there that you've so beautifully articulated yes it's exalted you need time so I'm really wondering if you're looking at long-term what goes into my memory day to day what becomes important to me what do I talk to people about what themes come out of it that I share or that impacts my life it's not immediate it's not what happens in the long period of time what happens the next morning the more I react to it so the question was basically that there's a comment basically that your reactions to theater change over time and can change it from the moment you leave the theater to the next morning to the next day to the six months from now a lot of the work that theater bay area is trying to kind of move this or at least one of the directions we're moving with intrinsic impact is to understand its connections and memory and its connection to the creation of specific types of memories you know I believe strongly that memory is one of the main underpinnings of future attendance remembering pleasurable or impactful experiences is the reason that you go back and do it again and so we are fascinated by that question about the changing perception of a show over time in this case we ask people to fill out the survey within 24 hours so some of them fill it out immediately afterwards some of them fill it out the next day and honestly some of them fill it out a week later and it's we can't tell when they fill it out so that is absolutely something that we need to get more specific about now I will say that by shifting to the email based system which is what we're doing now there's the possibility of actually time-stamping responses now we can't guarantee that that person went that night and so when they respond at 11.30 that night it's because they just saw the show there's a high likelihood of that and it would be fascinating to see the evolution of emotional reaction over the two to three days after a show and that's a great direction to go in it will also be interesting to see the response rate on the email stuff because even though it's quicker in some ways you also have people who will just forget to do it versus the survey in their hands well you know we piloted this the email surveying a little bit the response rate seems fine but as Alan pointed out earlier it's a subset of your audience it's not perfect that's the trade-off for not having to spend lots and lots of money doing it and it's also something that you can get around to a certain degree by asking people to forward the survey to the people they came with that decreases the loyalty bias or it decreases the decision maker bias a little bit but you're absolutely right it's a certain subset of people and it's the people who remember to do it it's the people who know how to use computers it's the people who rate their email there's a lot of caveats but they exist for all this type of stuff we should probably thank these guys thank you so much for being on this panel we are going to jump on to the conclusion because we're coming to the end of our time thank you guys you can go ahead and head off stage and so Alan why don't you lead the conversation great and this will be very brief really why engage audiences critical feedback we ask this question a lot I think the response rate itself says something that there are plenty of people who are very happy and perhaps even increasingly expect to be able to provide feedback and we think of feedback as a milestone in the audience engagement cycle okay we did some research last year for the San Francisco Foundation and wrote a publication called Making Sense of Audience Engagement which you can download from the homepage of my website here and really just try to look holistically at this whole business of engagement before afterwards and the chart is pretty much self-explanatory but the audience engagement cycle begins with marketing really it is contextualization there's often the only context people have going into a live performance which is why it's so important and then there's at some point people make a decision to attend whether they themselves decide or they got invited and then there's this opportunity to contextualize the work before the lights go down and most of that happens sort of the day before and over the hours before the lights go down where people are frantically paging through their program book trying to get that paragraph synopsis before the lights go down which is actually a story about keeping the lights up a little longer the exchange itself and all the things that can be done or might be done of adding layers of interpretive assistance on top of an arts experience it's very controversial antithetical for many people a fascinating discussion a few months ago in Toronto with a woman who trains the people who do audio descriptions for the blind and she was listening and I was saying what training do you give them it was really amazing and it just occurred to me why do you need to be blind to get an audio description there are probably people who would love to have the director whispering in their ear through a headset about what was about to happen or why someone moved upstage or downstage or some interesting aspect of the production sort of like the director's cut on a DVD it's the inside story of course that interferes with the authentic art experience it's artificially injecting content but I would argue some people would love that and might even pay extra for it so there's a rich area here about delivering interpretive assistance starting with super titles into arts experiences then the experience ends and then this whole meaning making process kicks in and some people as we've talked about today want to dive in and argue about it here's what it meant to me and other people I love the people at post discussions who sit in the back they're not ready to articulate, verbalize their own feelings but they really want to hear what other people have to say and their wheels are spinning and they're saying well that's not what I felt or there it is what I felt they're just not ready to verbalize it yet the feedback actually probably often happens right afterwards it might be moved up but it often happens on a kind of an ongoing basis and as Clay said we'd love to do a study people six months later a year later and understand how the impact has been packaged as a memory and what is remembered because you all remember arts experiences you had decades ago as if it were yesterday and they're still paying impact dividends they're still transforming your life and that's the amazing thing and then that feedback really actually is an input into the institution's thinking and as our panelists though articulately said it's not about asking audiences what they want it's understanding the impact of your artistic choices really gaining the sense of if I make this artistic choice it's likely to have this impact if I make a different artistic choice and really becoming curators of impact not just art I think you all do that instinctively already and this is perhaps just articulating it a little more so I just that's really my summary I just encourage you to think of audience feedback as not just taking data from your audience but as an investment in your audience's aesthetic development when people fill out a survey with thoughtful questions you're validating their opinion and you're in way teaching them how to have a reaction to a work of art which in the long run I believe is an investment in the audience so Clay, do you want to talk about practical applications? yeah, so one of the things that's come up a lot on this tour and that actually was a question that came up a little earlier today is that is fantastic but what do you do with it and especially when it's something like this that's talking about something that is essentially kind of ephemeral and very individualized and so we've been thinking a lot about sharing a lot of anecdotes from people about how they are actually taking small steps around their programming around their pre and post engagement activities around the way that they're engaging as a staff to actually practically apply this information so the most obvious one is to check impact against goals the fact that we started asking staff members to give their projected results for audience members has been really transformative to this research because it gives people a conversation starter and for example you can now in the dashboard you can actually divide it up by departments to see which departments are actually jiving most closely with your audience members this not as a way of kind of penalizing departments that don't seem to be getting it but as a way of engaging in a conversation about so why is it that the development people are really understanding I mean some of this can be obvious but why is it that the development people are really understanding kind of the high level donor responses why is it that you know this marketing staff is really getting this particular aspect of the audience the other thing and this is something that has happened because we've now transitioned to this email based online system is that the impact results are almost immediate they're available usually within 24 to 48 hours and soon will be quicker than that as the automatic link between the survey and the database is finished you know these companies that worked so hard distributing these surveys then had to wait two to three months to get their results and of course at that point they're great as data but they're not very actionable because the show is over and you've moved on to something else and you don't have a lot of time to spend thinking about how that could have been better creating targeted free and post engagement when you have access to results within 24 hours is something that is entirely possible you can create something that addresses the questions that seem to be tackling particular impacts and you can also create new pieces afterwards that for example are built out of questions that audience members themselves have had at South Coast Rep they have started doing question treats where they actually put questions up in the audience after the show in the lobby and they encourage people to congregate around them as kind of an informal talk back there's no expert curation they simply talk to strangers and kind of address those questions about it the other thing that's interesting and an interesting possibility is this idea of profiling a show so if you've got a premiere show you've got a show that's never been done before and you do this work and it's going on to another place there's a possibility of being able to create an impact profile sheet that passes on to that next next place so that they can know these are the type of questions you're probably going to get this is the type of impact you're probably going to see turns out there's a really high social engagement component in this so you should really be considering post-show social engagement activities like wine tastings or opportunities to talk or in the other direction this is a really intellectually demanding show so you need to make sure that your pharmaceutical materials are top notch because a lot of people will be using them also engaging departments and board members we've talked about this a lot but there is a kind of in the interviews with artistic directors and also in anecdotal conversations with people throughout this work in a lot of companies there's a kind of fundamental disconnect between marketing staff and artistic staff about engagement and that disconnect is more due to the fact that there's just different languages that have been spoken in those two departments for a very long time than it is to do with any particular animosity between them and so if you can get to a point where you can have some common ground and I'll tell you it's fascinating building these surveys with all these companies because we required an artistic staffer and a marketing staffer haggling it out over 21 questions out of 60 that would fit on 3 pages and when you get into that kind of conversation you start hearing things like well I don't care about that, well I do well that's a conversation starter right there and if you've got results on it then you can actually engage in a conversation as an organization about the goals, mission and potential outcomes that are unrelated to actual economics and then finally reporting to your staff, your board, your funders, your government officials why don't you go ahead Matt? So as we're moving on we're talking about engaging the field at large, this is something we're doing over the course of the next year and over the course of many years Alan is doing a bunch of different initiatives in many different countries including one with the NEA that I'm sure you can ask him about afterwards but in our goal our goal was to make more people able to have this conversation and the first thing you have to do when you're doing that is to get the amount of money it costs and so we have done that using Foundation Funds we've created this online dashboard and the Doris Duke Foundation which has at this point given four separate grants to support this work which has just been phenomenal their most recent grant included a component to subsidize 30 arts organizations who are interested in doing this work it subsidizes it not completely but it drastically reduces the cost to something even more manageable than the cost we've managed to get it down to anyway if you're interested in that then please email me at clayattheaterbayarea.org it doesn't matter if you're a theater or another arts organization please email me we are kind of the databases open and running and we encourage everyone who wants to try this out whether you end up getting the subsidy or not it's very affordable to most people so I just wanted to really quickly close by bringing it back to max because while all of this data is really exciting at the end of the day it's kind of heavy for me what all of this research has come down to and this has been two years and it's been a very surprising and engaging and enlightening journey I've heard so many amazing stories but what it all comes down to is the artistic experiences we have over a life and for me that started with max but somewhere along the way my artistic experience has moved me from wanting to be a lawyer to wanting to work at an arts service organization and as a director of communications and research around the actual impact of art on individuals over time but now we've got this amazing resource that has been compiled from some of the best minds in the country artistic leaders, executive leaders patrons, researchers thinkers of all types who are all trying to get at this question of how to increase the stickiness of arts experiences that constantly make up a life and so impact leads to memory which leads to return which leads to impact which leads to memory which leads to return and that's the thing you need to remember is that this isn't esoteric increasingly impact of the art you have increases the likelihood that people will come back and it increases the stickiness of the memories the specialness of the memories that they're having with your artistic experiences it makes them better advocates it makes them better attendees often spend money with you and you bring it right back around to economics again and so we're really proud of this work and we're really proud of what it can do we can know more about the power of what we do we can know more about the ability of art to transform the people who see it we can prove and improve our impact and make stronger, stickier memories for the people who come see it we can better explain our relevance to people who doubt our relevance every day we can bridge at anecdote and numbers and we can actually have a conversation where everyone's understanding everything and we can measure what we've always thought was kind of unmeasurable and that's why we're really proud of you who have done this work and that we hope you enjoy this presentation thank you and just so we're aware it is exactly one o'clock