 Here are Kato Katoa now my Harimai welcome everyone to the very first Australian Evaluation Society seminar for 2023 We are very lucky to have Professor Janice Tappinson here today introducing using the energy cultures framework to guide evaluation Interventions and practice and as you can tell right now this seminar is recorded So if you don't want to be on YouTube feel free to turn off your cameras This will be uploaded in about two weeks We can find most of our seminars at the AES AES website on YouTube I'm quite happy to send you the link along with the slides that Janet shares today And before we begin I'd like to acknowledge the traditional custodians of the lands which we all come from I come from Pona Cape, Wellington, Aotearoa, and I acknowledge the leaders past present and emerging So there we are so fortunate to hear from Professor Janice Tappinson about her work related to energy-related practices Janet is a research professor at the Center for Sustainability University of Otago Which she undertook the role of director for over a decade The Center of Sustainability is an interdisciplinary research center that specializes in research related environmental issues locally and globally Janice has a background in sociology planning and cultural geography and she has led multiple research programs too many to mention here And it's widely published and serves on several energy-related advisory groups in Aotearoa and internationally Janet, thank you so much for being here today. It is lovely to have you and we are all looking forward to learning from you for Anyone who is participating feel free to Send any questions in the chat function. I will monitor it But we will also have about 10 to 15 minutes at the end that you can Unmute yourself and ask Janet anything you want Towards the end of the seminar. So thank you and over to you Janet Kia ora koutou and thank you Marie and Nga Hinui, Kia koutou. It's really lovely to be here today with you and To share just hopefully a few insights from the work that we've undertaken and that's been Adopted quite widely internationally that that hopefully will help you as evaluators think about When we're measuring the effectiveness of an intervention, how do we decide what to measure what to evaluate and In this talk, I'm going to introduce what we now call the cultures framework moving on from the energy cultures framework Because it's very applicable to many topics, not just energy So applying the cultures framework as an analytical lens for assisting with the design of evaluations So I'm going to share my screen. Hopefully this is all going to work. Okay, and We can go from there. So Marie, is that visible to everybody? That is perfect Okay, so What I'm going to talk about is in five parts. So I'm going to introduce culture and then talk about First we're introducing the concept of culture and then talk about Investigating culture in relation to outcomes I'm going to introduce the cultures framework which links culture to outcomes Then I'm going to talk about policy and other interventions as being an attempt to change cultures And then I'm finally going to talk about the evaluation of interventions using the cultures framework as a framing for those evaluations But first of all, I just want to pay pay homage to the the energy cultures team who I worked with from 2009 to 2016 as you can see it was an incredibly varied Disciplinary team The the one is in the left hand photograph are the the initial team and then in a further project a whole lot of other people got added so in a sense Although it was developed 14 years ago It was at that stage it was to support integrated research on energy related behaviors But it has since been applied internationally in research and policy across a wide range of fields Not just energy, but many other topics as well as as you'll see And it's been used as a research framework in over a hundred applications Around the world that have been published in academic journals and theses and maybe other applications. I'm not aware of And what it is is particularly useful for unpeaking why change can be so hard for households businesses and organizations And also how my change when change occurs why why it does And my studies using the framework have used it to gain policy insights But in in light of an evaluation a number of studies have used it to examine the success or otherwise of Policy or other interventions which purposefully set out to achieve different outcomes So this was a holistic approach which started as you can see this frame here as an attempt to provide an Integrating framework across behavioral theories spanning psychology management sociology anthropology consumer psychology behavioral economics and others and it still works in that sort of Multidisciplinary integrating sense, but more importantly it works analytically as a framework in its own right a framework That's open for use with multiple theories and multiple methods of inquiry So we started writing about it in 2010 in this paper here in energy policy And and since then there've been numerous papers and and reports that have used the energy cultures framework And this was is one of the more recent ones Which is moving it away from just focusing on energy to looking at the relationship between cultural attributes and sustainable Sustainability outcome. So if you're interested in any backup Papers or reading there's a lot out there to access So let's try and climb into this talk. What do we mean by culture? Well, when we look at the social realm, we usually think of it Both in terms of the fact that we all have some generic human characteristics. We need food. We need warmth. We need caring relationships On the other hand, we're all unique individuals. We have unique genetics. We have unique personalities But but in between those two extremes of the of the collective and the individual We actually share many Characteristics that are similar to others in our families and our social groups in our workplaces in our ethnic groups within our countries And so forth and we call this culture so we often think about culture as being Well, actually when I ask people what culture means they have lots of different answers But generally people say well, doesn't it mean we kind of share similar values or beliefs or Have similar ideas about things And maybe we carry out similar kinds of Practices And then they kind of get lost about there I'm going to give you a little bit more structure around that So it is definitely about the ways that people think it is also about what we do and it is also About what we have and Importantly it is about how those things are interlinked and feed off and support each other And we call this the cultural ensemble and As academics we like to give things fancy names. So instead of have think and do we talk about materiality Motivators and activities So Specifically cultural features may include many things like in terms of materiality products, tools, technologies, infrastructure, artifacts, the food we eat and so forth In terms of activities it can include Direct actions, doing, making, using, acquiring, behaving, performing and routines, habits, practices, rituals and repeated behaviors In terms of motivators things that we share with others that actually affect how and why we carry out activities and what kind of material items we undertake And use things like norms, values, beliefs, knowledge and symbolism and meanings are all Important motivators in the decisions that we make and what we do with our lives, which may be visible to us We may be aware of them or we may be not consciously aware of these within our thought processes So when we talk about culture, we talk about cultural ensembles which are interrelated cultural features So the way that these things relate to each other and here's an example a Phone, a mobile phone, a relatively new item of materiality in human society So a technology that's associated with a number of motivators so it has become entirely normalized within particular societies Built around expectations, rules of use, aspirations, it's associated with a whole lot of symbolism and meanings So it indicates certain things about who we are and particularly what kind of phone we have might have a lot of messages hidden behind it It's involved with forms of knowledge and understandings about how to use it and how the use of it might affect other people For example, in relation to social media A number of activities are also associated with it in terms of acquiring, using and the sorts of routines or habits we have in terms of that use And the fact that we have an iPhone may well lead to new aspirations for other technologies that are associated with iPhones such as here, the speaker So here we are, this is just a picture of what I mean by when we talk about a cultural ensemble So my definition of culture is that it is distinctive patterns of motivators, of activities and materiality that form dynamic ensembles which are shared by a group of people and learned through both cognitive and bubbling processes So these features of cultural ensemble may be consciously shared For example, teenagers being incredibly conscious about the clothes that they wear or the hairstyles that they've got Or they may be unconsciously shared, we may not even be particularly aware that we are acting culturally, maybe until we go somewhere else where there is a different culture And then it'll suddenly hit us that our cultural, our expectations, our practices, our beliefs, our understandings are quite different to others And by that, by dynamic, I mean that these cultural features relate to and influence each other And we can see varied cultural ensembles across any given society And I'm going to dig a bit more deeply into that as we go through this talk But what culture is, what those ensembles consist of depends on what your particular focus is And where we're going with this talk is taking an outcomes focus on culture So what are the cultural ensembles that give rise to particular outcomes? And this symbol in the middle here is one I use to remind us that when we're talking about outcomes, we may be talking about environmental outcomes We may be talking about social and health outcomes and we may be talking about economic outcomes or others Example here of varied cultural ensembles across a population This is some findings from a research project that was part of the energy cultures research Where a national household survey identified four types of household ensembles that relate to different levels of energy efficiency One we call energy economical who had absolutely appalling material culture, really inefficient houses, street full heating systems But very, very efficient energy practices so trying to save as much money as possible Energy efficient who had very efficient material culture, very efficient energy practices Energy extravagant people who had not particularly good material culture and not particularly good energy practices But often were very high users of energy and energy easy who probably had Were more laid back about their practices but had very efficient material culture. Each of those had quite different outcomes in terms of efficiency And there have been other studies as well that have actually I looked across populations to identify similar patterns So when we look at culture in relation to outcomes it gives us a different perspective on culture to what you might typically think about as culture in your day to day lives So as an evaluator, if we take an outcomes focus, what is it that you're seeking to measure? Is it water quality in streams? Is it asthma rates amongst children? Is it greenhouse gas emissions from transport? Is it rates of energy poverty? Etc So from an outcomes focus looking at culture, the question then is who are the outcome actors that this particular intervention has sought to influence This is assuming that there is an intervention that has sought to improve one of these measures So is it farmers? Is it households? Is it commuters? Is it landlords? And then the question becomes what are the cultural ensembles? What are the combinations of materiality motivators and activities that are giving rise to or are associated with those outcomes? So what cultural features in particular are relevant to those outcomes Example here of different heating cultures, here's a kind of a typical student flat in Dunedin Probably appalling heating systems because their landlords can't be bothered to give them a heat pump Wrapping themselves up in blankets and hoties in order to keep warm and not particularly worried about the temperature of their house because they've got other things to spend their money and time doing And they probably have a high rate of chest infections In fact they do Whereas here's a different heating culture, a household with the heat pump A household that has curtains and draws them at night to keep the warmth in and carries out other activities And has an expectation of being warm and that is associated with a whole lot of meanings and values in their lives And probably having a much lower level of chest infections So that's just a simple example of showing you how I'm talking about the relationship between cultural ensembles and the outcomes from those So the starting point for any evaluation is really to say what features of cultural ensembles are associated with the outcome that I'm interested in Or maybe which cultural ensembles across this population give better or worse outcomes I'm putting those in quote marks because the better or worse is probably from the position of assuming there has been a policy intervention to change some particular outcomes So just picking up now on the cultures framework which moves from this idea of culture to give it a little bit more complexity in order to help us unpick these relationships between culture and outcomes So the cultures framework is an actor centered framework by which I mean it can apply to any actor or group of actors at any scale It can apply to an individual or a group of individuals We can apply it to a family or a household or a group of families We can apply it to small businesses, we can apply it to major businesses or collectives of businesses or sectors or government agencies So the framework as a whole, those sets of key ideas here are applicable at any scale The next element I want to introduce is this idea of agency, so having choice and having the ability to act on that choice So this dotted line around the outside of these items sees the ones that we are particularly interested in when we are evaluating is what is within the actor's realm of influence How extensive is the agency, can they change these things or can they not And these agencies constraints may be things like income, education, power, relative power compared to others, the degree of wellness that that person or that family has And agency can be reduced by the actor's situation, for example tenants have very little agency over the materiality of their lives, their homes And often the heating systems are determined by the choices that their landlord makes not themselves And this constrains the kinds of activities that they can carry out and constrain some elements about how they can think about their situation The second idea I want to introduce here is that of external influences So beyond the cultural ensemble, beyond the actor's realm of influence, there are a whole lot of things out there, the broader context which is influencing their cultural ensemble and influencing the degree of agency they have And some of those external influences support the status quo, they are tending to hold that pattern of cultural ensemble in one place and some external influences are driving change and we tend to be interested in both of those There are other influences that are probably not influential particularly so we want to try and pinpoint what these influences are that are holding this particular cultural ensemble in place So we want to try and pinpoint what these influences are that are preventing change or those ones that are tending to drive change And just as an example, if we look at commuting culture, the fact that many people still rely on their private cars to get to and from work, what's holding that in place, those patterns of material investment motivators and activities So one is continued investment in motorways, another is car advertising, another is policies on urban form and I'm sure you could come up with a whole lot more as well The final element of the cultures framework is about outcomes So when we look at outcomes, it is what are the outcomes of this combination of the cultural ensembles that people have into playing with their degree of agency and into playing with those external influences This reminds us that there isn't for any intervention for change, there's never going to be a direct line of dominoes between that intervention and the outcome you're seeking People aren't rational actors, they're not just motivated by money or values, their ability to change one aspect of their lives, for example, to stop using their car and start walking is constrained by the existence of other existing material items, other activities, their values, beliefs and norms and also the limits to what they can realistically change And those cultural ensembles, as I said, are also supported by these external influences beyond their control, all of these things are entangled and influential So rather than a line of dominoes from an intervention to an outcome that you can evaluate, we have a system which is entangled and complex to change What the cultures framework can help us do is unpick that system So what we see policy as is, or other interventions, I'm positing here that when we try and intervene in a situation to get a different outcome, we can see it as an attempt to change elements of culture So interventions aim to change one or more of these things in terms of materiality, in terms of activities or in terms of motivators at one or more scales So here's an example of policy to change outcomes, here comes a policy intervention It seeks to change possibly an external influence, maybe alternatively, it seeks to change the degree of agency that people have Maybe it seeks to achieve a material change within the group of actors Maybe it seeks to change a motivator and a lot of policy interventions are all about improving knowledge, so it is about information campaigns Or it may be directly to try and adjust an activity, for example, a nudge campaign And ultimately, there's an attempt to achieve a change in an outcome An example just to bring that to life, improving home energy performance So we might want to improve home energy performance in order to get a number of outcomes and increase in indoor temperatures, a reduction in humidity and less doctors visits, less sickness in the family and maybe a number of other measurable outcomes as well So policy intervention in terms of external influences might introduce minimum performance levels for rental housing It might seek to improve agency by offering heating and insulation subsidies It might seek to improve materiality by direct action in terms of adding insulation to homes or putting in heating system upgrades as a number of programs within this country are all being doing It might seek to change motivators by running information campaigns to improve understanding about heat retention and the causes of dampness Or it might seek to normalise the expectations that people should have warm dry homes Or it might seek to adjust people's activities so that they are better at retaining warmth in their homes, for example, or in curtains at night or ventilating spaces in order to reduce the number of homes So if we look at policy interventions in that sense, in relation to outcomes, they can work at one or several of those elements of the cultures framework A couple of examples here from some work that one of our PhD students did looking at energy poverty using the energy cultures framework And here are a couple of different families in relation to this energy poverty study Norma, a pseudonym who had a partner in two children She was in a rental accommodation, had limited income and obviously limited agency Her house looked lost to the sun early, it was broken down, she was in a rental accommodation, she was in a rental accommodation, she was in a rental accommodation, she had limited income and obviously limited agency Her house looked lost to the sun early, it was poorly built, it lacked insulation and she had a portable electric heater in terms of materiality In terms of her motivators, she was really careful about energy spending, she knew how to be frugal and efficient and she just wanted a warmer, healthier house And in terms of activities, she carried out everything she possibly could in order to reduce energy use and keep as warm as they could within those limitations In terms of that particular family, the ideal intervention would be to make material changes to the house If we then look at a different family, Eric, a homeowner, a single father with a four year old child who kept getting sick He had insulated what he could but he couldn't do the ceiling, he had a wood burner, he cut the firewood He lit the fire but his main response to being cold was to put more clothes on And he was accustomed to being cold since he was a student and frugality was a way of life, he was very accepting of cold So in order to introduce an intervention into this situation, maybe the best intervention would be to improve his understanding of the links between cold and damp and his child's health So what I'm saying here is that by understanding that the variety of cultural ensembles within even one particular outcome that we're concerned about, there might be a variety of different appropriate interventions according to what those cultures reveal Sometimes an intervention can have unintended consequences and I'm going to tell a story here about some work that was carried out in South Chile Where most households in the area that was studied here have these traditional firewood cook stoves And the aim of the intervention was to reduce outdoor air pollution because there wasn't a lot of wind in this town and it was very polluted outside so a lot of particulate matter from the wood stoves So they brought in this program to replace these old inefficient firewood appliances and to undertake thermal retrofits to improve the building envelope But the cultural context was an interesting one in that those wood stoves supported multiple activities and multiple needs for heating, for baking bread, for boiling water, for drying clothes and for drying the dwelling And it was a known technology that they could fix and maintain in their own right And in terms of motivators, these were part of their tradition, they had a whole lot of social attachments, they were associated with family and memories And they gave a sort of this sense of comfort and togetherness, there's a phrase that they had essentially meaning the snugness of being around the fire So when they carried out this intervention, as I mentioned before, one other thing was that in terms of that wider context firewood was a lot cheaper than the pellets or kerosene stoves which were part of the replacement process And what happened? Well there were unintended consequences, there was powerful resistance to change People resold their new appliances and put their old stoves back in again and ironically indoor air pollution increased because the building envelopes were less leaky and therefore held in the particulate matter inside So where the program intended to reduce outdoor air pollution, it actually ended up increasing indoor air pollution So I'm now going to move through to the topic which hopefully is of most interest to you, which is how do we evaluate interventions using the cultures framework So in terms of using the framework, my contention is that all interventions, whether they're a policy intervention, i.e. an intervention that is introduced by a policy agency Or it might be an intervention that's carried out by an NGO or any other organization or even at a community level All interventions seek to change one or more of these elements here, they seek to change motivators, they seek to change activities, they seek to change materiality Or they seek to change agency, that is the ability of people to make free choices and act on those The framework can be used as an evaluation to regardless of whether an intervention that you're measuring was consciously designed with the cultural framing or not It has been used quite frequently to evaluate interventions of various sorts that were not undertaken in relation to this kind of cultural framing But this framing helped understand how and why that intervention was successful or not And importantly it's suitable for use with multiple evaluation methods and indicators And it suits the use of both qualitative and quantitative measures I'm going to show you some examples in this part of the talk And I just want to introduce these words proximal and distal changes to you just to because I think it's quite useful Often as evaluators I think we tend to focus on what are the outcome measures and I call these distal changes So they're changes to the outcome sought by the intervention which might relate to energy or health or wood quality or anything else But within that there's a whole lot of what I call proximal changes and these are the direct changes to materiality motivators, activities or agency And they might be a direct result of the intervention or they might be a consequential result and I'll talk about some of those in a minute And again visual qualitatively or quantitatively We also can see feedback loops so changed outcomes and this is why this arrow is a double-ended one here So although we often think about the impact of an intervention flowing through cultural change into different outcomes It can also act the other way so for example if we had a farmer there was some kind of policy intervention that forced a farmer to start fencing his waterways Keeping his stock out of the water maybe doing some planting either side of the stream But then maybe that outcome of improved water quality actually had some other longer term changes on that farmer's motivators So maybe seeing his children playing in the streams, maybe the fact that he could now catch fish in the streams Maybe the fact that his neighbours and people who came and swam in the waterhole down the stream of it kept telling him what a good job he'd done Maybe some of those things led to further changes in terms of his motivators, his material interventions and his activities Maybe he decided to put in much more extensive planting for example to protect the upstream water quality for downstream users So just be aware that these consequential changes can continue over time I'm going to tell you an example of a really interesting study that was undertaken in Ireland using the cultures framework And this was to do with an intervention that aimed to reduce electricity and gas consumption and create warmer homes in a bunch of social housing within Ireland So they set out in this instance to formally evaluate before and after the retrofit So what did they evaluate? Well in terms of outcomes they were interested in daily average gas and electricity consumption They were interested in average indoor temperatures, the time spent at less than thermal comfort levels So how long was the house cold for and any visible mold, typical things that you would measure But they also measured other things, they measured a number of aspects to do with dwelling characteristics, heating devices and so forth They measured a whole lot of activities before and after how people operated their space of water heating How often they used key appliances, how long they heated for what their ventilation practices for what kind of appliances they used And they also looked at a number of motivators such as perceived comfort levels, perceptions of appliances, their aspirations and environmental attitudes So the intervention was undertaken, included wall and ceiling insulation, double glazing ventilation systems and new heating systems that had thermostats But it didn't quite work, there were unintended consequences, there were lower than expected improvements in warmth and savings What this analysis enabled them to see was that this was due to other aspects of people's energy cultures And part of that was they were carrying through frugality ideas from their previous lives And they had a lack of understanding about how to operate the new technologies They undertook inefficient routines, they lacked the skills to operate systems and they often manually overrode the thermostats in order to align with their ideas of frugality So this obviously then had some implications for recommendations for changes to the approach that might be undertaken for any future interventions So in terms of deciding what to evaluate, pre-intervention, if you can, ideally you would set up measures of the outcomes that the interventions sought to achieve The cultural features that are relevant to that outcome that have a direct relationship to that and any limitations in the Actors Agency And also don't forget that there may be external influences that are tending to drive or to prevent change, they may also affect the success of the intervention An example here from a study that was undertaken of the US Navy energy culture in this wonderful paper called How Many Admiral's Does It Take to Change a Lightbulb Was seeking to understand why the Navy had almost zero success in attempts over, I think, about 12 years to introduce energy efficient lighting into their ships And one of the big barriers to that was these broader external influences that were tending to lock Navy energy culture into a certain pattern of behaviors that it was very, very hard to break out of And then post-intervention, obviously again changes in the outcomes of what the intervention sought to achieve, changes in cultural features that are relevant to the outcome and changes in the Actors Agency Bear in mind though that the consequences of any intervention can be far greater than what you anticipated and I want to finish with a final story And this is in Banuatu So here there was a situation where the villages, there was no very, very little in the way of reticulated energy, so no mains power And people generally relied on these lamps to light their homes The intervention was from Australia Aid who applied a supply-side subsidy for NGOs to provide these solar lamps at a cheaper price than would usually be the case So what happened? And this was an incredibly rapid transformation across Banuatu which was far greater than AusAid either imagined So solar lamps started to replace kerosene lamps in people's homes What that led to was a whole lot of changes in activities, women and children started using lamps, previously they very rarely used the old kerosene lamps because they were dangerous and expensive New evening practices started to evolve with women weaving children doing homework They supported social interactions, people could take lights out to go fishing at night And there was less need for the families to engage in the cash economy because they didn't have to go and buy kerosene This was also associated with a whole lot of changes in terms of motivators, suddenly light was for everybody And sunlight is from God, it's free energy and it is something that is incredibly trustworthy compared to kerosene This is a poster from Banuatu And a stronger version to kerosene started emerging And that led to aspirations for more solar-related technologies such as solar panels, phones and chargers And what was notable here was how quickly this spread of different forms of lighting spread across the islands and how quickly these other changes started to occur So it's a great example of how an intervention can have more than anticipated outcomes So here we go, the culture's framework, it is a framework, it is not a method, but it is frameworks of thinking about the relationship between an intervention and an outcome and the elements that might be useful to evaluate as an evaluator So it invites you to evaluate change in cultural elements which enables a holistic assessment of the impacts of an intervention rather than just looking at it as in terms of one or two or not having a particular method for choosing which elements to evaluate It assists you to think about a more holistic approach It also points to the importance of evaluating change in agency to assist in understanding how and why constraints, these kind of constraints may be affected in the effectiveness of an intervention Evaluating those outcome measures allows us to focus on the ultimate intent of the intervention and evaluating both the proximal and distal changes allows assessment of both those direct and indirect impacts It also helps us identify unintended consequences either in terms of the outcomes or in terms of change in culture as we saw with the Vanuatu example Finally, this is a we shameless plug. I've written a book called culture and sustainability exploring stability and transformation with the cultures framework, which is actually due to be published online towards the end of next week and will be available in terms of hard copies from around the 22nd of April And it will be free to download its open access book and there's a whole chapter here about using the cultures framework in relation to policy which includes its use for evaluation And another chapter which focuses on its use in terms of a research approach And that's the end of my talk and we've got 15 minutes left for questions So I want to stop my share Janet, thank you so much that there was a brilliant fascinating presentation with plenty of illustrative quest case studies. I can see there's one hand up and one question in the chat but I'm going to be selfish and ask the first question Just to get it I just have a hypothetical question for you, especially referring to your study in Ireland, the social housing in Ireland Hypothetically say that it was impossible to collect data pre intervention Post intervention, how would you determine the previous state? Would you ask people's subjective opinions of what it was like before for you? Yes, yes, so some studies have done that and I didn't quote anyone here But there have been studies that have looked at basically just from a post intervention point and generally interviews with participants and interviews with also like policy professionals and NGOs who work with these people are also often involved So whatever sources you can find to establish what was the pre intervention situation in a post intervention evaluation Thank you so much. I'm just going to unmute Seno, would you like to ask Janet a question? You have your hand up Yes, hi everyone, sorry about the background noise. I just wanted to say thank you for the wonderful presentation and I wanted to ask how do we use this framework for an NGO, are there any specific resources that you could provide us or guide us towards that Using this framework to troubleshoot or have interventions for small organizations like community health organizations. Thank you Yeah, and what I will say is that it has been used I guess by NGOs as a framing for helping them think about interventions in the first place And they have found it very useful because often we don't tend to think in a very structured way about interventions and this helps give a structure around always reminding us that if we're looking at a material intervention there will also be other aspects of culture that may well be affected by that material change Or maybe even more important to work with than the one that we immediately leap to. So that's one thing I would suggest And the other thing is that for NGOs, so that study in Ireland for example, that was funded as a research project and it was quite comprehensive you saw all the things that were measured in that one You don't have to measure so many things, but I think it's important to think about Maybe one or two key things you can measure in terms of each of those elements. So if you're looking to improve health and you've got it, you know, maybe an intervention in terms of a material item Then one or two measures to do with what are the changes in activities that relate to that material item? What are some of the one or two changes in terms of key motivators? So is it in terms of people's expectations about warmth that changes it? Is it levels of knowledge? Is it certain beliefs that are holding people back? Those sorts of things and each case will be obviously really different and and if you're working closely with communities you will know you will get a you will have a really good sense of what some of those might be So so in fact your your gut feeling as a as someone who works closely with people is a really important thing to trust Thank you so much. Thanks a lot We have two similar questions in the chat section. David is asking how this approach fits into the practice of Elizabeth Schell David if you would like to unmute yourself And ask Janet directly that would be great Well I'm familiar with practice theory of this and it's clearly Yeah that's that's what it's about. I've just read some of her work and I see some similarities so I just wanted if you'd like to comment on that Yeah yeah and I have written quite extensively on this so so practice theory is is is certain is is one approach that that gets at some of these issues in a slightly similar way The difference one of the key differences is that practice theory focuses on practices that is it focuses on regularised routines within people's lives and then the focus is very much the exploration of practices in their own right but as maybe related to Norms expectations materiality and so forth What the practice theory also is not particularly structured about is some of the other elements that I'm laying out here so so it doesn't take a particularly structured approach to thinking about agency or to or to considering external influences Or necessarily to this this sort of the outcomes focus that I'm talking about so practices become the core of the focus and indeed practices are are almost seen in some practice theory as having a life of their own they they invite participants they they they continue to exist regardless of whether people Essentially are pulled into them or maybe drop out of them so as I said I've written quite extensively about the difference between practice theory and this approach it's a valuable theory it can be used in association with the cultures framework and in fact has been and is one of but but there are other many other theories of Behavior or or social change. That are also relevant to helping us examine aspects of what I'm trying to lay out here with the cultures framework. Thanks Janet. There's a similar question by Rachel Who's asking for if if it's similar to a realist evaluation that that focuses on mechanisms and outcomes. I've not heard of realist evaluation evaluated by trade or training so that's anything I'm familiar with. I can probably pitch in there Rachel I think it is. I really think it is because it's collecting evidence through empirical data so it's it's not really. It's collecting real real evidential data in sort of hypothetical assumption so I really think there's a lot of similarities with realist evaluation. We have a question on the RMA by smithy Who asks how this might be used in ongoing evaluatory processes under the resource management act to anticipate social economic impacts in consenting decisions. Smithy feel free to unmute yourself if you want to ask Janet directly. Thanks Janet. So essentially I guess the challenges that I think when people are trying to sort of make decisions on what's the most appropriate Next step based on sort of a current snapshot of you know what exists and you're trying to sort of make that assessment based on what you can see as likely impact so this isn't necessarily an intervention that's already happened and you're trying to project forward. There are so many sort of I guess practical considerations that come into play one is timing you know it's highly time pressured. But it's also sort of I guess based on evidence that is supported from one one one source if this is speaking sort of genetically normally it's the person who wants the intervention like the new building or the new that puts forward that value the data. And I guess my question is this framework seems to suggest sort of seems to assume a neutrality to where that information or the data is coming from to support that assessment it's assuming that it's not motivated if that's the word or it's not it's not inherently biasing a certain outcome. But how do you see this playing out in in the real world where things are sort of really contested and messy and sort of and and and where data and information is not mutually delivered to you as the person making a choice or a decision in your evaluation, if that makes sense. Yeah. Thanks for the I mean I must say I've never even thought about it in relation to the kinds of evaluation that are undertaken under the RMA. It's a it's a very different, different situation I think. I mean there there it's often about a choice about whether to grant consent to something or not not grant consent to something I don't I don't think this is applicable to that at all. What what this is about is a framework for in this in this instance the way I'm presenting it here today is is as a framework to help us think through a policy intervention or an intervention that is undertaken by at whatever level by a community group, a government agency, you know, an NGO, a Marae committee to to consciously try and get an improved outcome for for a particular part of the population, and to to unpick what that process might mean in terms of changes to, you know, those cultural elements as you understand, I think that's that's kind of a different evaluation situation to evaluations on the RMA, which I think are a very different kind of beast. Thanks, Janet. Janet, we have a question from Mel. Is there tell my perspective on the application of this framework. Is there a tell my perspective, is that what you said. Yeah. Mel, feel, feel free to unmute yourself if you want to elaborate. Yeah, instead tell my perspective. Just, just a question on the considerations of it's probably more relevant in the context of New Zealand around how this framework, has it been tested for any project specifically looking at how Maori perceive culture and implications of those interventions within their world and to what extent does this framework. fit into those perspectives. It's, yeah, I guess I can answer that in two ways. First of all, in terms of I guess the core ideas of, of the elements of culture that is that's. Yeah, I've both, I've received very positive feedback from from Mari who are who I've discussed this with. I'm not aware of any specific applications, particularly to like about I situation directly with this framework. But what I can say is that it has been applied to many different cultures across the world in many situations in third world countries and, you know, in first world countries in, you know, Africa, South America, Asia. It's kind of a, it's a, it's a universal set of ideas that is applicable anywhere, but in terms of it's an example I can that I can quote of its direct application in a Maori only community. No, I can't. I don't know about that. It may have been, but I'm not aware of it. Thanks Janet Phoebe, would you like to unmute yourself and ask a question. You. Hi, Janet. I don't know if I can turn my camera on. Yeah. And I can't. So I'll just ask my question. Yeah. So thank you for your presentation. And I've just mentioned that I have actually read about the framework before, but I definitely got more out of it listening to your talk. So I'm glad I came along today. I work in, I'm working at the moment in evaluating energy efficiency interventions. And I've seen in the literature that in this field there's been quite a few projects that have had probably about half of the policies I've seen designed for energy and efficiency improvements have had unintended consequences or possibly even more. And so it really seems to me like that's one of the particular values from, from my perspective of your framework is giving some meaning around those unintended consequences because often they're not sort of developed or gone into March is just to say, you know, you do read quite a few papers where you just said, oh, it didn't work like we expected. And that's it. So I think that's a really valuable aspect of your framework there. And I was also thinking that, you know, longer term the exciting thing from my point of view to could be when enough projects have used the framework to do some kind of like meta analysis of those unintended consequences, particularly with relation to energy efficiency and I was just wondering how close do you think it has that have we got had enough projects using it yet to start start looking across those kind of things to an extent so so in the book. I have done a bit of a meta analysis across lots of uses of the framework. But what I was focusing on there was was how how research using the framework helps us both understand what I call cultural stasis that is the fact that culture tends to be quite resilient to change. And then another set of studies which gave us insights into processes of cultural change so what what is what might stimulate changing culture, what might be the starting point, how might that flow through to consequential changes. I've not actually looked specifically at the idea of a meta analysis in relation to energy efficiency interventions but you could be right there could well be enough there's quite a few like it, although it has been applied to a lot of different topics. It still is mostly applied to energy related studies of one sort or another. Maybe it's a good project for a master's student. Thank you. Thank you. Thank you everyone for coming and a very special thank you to Janet for this brilliant presentation. Unfortunately, unfortunately, we are three minutes past closing time. Janet, are you happy to share your slides with everyone who participated. Thank you so much. And the title of your upcoming book is culture and sustainability. I will definitely keep an eye out for that. We're getting many comments of praise and thank you in the chat section. As an evaluator myself, I got a lot out of your presentation Janet and I look forward to reading your book and to applying the cultures framework into some real life, real life practical evaluations. So we're all very grateful. Thank you so much. And thank you everyone for joining in and for participating fully.