 So it is my great privilege today to receive for the turn of fourth all of our seminar of this year Someone that is well known in the house Bert Loreden for our colleagues from the University of Antwerp Many of you knows how he wrote extensively on mechanism But now he's moving to a new and triggering ground. So after a one-hour talk of Bert We will have come in by Pierre-André Come on by Pierre-André and after that a general discussion. So you have the floor Bert Well, thank you very much It's an honor to be here. Thank you for the invitation It's decades since I've been able to talk about my research for a live audience It's nice to be able to do that again I Must say that I had expected as some master students in this room. So the first slide We should not make it very long and deeply philosophical discussion. It's just to Point to some of the key issues that I'm trying to address. So suppose you have an earn and you know that it has Only black and white balls, but there are plenty of them and you want to know how many black how many white To make it a bit more concrete. Let's test the hypothesis that 80% of them are black So you need observations, of course to test this hypothesis And let's say that the observation consists of a report on the total of 20 of the balls That have been taken from this earn 17 of those 20 are black. The rest is white Then the question is whether this observation confirms this hypothesis or doesn't So let's pretend your master's students. What's your first reaction? And we will not use much time What's your first reaction? Yes, yes, no Do I know? Okay, that's nice. Thank you for saying yes So I will not talk well, I will talk about black and white balls, but not for very long and I will talk about the concept of evidence in the Rescript Research Program Called evidence-based management I'm a philosopher of science and only a philosopher of science, so I'm not in I'm not an economist. I'm not in management studies. I'm not a manager, whatever I'm only a philosopher of science. So is Eric Green. He used to be my supervisor back at the time at Dent University and together we are teaching a course in the research master. We have a joint research master program with universities of Ghent, Antwerp and the Vrij Universiteit Bössel and the course we are teaching together is called philosophy of biomedical and social sciences and one of the topics we have been discussing for past years is evidence-based medicine evidence-based policy and one of our students in the last year and two years ago was Anne Beewerkens She's a philosophy student, but she has before she's an older student, before she started studying philosophy, she has had a long career in management functions in HR departments and so on and so we asked the students to write a paper on for instance evidence-based policy and she stumbled upon this evidence-based management project and now together we're co-authoring a paper on the concept of evidence-based evidence in evidence-based management starting from discussions last year Evidence-based management is a research program initiated mostly by Denise Rousseau. She's in Carnegie Mellon And together with Eric Barnes, I will say a little bit more about these two people but not very much about the people themselves. They promote evidence-based management as a way to use evidence to make better organizational decisions and Eric, Anne and I were very sympathetic towards that project so we really like it except for the fact that when you look at the details of how they talk about evidence that's somehow misleading in some sense too liberal in a way that undermines their own aspirations. So what we want to do is using insights that are not very new or innovative from the methodology of the social sciences and from philosophy of science to advocate a more strict definition of evidence which can be applied to evidence-based management and therefore or in that sense help to improve or to boost the program. I apologize for the Can you read it at the back? Okay, so I will start by briefly characterizing the research project itself say a little bit more about why they have initiated it and then I will say a bit more about how they define evidence as opposed to information and as opposed to data. And then I will show that the way they talk about evidence is in as a two-place relation whereas we advocate a three-place account of information, sorry, of evidence using, well, I will start with a very simple toy example then I will move to some more genuine methodology of social sciences but all very basic. I'll do the the added complexity that you can wish from from social sciences or from biomedical sciences, whatever will only strengthen our point mainly that evidence should be treated as a three-place relation. Then we will present this logical procedural approach to give a more strict definition of what we mean and then I will turn to the book and they talk about using evidence from practitioners they talk about evidence from the organization that you're running as a manager about evidence from stakeholders and then again we will show that they give really valuable advice, valuable for companies, for managers but sometimes what I say is misleading and that can be remedied by our three-place account. And at the end I may zoom out and touch on the question whatever role as for now mostly talking about myself and people like me philosophers of science who are not in in management studies as well how should we relate to the field of managers and given that we are not part of that field. This is Denise Lucille from Parlii Human and she's in organizational behavior and public policy so she's not a philosopher. Yelik Bayans is the managing director of a center they have created it's based in the Netherlands the center for evidence-based management so they're we're just trying to to to reflect intellectually academically on the notion of evidence-based management they're really trying to promote it to people in the field and one of the initial observations that drives their program is the fact that management research has shown that very often decisions taken by managers taken by the organizations fail so they talk about roughly half of the decisions do not give rise to the intended result. One of the reasons being that such decisions are very often driven by the opinion of the highest paid person in the organization the hippo is the highest paid person's opinion and whatever he or she thinks has a very strong influence on decisions being made and Eric Bayans and Denise Lucille say well if we change the way management is done in practice and if we stimulate managers to use more evidence that will help to improve managerial decisions so this is how they define the research project evidence-based management in right is about making decisions through the conscientious explicit and judicious use of the best available evidence from multiple sources and what you need to do is you have a practical issue or problem you try to solve you have to turn it into something that you can into a question that you can try to answer using evidence then you go searching for the evidence once you have it you have to assess whether it's reliable enough whether it's relevant enough then you put it all together and you incorporate it into the decision-making process at the end to check whether you did fine and by doing so you hope to increase the likelihood of a favorable outcome instead of just following the hippo in the coffee room Alexander as well is this similar to or is this inspired by evidence-based medicine I presume you all have heard about evidence-based medicine well if you you compare this the quote on the previous slide with this it's a quote from second at all one of the founding sources of evidence-based medicine it's it's almost an exact copy so it's very strongly inspired by evidence-based medicine but I will not talk about ebm so goal is to use evidence in order to improve organizational decisions and evidence is not the same as data in their book and it's not the same as information data it's a bit fishy how in some sense you could think about it in terms of just ones and zeros but also text can be data words can be data images can be data and the point of data is that by itself it's completely meaningless it's just words just strings of numbers just pictures it's not meaningful because it's it's considered on its own without any context it becomes meaningful by putting it in a context by interpreting it but then it becomes information in by definition it becomes information so by organizing the data structuring it analyzing it and interpreting it the data becomes information but information is meaningful but not by definition evidence it's only evidence and if it's if it's related to a specific claim or hypothesis or theory that you want to test so evidence is always evidence for or against a claim hypothesis so it's information supporting such a claim or contradicting such a claim so you can have positive evidence you can have negative evidence so they define information in terms of data and evidence in terms of information but this definition of evidence is strictly two-place we argue and we want a three-place definition instead it's two-place because they just relate the meaningful data the information to the hypothesis and nothing else whereas we will argue that you need to take into account the method that has been used the procedure that has been used to obtain the data and has the information as a third component in the equation and we will start illustrating this idea by going back to prehistory of philosophy of science so if you look at how Hemple wrote about confirmation you see that he treated confirmation in a strictly two-place sense and that made him vulnerable to some some counter arguments so at the time claims like all ravens are black word typical examples being used to talk about confirmation how can you confirm the hypothesis the conditional hypothesis that for everything that exists if it is a raven then it is black well you look at positive instances but if you have an observation report involving three objects a b and c and all of them are ravens and it's also the case that all of them are black then using Hemple's jargon from this observation report you can deduce by ordinary classical logic the development of the hypothesis that for every object if it is a raven then it is black so the development is what the hypothesis would say if only the objects mentioned in the observation report exist so since from the hypothesis well the hypothesis would come down to this development if only those three objects exist and since you can derive this from the observation report the report directly confirms the hypothesis what is not mentioned at all is or a question that has not been raised on the previous slide is how we came to bear this observation report came from so how was this observation report obtained that's not something that he paid much attention to so confirmation is just about the observation report in relation to the hypothesis purely to place now I will present two toy examples and then we will move to to some things like parameter estimation in scientific practice but first the toy examples and then an alternative definition of Empelian confirmation that makes it a free place and what we will do here we will repeat in the coming slides with regards to practically or scientifically more relevant issues the first toy example comes from Sir Archer of Eddington the one who confirmed general relativity during the solar eclipse in 1919 he wrote a book about philosophy of the physical sciences and he tells a story about a scientist an etiologist who wants to investigate sea creatures so he wants to observe sea creatures and test hypothesis so what he does he takes a fishers net and catches sea creatures and the observation report consists of many sea creatures that all have fields and that all are relatively large there are none of these sea creatures in his catch are very small so he says well this seems to confirm my hypothesis that all sea creatures have fields and following ample that would be true it also confirms a hypothesis that no sea creature is less than two inches long but then I quote Eddington an onlooker may object that the first generalization is wrong there are plenty of sea creatures under two inches long this onlooker may say however your net is not adapted to catching them because the meshes of your net are just too wide so the fact that all your observations have confirmed in ample sense your hypothesis does not suffice another example suppose that you want to test the hypothesis that all ivuses are red you observe a hundred ivuses and you notice that all of them are red according to ample this would mean that you have confirmed the hypothesis but as soon as you realize that you have observed these hundred ivuses in a zoo and that the management of that zoo has a peculiar fullness of red ivuses for example for aesthetic reasons then you realize that this observation report is useless and as matter of fact there exists green ivuses white ones that black ones and so so what we need if we want to have evidence for or against the conditional hypothesis we need an observation report and that entails in ample sense the development of the hypothesis for the class of objects mentioned in the report that's just ample but the report itself should have been obtained in a way that does not system create systematic bias and the net was a problematic procedure going to the zoo was a problematic procedure and so on and so forth so that's the basic insight which is not only to be applied to conditional hypotheses but also for instance to parameter estimation as it is practiced in social sciences so for example the 20 balls that I talked about at the beginning and the fact that 17 of them are black three of them are white does this confirm the hypothesis that at least 80 percent of the balls in the urn are black it depends on how they were selected if I have carefully selected 17 black balls maybe the only 17 black balls among two million white ones then then you see that that this observation report does not confirm the hypothesis at all it would confirm the hypothesis if if the ball the 20 balls would have been selected in a random fashion and the same holds for parameter estimation in the social sciences if you want to know how many Belgians or what percentage of people in Belgium are in favor of free public transportation you cannot interview all of them so you need to interview a sample and then you really can derive a sample statistics a 45 percent of the people I have interviewed are in favor so using inferential statistics I can say for certain but with a high reliability how many people in the population are in favor of free public transportation but only if my sample did not create systematic bias I've taken a train from Ghent to to here if I would have interviewed a hundred people on those various trains then maybe I would have encountered many more people in favor of free public transportation than if I would have selected them randomly so a random sample or there are alternatives that also typically give rise to a representative sample is what you need for parameter estimation what I would have done on the training is convenient sampling it's just interviewing those people who are easiest to interview likewise and and scientific studies should not only be conducted on at university students but these happen to be but we quote one of these social science methodology books saying exactly this organism of many social sciences that's convenient sampling and you cannot do you cannot use convenient sampling for parameter estimation you can use it to to create an exploratory sample to do exploratory research so to do come up with new hypotheses new ideas that then have to be tested in improperly so just like what we did for example and we can now do for parameter estimation when do you have proper evidence for parameter estimation well if you have a sample and that from which you can mathematically derive a parameter estimation and of course you need to use the right statistical tools and there are many of them but all of these require that your sample itself is not problematic so it must be representative that must be you need a sample that has been obtained using the procedure or method that does not create systematic bias and typically guarantees that the sample is represented even if you take a random sample you can be unlocking you can by accident end up with a sample that is not representative and then you should not exclude that possibility otherwise we would advocate some friction free epistemology that's not our goal but you should use methods that work most of the time so generalizing from the two proposals an adequate characterization of whatever type of scientific evidence should have a logical component it should say using which methods be it deductive logic in the case of sample or inferential statistics in the case of parameter estimation and so on and so forth so logic is logic in the broad sense including mathematics so you should say how conclusions are derived from observations but you also need a procedural component saying how the observations themselves have been collected so apart from the information the meaningful data and the hypothesis you also need to pay attention to the method that has been used now we will turn to the book by a we will show that most of the time they what they write is correct sometimes they give misleading information which we fear would lead managers to make bad decisions to take shortcuts we wonder this is a philosophers of science stressing things that methodologists of social sciences know for a very long time is this really relevant for people in the field well we think it is and to motivate part of the reasons why you think it is let's turn to a quote at the very beginning of their book Housseau and Byron's writes that evidence-based management challenges conventional wisdom authority and traditions think again about these hippos regarding the way decisions are made it raises the seldom discussed issue in contemporary organizations namely the quality of the evidence that is being used so quality of evidence is very important challenging conventional wisdom is very important again they refer to evidence-based medicine which was the first domain in which going evidence-based was publicly advocated since sometime you have evidence-based policy and now they they jump on their train and advocate evidence-based management but the critical cradle of evidence quality is central and we want to add to this project and add into this project is one of the advantages of of this paper reclaim it helps to challenge this conventional wisdom and it does and I think this would be using to the ears of managers it makes the application of evidence-based management more efficient if you follow our advice this is a schema from their book showing the different types of evidence so in one of the first quotes I gave they talk about multiple sources of evidence one of the sources is published scientific literature as a manager or as a company and you can turn to published scientific research find it we will not say much about that the point is that this scientific research is mostly very general it it covers larger domains whereas you as as a company as an organization want to use this to solve specific problems taking into account the specific features of your company so you need evidence from the organization itself Rousseau and Byron stress the importance of relying on the expertise of practitioners and they also advocate taking into account the values and perceptions of stakeholders three I will start with the practitioners then the evidence from the organization and find the evidence from the stakeholders to to show what can be improved in our opinion practitioners are people like managers business leaders consultants and so on and so forth who have specific expertise that can be valuable in in making new decisions it's really about their expertise it's it's not just about any intuition or personal opinion they may have that's important and the difference between expertise and mere intuition is that this expertise is specialized knowledge that is the result of practicing the same thing over and over again in a setting that has been relatively stable the environments changes constantly then then you don't practice the same thing over and over again and what is needed is also that the people who have practiced this received direct and objective feedback if you can do the same thing over and over again in a stable environment receiving direct and objective feedback then you become an expert and if you're an expert then maybe it's useful that people turn towards you and ask what do you think and by the same was so discussed several methods that you can use in order to try to find out what practitioners and many of those are highly reliable from a social scientific point of view but then all of a sudden they talk about walking around and asking so you just walk around in your company and you ask people maybe I should read it around so the quickest and easiest way to gather evidence from practitioners is by walking around and asking of course this method is prone to bias but sometimes wandering around in an unstructured manner through the workplace and asking people randomly their judgment about an assumed problem or preferred solution is a good way to start so one of the problems you immediately encounter is things like a selection bias so the question is why do they advocate walking around and asking given that they stress the importance of the reliability of evidence and given that they know that this is not a reliable method when characterizing sources of evidence they characterize evidence from practitioners at the beginning of their book in this way the first source of evidence is the professional experience and judgment of the people I have mentioned it's not the same as intuition or opinion for the reasons I've specified and so on and so forth but given how they treat things like walking around and asking what they are really defining here is the first source of information in our framework and in their framework not of evidence so what we need is a more strict account of evidence from practitioners it is information all evidence is information and it is obtained from expert practitioners not just from anybody that's just staying very close to a bias and so on it has to be critically appraised that's not something we add that's something they stress over and over again in the four chapter of their book because what you ask if you ask what people think even though they're experts and you're inquiring about their domain of expertise you have to pay attention to the the risk that their judgment is prone to cognitive biases and so what you need is to critically appraise what they say or what they write in order to filter out all these cognitive biases which by itself can be time consuming so these two things are not something new but what we think should be stressed more in Bayreuth's book before handing it over to people in the field is stressing that methods you use to interview practitioners these methods should avoid systematic bias and if you do that you will be very you will be much better at challenging conventional wisdom one of the aims a bayreuth centre themselves have and because if you just walk around and ask people who love you ask well those people who are in your own organization whereas if you want to have representative at several if you have a representative sample of relevant experts you also gain information from people outside your company and that will improve the quality of your evidence and hence the quality of your decision rather than saying well you could walk around and ask people but be careful doing that we would say just don't think of walking around as a way of collecting evidence because it's very hard to try to ignore what what you seem to have learned afterwards and also particularly appraising information is time consuming so rather than first gathering information and critically appraising it and then checking whether it's really it should really count as evidence that's that's the way things are presented in this six ASAP procedure you should first be more restrictive when it comes to gathering information in order to avoid time consuming unnecessary time consuming critical so at least for the evidence about practitioners having a more strict definition of evidence and will be beneficial to organizations and managers in the same holds for evidence from the organization I will first discuss two side issues that will play a role at the end of this this subsection if you wish the first side issue is that the way by example so talk about evidence from the organization somehow overlaps with how they talk about evidence from stakeholders so they talk about information you find within your organization itself when they make a distinction between hard numbers such as cash flow and so on on the one hand and then soft elements perceptions of an organization's culture for example this is very similar to what they later call evidence from stakeholders by using an example they themselves give I will try to illustrate how intertwined these two things are and then I will argue that we should keep evidence from the organization in terms of hard numbers apart from evidence from stakeholders in terms of more soft elements and subjective feelings so the example is about a large input they think and give examples from managerial practice all the time and one of the examples is about a large insurance company that decides to change from a regional structure so people selling stuff people selling stuff being responsible for specific region is turned into people selling stuff being specializing in specific products so the board of a large insurance company has plans to change its regional structure to a project product based structure and according to the board that will secure the company's market presence and drive greater customer focus but companies sales managers strongly disagree with this change arguing that they change the region based structure will make it harder to build good relationships with customers and will therefore harm customer service so you have a decision or decision making internal criticism and then the question is how can you use evidence to find out whether it's true that turning to product based structure will be beneficial or that it will be harmful so what that company did was analyzing organizational data and those data revealed that customer satisfaction was well above industry average that's one thing another thing is that and further data analysis showed a strong negative correlation between on the one hand the account managers monthly travel expenses these are part numbers and on the other hand customer satisfaction suggesting that sales managers who live close to their customers score higher on customer satisfaction so the evidence could be used to argue against this intended reorganization but the point is that monthly travel expenses are hard numbers and what you need to do to do to to gather these to assess these and so on and so forth is very different from the challenges that you're facing when trying to assess customer satisfaction and instead of allowing for two very different types of evidence from the organization and then having some some distinct type of information from stakeholders which completely overlap so when I will talk about stakeholders this is about customers about suppliers about employees and it's about their subjective feelings and perceptions about soft elements that's that's just a copy from what is written here so in the interest of conceptual clarity it's better to take these apart but not just in the interest of conceptual clarity but first and foremost because the challenges you're facing when assessing these types of evidence are different second side issue is related to the question how easy it is to obtain evidence from your own organization and strangely enough by and so say on the one hand that it's easy to obtain and on the other hand that it's not easy to obtain why is it easy to obtain organizational evidence well because it's easy to obtain organizational data and therefore it's easy to obtain organizational information and then for some reason they equate organizational information with organizational evidence so they say well companies continuously produce data often automatically by definition data is meaningless so these are just for instance lists of numbers but well you know your company so you can start analyzing organizing structuring those data until it becomes meaningful for instance instead of a list of numbers it becomes a list of birth dates of employees and that's something meaningful and something you can use to base a decision done and then they say well in the context of this book organizational evidence refers to data and the ways it has been transformed to make it more interpretable that's just a definition of information and then they don't add anything about well but be careful because expiancy so there's a very strange step they make which may need managers or organizations on the other hand they say well no it's not easy to obtain organizational evidence because the methods you need to obtain this evidence are very similar to the methods you need to to obtain scientific evidence and all the possible methodological problems you may encounter in scientific practice are important here as well so it's difficult scientific research is difficult gathering analyzing organizational evidence is very similar hence it cannot be so easy so that was the second side issue so what we would like to propose is a more strict definition of evidence from the organization restricting it to hard numbers keeping all the soft elements we're highlighting all the soft elements to evidence from stakeholders and when is this information derived from the hard numbers evidence well if and only if it was guided by a causal role I would say a little bit more about this and in such a way that the research design made knows that are useful to test that causal model so it's about hard numbers the causal model is a term that russo ambiance use themselves but they use various terms they use the word causal model but also logic model also a theory of change we think that the word causal model is the best concept because it's about the mechanisms or the causal processes underlying the organizational problem that you're trying to solve and underlying the solution or the relation between the solution and the problem that you're trying to implement on the next slide I will cite an example from yahoo which by and so use themselves and then the research design is included in this proposal because of our logical procedural account of ethics so the role of this causal model they illustrate using an example from yahoo this is my and in almost 10 years ago she decided that and it's funny that we talked about this issue the right before the start of my talk she decided that people should no longer work from home she sent out a memo saying from now on it's no longer allowed to work from home and the reason is that she thought that this has that working from home has a negative effect on productivity why well because she implicitly used some assumptions about causal processing processes playing a role including the assumption that the speed and quality are sacrificed or often sacrificed when we work from home this was rather a hippo the highest paid person's opinion is this a correct opinion this is an incorrect opinion well I also argue what should be done is you should try to make that causal model explicit and untested so you should find out that this is one of the assumptions driving the decision or better the intended decision and then you should try to gather evidence about how well do people when they work from home how do people when they don't is there a difference is there a relative difference and so on and so forth so that's why this causal model is included in so this is not new this is taken from by itself and but by reasoning in terms of such a causal model you can much better try to challenge challenge the hippos that's that's one benefit it's more efficient because you you can better avoid by by starting from a specific causal model you can avoid what about some of so-called efficient expeditions this has nothing to do with Eddington and his fishing net fishing expeditions is their term for data mining if you engage in data mining you run the risk of finding correlations that have have no practical value for instance the notion of a spurious correlation is useful here not the spurious correlations that derive from common cost structures these are in many cases practically useful but the spurious correlations that just result from from time series and and they give the example it's one of these classic examples the divorce rate in Maine during a certain time frame on the one hand and the per capita consumption of margarine I think also in Maine and they have both been increasing for some time and since they were both increasing there was a positive correlation between the two but that's a positive correlation that is it's completely useless and it's a correlation you can find if you're just engaged in data mining but not if you start from a causal model that that that that keeps you focused on useful knowledge and again quality of evidence is better secured then evidence from stakeholders these are all the people that affect decisions and or are affected by decisions from the organization they can be employees for example inside the organization or people living in the neighborhood of a factory customers suppliers and so on and so forth and what you want to know and and they they argue for this based on ethical motivations the bounce owners have said well for ethical reasons you should take their interests their values their concerns into account as much as possible and they give a number of examples and then they conclude and this is somehow kind of a definition if you wish of what they mean by evidence from stakeholders this concerns subjective feelings and perceptions that cannot be considered as objectifiable facts regarding a soon problem or proposed solution but it is relevant to the decision and it constitutes a stakeholder evidence it's kind of tautological that the evidence from stakeholders constitutes a stakeholder evidence but anyway that's that's a side issue so what they want is to gather evidence from those stakeholder concerns and values and of course they add in a way that is as valid as and as reliable as possible and then they they they go through all sorts of techniques and and things you should take into account including representativeness of the sample and then they say well maybe you could start with a focus group of six people for example six employees and you ask their opinion and then then maybe if you don't have six employees but hundreds then then maybe you should add to the results of this focus group also a larger survey that is in our opinion potentially misleading because you end up with information from this focus group that is not evidence but it's hard to to discount it as being non-evidence and it's also inefficient if you invest in focus groups that will not lead to evidence so it's better to have an account of evidence also from stakeholders that stresses the importance of the use of procedures or methods that do not systematically create bias so to conclude we support the project many of the things they write but they try to write in as digestible a way as possible it's it's a page turn and most of what they write is is justifiable but then every now and then they seem to conflate their own definitions of evidence information and data and although they want evidence to be reliable they advocate procedures that clearly make you vulnerable to misleading evidence so you should be careful not to treat information that is not evidence as evidence by applying this conceptual distinction much more radically and when they advocate exploratory methods well it's okay to use exploratory methods but don't fool yourself and don't think you're collecting evidence and when you do that and you don't mislead yourself your company will be more efficient and your decisions will be more okay so let's start yeah I'll try to find a sheet of paper it's just me in the way over here yeah okay yeah make work if you can go so it's on yeah yeah so now it's part of the comments that for the beginning of the discussion and it's my pleasure again to to see for the first time Pierre-André from the Char Ouvreur who is Chargé de Recherche FNRS in another center the center the interdisciplinary center of the Char Ouvreur about justice economy and social ethics social ethics yes thank you very much I'm very happy to be invited here to comment on this very interesting presentation and thank you very much for sharing with me on your draft paper which I found super interesting although I have to apologize because I am not a philosopher of science my research area as it has been said is more in the political philosophy and ethics field however I happen to have a degree in management my degree of management from a previous life and therefore I have a personal interest in the topic and more precisely the title of my degree is the master of science in management and I have to say I've been asking myself for quite a few years how scientific management exactly is so therefore I'm very happy to learn more about that today um so as I said I had the chance to read your draft version of the paper and I found it pretty convincing I think it shows quite well that evidence-based management as theorized by balance and also although it looks like a very sound approach at first sight a very common sense actually suffers from quite significant shortcomings I find your main thesis about evidence of the three-place relation very convincing that it's not only about hypothesis and information but also about the adequate method to gather and biased information and I think this clarification is welcome that it is not only about logic and observation but also about procedures because indeed it seems to me that managers management consultants very often do not embarrass themselves with strict methodological procedures when they want to prove a point actually it seems to me that they usually try to confirm their hypothesis rather than really try to test and test it maybe it has to do with these hippos you mentioned like trying to confirm the hypothesis that come from the top of the deliarchy well at least that's my very biased experience from what my friends who still work in this field tell me so that's my equivalent of walking around and asking I think the second very viable clarification that you provide in your paper is that you show that there are different epistemic issues related on the one hand to the heart numbers and on the other hand to the soft elements as they are called these soft elements so the perceptions by managers employees external stakeholders you show that they need to be carefully handled and gathered and that random interviews and representative surveys are better than these focus groups that are very commonly used actually in management in marketing although they obviously are not very reliable as a source of evidence but you also show that the heart numbers may also not be so reliable and that they may include some bias and what came to my mind was actually the example of the accounting standards which profoundly influence how the financial statements of a company look like what they look like and what they say and actually show that these heart numbers they're not pure facts they carry a lot of interpretation let's just think about a monetization for for example in financial statement that's pure view of the mind actually that there's a decrease in the value of the capital assets I also very much liked your comment on the data I think well right now who are very enthusiastic about the data is probably also I presume a driver of the enthusiasm for evidence-based management that you can gather so much data that management is going to become very scientific but you show that there's also a threat in the data because this is by the time we're so who appointed us to that threat but to steal that credit it's quite interesting that to show that there's a threat because without inadequate causation theory it can lead to putting your trust in spurious correlations like that of the divorce rate and the rate of consumption of margarine in Maine and so therefore I found a paper very convincing and I've asked myself three questions when reading a paper that I would like to share with you the first is about the reasons for such a lack of method in evidence-based management the second is more generally about the epistemic status of management and the third one I apologize a bit out of your scope but it's about the I think very important ethical implications of this type of management so first my biggest question as I read the paper was why why does evidence-based management and more generally managers management consultancies disregard the importance of these methodological procedures actually I find it quite surprising when you compare to our established these methods are in the natural science in the social sciences it's quite surprising that these very well established methods are not translated into this area and indeed to my knowledge there's no specific methodological training in management consultancies like the interns when they start there they learn by doing and most of them I assume do not have training in social science so they're more or less have soft skills but not these methodological skills and so you also mentioned that convenience sampling is cheap and easy and I think that's a big reason why the more satisfying methods are not used because managers and management consultancies want to go fast want to do things cheap cheaply and but to a deeper level one may ask why would people prefer this quick and dirty approach to a more scientific one if for example maybe more efficient and be a better way to to manage the company and my so that's that's my question do you have any clues why these methods are so much disregarded and my guess is that as long as it works they do not really care about scientific knowledge whether it's good or bad science but being happy to have your comment on that my second question would you like him to answer first or do you want to do the three questions what do you prefer maybe I'm a bit tired so one by one and so what are the reasons for this lack of methodological rigor I wrote down and then you you said two things in management on one hand and in evidence-based management on the other hand so maybe it's good to to keep these sub-questions apart why is it absent in management well you gave some of the reasons yourself I guess it's easier and it's cheaper and the evidence-based management initiative which tries to to to to to raise methodological standards and management is relatively recent so that's the second reason and then you said as long as it works the the cheap and easy methods should be okay question is how do you know well what do you mean by it works it it's one interpretation could be it meets a certain standard we do have and that's easy to to check another interpretation would be it works better than alternative methods that's a kind of factual criterion but how can you find out if you don't apply the alternative so and maybe the the line of reasoning is that it works and I side with Housseau and the islands that if you would treat it counterfactually and if you would try to to do your best to to to test the counterfactual by implementing a more scientific approach as well then most likely you will find out that going scientific although it's it may take more time maybe worth the effort at the beginning of their book they devote some time to the question whether companies do have time some decisions have to be taken in five minutes and then you don't have time to to set up large surveys and and do a systematic review of the available literature but I think that large companies should be more future looking and and try to invest in gaining the scientifically based knowledge over a larger period of time that could be used and and implemented when five minute decisions have to be made so that would be what I think is the answer to your question when focusing on management in practice then the question is why is why can you find problematic advice in a book about evidence-based management that tries to be as as helpful as possible I guess one possible explanation could be that they try to make it a page turn so although they they write about but it's in a sense maybe I've said this at the beginning of the talk it reads like a handbook of social scientific methodology but also a book full of anecdotes about Yahoo and the challenger and and and insurance companies considering to restructure so they they they're trying to do both at the same time and maybe that could be one reason also the book is written by Byron's and who's so plus four other people so sometimes when we hint at the lack of conceptual rigor maybe that's the the philosophers um professional uh this formation or how should I call it professional disease thank you um could also be explained by the fact that this is um collaborative work so maybe a second edition of the book could could be improved in that sense and then yeah by applying a more severe or strict definition of evidence the whole project could be improved that's just repeating the main point of the the talk so that would be so so I would be far less critical of evidence-based management as a program than of management in practice because I much of that and in this sense I just sighed with oh so great thank you um well my second question is a bit related to the first one that I tried to play the devil's advocate a bit more um I'm wondering about the exact epistemic status of management and whether there can be such a thing as evidence-based management it's a bit of a knife question but I was wondering whether we can really call management a science of or if it's more than art um I find you cite that one point in your library you cite Ben who so will make an analogy between the good practitioner and a good violin player and therefore I find that analogy quite telling in the sense that it seems that to them it's about experience in the broader sense and not experimentation like testing specific hypothesis and I was wondering at that point whether we can really say whether management statements in management style in mind science can really be falsified or not um and can we for example say that an organization or structure a feels better than an organizational structure e in terms of net income or share value I don't know um do we really have the counterfactuals that you mentioned in your reply um in any case it seems that in the famous piece that in case that is in management schools which are really rather better of the teaching of management we are very very far from control experiments in labs but we are also very far from surveys and social sciences in terms of methods and therefore I was wondering whether it can really be there can really be counterfactuals whether you can really treat it as a falsifiable field of knowledge okay thank you again I see two sub questions one one is related to the question whether management is a science or an art and then the other about falsifiability of claims and which methods they use to test claims um there's I would definitely be willing to grant that that there is an art-like component to management just like there is an art-like component to medicine um so um I've shown the the the the deliberate um uh parallels between how biologists also define evidence-based management and how second at all defined evidence-based medicine um second at all reacted against a state of of the science of medicine the practice of medicine where medical doctors just made medical decisions because of the way they had made medical decisions before and just because they were in authority in their field and second at all claimed that scientific evidence should be brought into medical practice in a way more systematic way um in medicine of course you can have these randomized controlled trials um but um within the field of evidence-based medicine even though not even though on the one hand you have this um strong reliance on scientific evidence and in particular on randomized control the results of randomized controlled trials then there is a discussion about what is still the role of the professionals implementing uh the the um the knowledge evidence-based medicine has sometimes been interpreted as trying to get rid of all the um um uh specific input from the professionals themselves but then when you read second at all they they really uh tried to to make clear that um it's scientific evidence on top of practitioners expertise and not instead of and the same you see in in this evidence-based management the fact that um evidence from practitioners is uh one of the four sources of evidence shows how much they um value the art uh uh arty side should I phrase it off of management practice so um summing up is it a science or an art I think it's it's a false dilemma it's at least the second but the first can be uh added uh and that's useful as long as it's done properly and that brings me then to the uh second sub question um how can you you test claims about management practice given that randomized controlled trial you talk about lab trials even uh abstracting away from uh investigating things in labs there is less room for randomized controlled trials and for instance in experimental psychology but you could bring in uh insights from experimental psychology you could bring in insights from other social sciences I've had discussions with the students uh the students every time I um wanted to to make room for at least the abstract possibility of conducting randomized controlled experiments uh with uh companies uh one set of companies doing a strategy a uh implementing strategy a second set of companies uh implementing strategy b in principle you could do this whether you will be able to convince companies to to um to engage in what might seem risky uh alternative strategies I don't know and she may be right there most of the time they would be uh to to conservative but randomized controlled trials is not the only uh methodology available so even if you gather purely observational uh evidence about how companies happen to to um to behave um where you did not as a researcher yourself decide what they had to do even then you can derive valuable causal conclusions it's harder than in when relying on randomized controlled trials but it's not impossible so can management claims or claims in management science be tested and falsified I would say yes um much of the social scientific literature is based on non-experimental observational evidence so just use those techniques and so my third question is a bit out of the scope of the presentation but it's about the ethical implications of each form of management implications ethical I was wondering what what's the effect of evidence based management on the responsibility of managers if there is evidence that a course of action or decision a is better than an alternative decision b then um managers liable for every failure to reach their goals because they have disregarding the evidence well one could think there's kind of a slippery slope uh there and that it could be also a way for these hippos to shift the responsibility down to the middle managers to the lower managers in the company well it it reminded me a little bit of this form of management which is consistent delegating the responsibilities by describing the goal and then middle managers have total freedom to whatever they want as long as they reach the goal and I was wondering whether there's some sort of synergy between this authority delegation and responsabilization of middle managers and evidence-based management to see what I mean yeah well try to see what I know the net assist um if we if we if we make the parallel again with evidence-based medicine can we say that well we are used to consider that like a surgeon has quite a big responsibility whereas manager maybe has a very limited responsibility with what he's doing does it mean that with evidence-based management we're going to consider managers to be much more liable because there is evidence because if it's an odd and there's no evidence and it's not scientific you can say okay it's not therefore it doesn't work but if there's evidence it's quite a different question I don't know well so so this is just gut feeling but I think that um you have described the line of reasoning which makes the managers more liable and you've also described the line of reasoning making them less liable by shifting new responsibilities delegating responsibilities and I think that you can in principle go either way with evidence-based management and you can go either way without evidence-based management so um a medical doctor applying um outdated um procedures or or therapies could I guess in some sense be held responsible for uh unfortunate outcomes um so in a sense well by reasoning in a parallel fashion you could say that if there exists evidence saying that strategy A is better than strategy B in whatever sense of better because the word better can cover expected extra sales or or lesser pollution or whatever um but if it exists then then at least in large companies in big companies which could have resources for for retrieving evidence assessing evidence following the six A step procedure so if you um did not do a good effort to find the the the right available evidence then in some sense you could be held responsible um on the other hand you may be right that that managers could have now maybe I'm now to to cynical so you got this some managers do have a tendency to to escape liabilities and and one way to do that could be by by blaming the middle managers who were in charge of so I think that all the existing problems can be reinforced or can be um can be tenuated it but you can go either way but that's that's just now improvising okay now we open the floor to any question even online so for this claim I'm a philosopher of medicine no because so I was not familiar with this movement even though I have also a small degree in business but in any case when I I'm presented with what they call evidence-based management as a person that is familiar with evidence-based medicine it's very weird uh I feel like you made a good job at being sure sure because it's very strange like the way they are defining evidence it's very strange because I was I was expecting the approach being about scientific research management and in the end what you presented is like no it's not it's too complicated so we don't care about the scientific research no no um one of the uh sources of evidence is scientific research findings and we just happen not to talk about that fourth component in our paper they do talk about it and they mostly focus on on the well what they talk about is how systematic reviews should be done so in in that sense it's it's very similar to um to co-creating collaboration style uh reasoning the point is um well at least that's that's an analogy that I find uh illuminating in evidence-based well in medicine you can try to find as many research papers as possible you still need to translate all those all that knowledge to the concrete patient in front of you so even in medicine you need more information more evidence than the purely scientific evidence if you define scientific evidence in terms of outcomes of randomized controlled trials and so yeah and uh they wouldn't call it evidence uh okay it's information about the patient it's not evidence because evidence to just make specific procedures to have medical knowledge uh I mean some laws level in the yeah he can be just any total knowledge of professional experience but they are pretty much you know on the lower ends of the hierarchy and for instance uh take all the various inconsistencies I mean for example in the case of medicine then the values of the patients they are not evidence like it's not they are not in the just other medical knowledge there are values that you have to take into account in taking a decision and that's where the where medicine meets ethics so it's weird because then it's uh I feel like they have really like you actually exactly like you said the concept of evidence is very fluid like very strange um and maybe because they don't want to go full or we want more science because it's too expensive I don't know well I'll try again to defend them what you need in when you want to apply scientific medical knowledge is a reliable diagnosis of the characteristics of the patients and here you need a reliable diagnosis of the characteristics of the company and its social environment and you can acquire information about the organization and its environment in a good way or in a bad way just like you can um run reliable tests on a patient and or reliable tests so I see more parallels um perhaps than than than you do um I guess I feel like it's a good catchphrase catchphrase to say evidence based management but at least it's broader than that okay well I'll um think about the possible disanalogies more than than than we have done so far we have taken the analogies for granted and then ignored the parallels with evidence based medicine and and as a result I I did not systematically think about these analogies uh so yeah I want to ask my question because it's also connected to the disanalogies although I don't know much about uh philosophy of medicine or or your talk but I was thinking um in the case of medicine the goal is clear and we know that what we want is that the patient is alive has a good life etc already companies um so the question is is it possible that we have um same data same observations and evidence in two different companies but the management is completely is completely different because the goals are even in one case something more like equality etc and the other is making as much money as possible and then it's a margin and then I don't know I wanted to push this disanalogy in the case of medicine and in the case of management and whether you would agree that it's possible to have exactly the same evidence but two different correct management strategies I think it's definitely possible to have the very same data evidence and two different managerial strategies whether you could both call them correct um well what I have in mind is if you have uh clear evidence that um production methods a is both um more environment friendly and more expensive and the other one is more polluting and cheaper then you can easily see two different managerial strategies emerging whether both can be called correct that's then that depends on on your ethical framework from a profit maximizing maximizing a perspective the one will be the better and the other one will be wrong from an environmentalist perspective vice versa but that and now I am zooming out a little bit from from the specific question that other things other determinants do play a role in management decisions other than uh the available evidence that that's for certain whether that's a big or a medium sized disanalogy with evidence-based medicine or with medicine um you seem to go in the direction of being a big disanalogy but even in medicine you can either go for maximizing life expectancy or maximizing quality adjusted life years and so on and so forth or for minimizing costs for society but there as well you can go in different directions starting from the same evidence because values and goals do play a role yeah this was this was really cool thanks um I also have an analogies question in a different direction so um because one thing that I thought was neat about this was you know it struck me that your argument in the center surrounding moving from the two-place evidence relation to a three-place evidence relation I mean that's just old-fashioned cool bread and butter philosophy of science right and I mean that's gonna I mean you're getting the Eddington example there right that's going to apply to it seems to me you know anybody who cares about taking empirical data and acting on it right and so what interests me here is so I wonder then what you think I'm trying to put this in a way that doesn't make it sound like way too broad and vague but let me just make it broad and vague and hopefully you'll understand what I mean anyway um does this just sort of mean that in that sense does that just make evidence-based management another kind of candidate science and therefore being being confronted with a general argument about the role of evidence in philosophy of science and so we just kind of have uh you know we have an argument here that applies pretty broadly across philosophy of science you're specializing it for the case of what sort of starts to feel like one particular kind of empirical science so if you do you want to take really seriously this idea that the apparent analogy that comes out of the way that you analyze what they're doing that this is just a proposal to turn management into a social science so if they're going to do that they better do it well um I mean are you are you happy that we can be kind of to kind of bite down on that inference to like let to follow them in that direction or do you want to be a little more careful this because this relates to your to the you know to Pierre I'm going to question about this kind of art science balance because I think that's that's really interesting and I'm not yeah I wasn't 100% clear on how how you were seeing that relationship sort of applying this general this general philosophy of science argument might make you think that we really have a pure science kind of way of thinking about what evidence-based management people are doing but also there's this kind of art side so yeah how are you seeing that kind of broader positioning I hope that was that was sufficiently clear for you to be able to answer it okay um so I think that management studies should be as scientific as possible and as a matter of fact I would say that management studies is just a kind of social science with input from psychology and maybe from other disciplines but broadly speaking it's a sub-discipline within social sciences and it should be as methodologically sound as possible and so that's one thing then the question is management itself be scientific that that's a bit a strange question management itself is a practice which has art-like features which involves values and goals that themselves are not scientifically grounded but the to go from a present situation to a set goal many times or many ways and it's best to use the best way and therefore scientific evidence and other reliable information and information gathered in the right way can be helpful no I think I think already actually yeah let me let me let me jump back in because I think I think already the management management studies distinction is super helpful for me that that already clarifies a lot I think the kind of the question that I had about about sort of which part is the kind of analogized bit to a social science and which part has the sort of technique aspect you know I think that already yeah that already helps so in that sense saying sure yeah this is you know this is a supposed methodology to render management studies a kind of fully empirical at least potentially approach but that it's that it's that management studies management relationship it's going to be kind of tense and interesting and weird that's a that's a I hadn't thought of it in those terms before but that's a cool that's a cool thing to point out but they also advocate and I think they're right to have better methods employed by companies organizations themselves so I made the distinction between management and management studies but I'm not trying to blur it again yeah yeah because the the whole point of their project is that maybe in my management should pay more attention to management studies sure and sure and then then question is as you said well our basic point is not new at all and then question is how relevant is this paper it's not relevant for philosophers of science we will not send it to a philosophy of science journal is it relevant for management studies well you would hope it is not if it is that's problematic so so I think that if you would talk about this with Denise will so for example she would say yeah we all know that then the question is why the book itself is sometimes misleading so I do think it's still relevant in that sense and it is relevant for managers and the organizations which they lead because for the very very same reasons that evidence-based management as a research program itself is relatively new relatively innovative and useful I think at relevant for the field can you go back to the slide where there are the four type of evidence the four type of evidence four types of evidence thank you is this yeah because I was wondering like if we remove the scientific part and maybe let's take hold of values is there any manager who would argue I take my decision without based on any facts and on any experience I feel like it's like nobody would argue this fact well it depends what you mean by facts and by experience so one so and no no manager would say I take decisions regardless of facts and regardless of experience but the question is how do you gather the facts and the experience do you walk around just meet with the three colleagues that you know best and follow their opinion you know already or do you more systematically try to investigate what people on work for are thinking these are two different ways to do the same mainly raising decisions on facts and experience and walking around is cheaper and easier and quicker but less reliable people are being it's good to walk around well the now I go back to the very first slide it's the cheapest so so the the roughly half of management decisions fail because most of the time it's um this failure now I'm quoting byron son was again is tied to managers who rush to judgment impose their preferred solutions fail to confront the politics behind decisions ignore uncertainty downplay risks and discourage search for alternative so that happens all the time even by people who think they the best available management research suggests that so unless you can prove that this is not true but this is is um what the best available management research suggests but then I feel like they I guess they don't want to make enemies because the way they present stuff is a little bit like especially the part on expertise like it's very soon you should tell to someone you have the real expectations that you you learn stuff by repeating decision and basically by confirmation bias but also no you should also follow some more rigorous methods so it's very strange because if you in that no no think about engineering they do exactly the same it's not it's not pure scientific but for instance in medicine if you take this description of expertise you don't have medicine you have history of medicine when we had no treasure at war basically just confirmation bias okay and so it's limited to be a little like I feel like the to make the so the advice is to make use of expertise from practitioners and that itself comes with two risks first risk is that you do not represent the sample of the practitioners that's why they stress methods like surveys and so on and why we stress that the fact that this has to be has to be stressed much more maybe the O in hippo is relevant here the O stands for opinion so many of these decisions if what I've just coded is is correct many of these decisions in the end are based on opinions the CEO are not experts yeah decisions you said they were management decisions yeah so they fail CEO decisions why is it good expertise like what do they have no no no if you have a baker that keeps making bad bread they wouldn't say they have a very good expertise so you have to identify the good bakery yeah they I think they use the example of a baker themselves so the decisions I think they're referring to here half of which fail are decisions that are not based on expertise itself well the hippo is an opinion so they're made by the highest rate person or a select group of highly paid people but that's why I can't agree like if the field as it is like we are so scientific and rely only on subjective expertise like it's like medicine in the 19th century like I don't see the difference yeah but the advice is have a look in your organization and outside and see what the judgment not just the opinion the judgment is of experts where expertise is then they use the example of a baker a baker is an expert because a baker can bake loads of bread almost every day of a year after after years in a relatively stable environment and he or she gets direct feedback if the produce is bad yeah but I think I have a better analogy is that because I'm teaching when engineers do risk assessment so it cannot be completely scientific because there's a normative aspect that is based on experience and judgment and some kind of difficult way to modelize the way they see what society is ready to accept in a certain circumstance but of course if they go against scientific data it's bad so they have to take into account the best scientific data but it but it cannot be reduced so my impression is that what they would what you were saying is that if they take in account the the experience of expert they have to do something similar to find to identify among the experts or by assembling or by another reliable methods which one have the better success for this specific task that cannot be just scientific what I was surprised is that it seemed that they have a lot of and maybe your team too confidence in random sampling for engineering expert we would say random sampling is depends of the population of engineers expert that you want to sample maybe it's not the best way to get the best represent the best random sampling is not the only yeah it's not the only but the idea is to identify these individuals because it's not strictly scientific it's part of this know-how yeah but in engineering firm they have the ecosystem too so in engineering firm if you look at this young many are very bad because they are made by the guy that is paid the highest the goal therefore is to find the good experts no i'm just going to tell about who is a good expert when there is no scientific evidence but can i be an expert if he's just relies on his subjective experience can i chat something so first of all these practitioners these experts should not be equated with the highest paid persons that's one of them and second what you say about evidence-based medicine about medicine is that if medicine would have ticked solely to this kind of evidence then there would have been a problem or to rephrase it the major advances in medicine happened only after looking more broadly than just what what experts thought and even a little bit like moving because a lot of what this is an argument made by alex in the in a paper but it's very interesting the idea that this part is what the where you find the most bias and the history of medicine is basically just a history of that like professional experience and judgment and confirmation bias all that bias and then possible effect and then yeah well i want to add one more thing and how well i will add two things first of all they do not say who's ambiance that we own that that managers should only use this evidence this is only one of the four sources just like medicine also use the results of randomized controls trials and so forth and you're right that judgments even of experts can be prone to all kinds of bias but that's what bias and also are aware of and what and that's why they say it's basically that you need to critically appraise these judgments i don't know if you can evidence is it's very similar to confusing medical evidence and just professional consensus it can be very different yeah the first one can contradict the second and then that's true but here as well well so if you do not cut out the other sources of evidence in particular the scientific research findings then you have tools to define the strategy it looks cleaner in medicine but there's probably the same problems problems are different and like you said they do so also maybe that's why they make a strategy like a conscious strategy not to go through all but thinking about disanalogies with evidence-based medicines is definitely useful for us and also the bread analogy i'm not convinced because thank you we have bread my friends we have good bread as far as but the bread is good in management like maybe it's stable and we're lacking we're like knowing as many management is like blogging in the 19th century management studies that that's the goal of my business i don't want to push it but because it's capitalist i think it's the non-starter for you what not at all okay it's just uh it's the value of imply maybe not at all i'm just very confused about what so since we passed the time yes virginity i had a business yeah i had sorry i had five questions but i and you have more and we should continue that in front of a beer and to leave the people at home to take their home thank you again our speaker