 Thank you and thanks for coming along to hear me today. It's great to be in New Zealand and at the National Digital Forum for the first time. So I'm here to talk to you about grey literature. Anybody know anybody heard of grey literature? Yes, a few people. So a few years ago I was like many of you and didn't know what grey literature was, never heard of it, wasn't subject to any grubby innuendos about salacious novels and so on. But as Brenda said in 2006 I got my first library job running a service called Australian Policy Online. Let's see if I can. So Policy Online has been going since 2002 and provides, brings together research and information on public policy related issues. So reports, discussion papers, briefings, reviews and so on and also now includes New Zealand material thanks to a partnership with Victoria University of Wellington. But when I started working on Policy Online I didn't really know what it was that I was actually doing. Each week I would go around to various websites, academic university centres, government departments, think tanks and find documents that they had on their website, publications that they'd put out, catalog it and we'd do a newsletter and out it would go. Partly because of copyright but probably more because the site was started by academics and a journalist. We just linked out to the material. So we didn't actually hold a copy of an item. We linked out to it. And one day I made a terrible discovery which was that our public, our database, our collection was being, was disappearing. But the links that we were sending people to were breaking that the organisations that had produced them sometimes were completely disappeared, government departments would completely disappear and that the substantial material that had actually been the basis for our public policy decision making was no longer available online. This is part of what can be considered the evidence base. So I asked myself where is the evidence if it's disappearing and if we're not actually collecting it, who is collecting it? Over time I realised that I was working with grey literature and this is a term that's used quite a lot in health and criminology. It's become more understood because of the Cochrane review process in health says that you must look at the published and grey literature. It's a bit of an academic term but it's also sometimes understood in libraries. It's got a long history, dates back to, I would argue, political pamphlets of the 16th century and libraries have long struggled to be able to find it and collect it because it's outside of the normal publishing channels. So you may not be such a familiar area to you but it is worth thinking about when we think about public policy and where the resources and the research comes from that informs public policy, where is that actually coming from? There's the assumption that it's coming from peer-reviewed journals but actually it's from a lot of what we would call grey literature. So what are collection services doing about making sure in the digital era that this material is all collected and available? So I asked myself this question and I was fortunate enough to get some support from the Australian Research Council working with Professor Julian Thomas and Professor John Houghton and with partners from the National Library of Australia, the National and State Libraries of Australia which includes National Library of New Zealand, the Australian Council for Educational Research and the EIDOS Institute. And in many ways these reflect the key sectors that are concerned about this issue which is government and collecting services, universities and academics, think tanks, commercial producers of research material. All of these have got a stake in making sure that this material is discoverable. So we'll get on to the terrible question of what is grey literature. I really don't like to get bogged down in it too much because it becomes this, you know, what is and isn't in the list but generally speaking, it's known as grey as opposed to white which might be published, books, journals, media magazines, material that goes through an established channel, usually some level of commercial ownership or selling but not always. Then there's the, I think the grey literature that I'm talking about is material produced by organisations. Reports, discussion papers, this is particularly a set of resources that would appear in the public policy area in other areas like archaeology or engineering. You might have a different set of kinds of materials, conference papers, data sets, submissions and evaluations and so on. And then there's other material that sort of is a bit more maybe ephemeral or not quite of such public and substantial nature and some people call that black. If you get into it, some people actually reverse the black and the white so I wouldn't worry about any of that too much. The kind of material that I'm talking about today, here's some examples. So as you can see, it covers a lot of important topics, you know, our higher education system, democracy in New Zealand, exposure to natural hazards, some of these other ones are from Australia, indigenous issues, human rights. So these are the kind of research that really touches on the most pressing issues facing us as a nation. So as part of the research project, we've done three online surveys and it's mainly what I'm going to talk to you about today is some of the results of that and to think about what we've found out about the evidence trail. So we did an online survey of around about a thousand information users and they came from government NGOs, education and a small number from the commercial sector. We looked at producing organisations so they were meant to respond as organisations rather than as individuals so the numbers are considerably less, but they are still representing whole organisations. Again, we've got representation from government, from NGOs, from education and the commercial sector and collecting organisations, 114 collecting organisations from government NGOs and education. So I think we've got a reasonably good sample to base it from and having around a thousand respondents is generally regarded as being statistically representative. So what are we talking about when we're talking about evidence? So I had to do some really daggy things just because I've got so many slides of survey results that I'd excite you all with some very daggy word art. So we asked our information users what resources they used for public policy. So most of these people are people engaged in information and research for policy and practice. They told us that they use regularly reports, journal articles, discussion papers, briefing guides and reviews and news reports and conference papers. So these are the most used material used regularly or occasionally by this group of people. We also got data sets that are fairly high, a number given the detailed nature of them and submissions and evaluations I think are just two items to watch. They're not used quite so much but they are heavily used particularly in civil society organisations or NGOs. What is the most important for this group of people? Reports are the most important. They are more important than journal articles and also very important are discussion papers and briefing guides and reviews. You can see from the breakdown in the sectors that there's a bit of difference. So academics can get access to books and therefore they're much higher use of the book. They are also higher use of journal articles. NGOs on the other hand really favour reports, discussion papers, briefings and as I mentioned evaluations and submissions are really important for the civil society organisations. So I think this also shows that within the policy area there are differences within government civil society and education sectors in their approach to material. But still overall they're not wildly dissimilar. How much is it used? So around half of the users said that they use grey literature, 60% or more of the material that they consult is grey literature and for a quarter it's 80%. So this is really heavily used source of material. And we asked what impact would it have if you couldn't access grey literature. And 81% said it would have a higher to severe impact. So this is I guess one of the points of the research was to really establish the value of this material. It's often dismissed as not being as credible or as important and we've really been able to establish that it is extremely valuable. Why? Why do people use it? So policy and practice work was one reason for about 36% of people. They said policy development, advisory work and program design within government requires a good grasp of domain expertise and this is greatly helped by grey literature. And another reason is that it provides a broad view of the research environment. So grey lit provides another part of the complete information picture and without it I wouldn't want to rely on only peer reviewed material. Some of the most innovative and thought provoking material is grey lit and important in a changing world. So these are, it's a unique source of information and as the bottom quote indicates it often taps into the voice and the knowledge of particular groups, in this case a women's health policy group. So that's often what people are looking for. So as part of this research we've actually done some number crunching based on the responses that we got in terms of the time that people said that they spent using grey literature and other data and we extrapolated that at national level and we reckon that the activity value of using grey literature is $33 billion nationally per annum. So who's evidence, who is putting this out? Organisations are producing grey literature on a, about a third on a weekly basis. So this is quite a fairly high level of production and most of them are putting it out at least around quarterly. So this is mainly digital material and I think it starts to get a sense of the scale of the amount of material that's out there. What are they producing? Conference papers were the highest one followed by discussion papers and reports, briefings and guides, submissions, webpages and information sheets and media releases. Again you can see there's quite a bit of difference across the sectors with government being a big producer of reports and information sheets, whereas submissions are produced a lot by NGOs and journal articles obviously are produced more by academia. So again the production is a little bit different. I haven't put it on here but we also asked what was the most important item and conference papers are actually a bit lower and we think that it's a dissemination tool more than actually seen as a really important in terms of policy. Why do these organisations produce this material? Because they want to influence the policy decision and they want to be part of the evidence base. They want to inform public policy and practice so they're the two big reasons. Knowledge translation is also an important part of it so sometimes it's important to realise that Grey Literature can be a product of academic research it may be the same research that's also in a peer review journal and to maximise public access to research. We estimate that the national Grey Literature production costs are also around 30 billion dollars per annum. So the big question that we need to sort of figure out is where is this material? If it's disappearing from these websites as I discovered then what are we doing about collecting it? Most of these organisations put their material straight online. They upload it to their own organisation, 84% of them often and always do that, often or always do that. A few say that they have a repository. I have my doubts. There's some education they're using their institutional repositories but still not as much as you would think given in Australia every university has an institutional repository. Many of these institutional repositories don't actually collect Grey Literature material only the journal material so it's a big missed opportunity there. Complying with the national deposit legislation this is clearly not happening. So only 20% of organisations are bothering to provide a print or a digital copy to a national library. How do people find out about information? Well they go to these websites. So there is possibly a feeling narrow circuit of exchange of information from key organisations to people involved in policy and they're possibly not getting as broad a spectrum of materials that they should be. EMA newsletters are really important especially for NGOs and colleagues sharing information. So that's been shown in other studies of public policy that colleagues sharing information is a really important part of it. How often are the and so how often are following services used? So Google is clearly going to be a key way of accessing material. Getting it through an organisation's website, a subject repository or gateway or a library catalogue. Libraries are fairly low down and we generally found that people aren't really aware of the Internet Archive and places like that to find. And we have three comments in the survey saying oh I didn't know that that existed I'd do now. So that was useful but yeah there isn't a sense of that as a place to go. So what gets collected? Journal articles and books. So the standard sort of print based material. Reports and conference papers and audio video and discussion. So this would give you some hope. I do have to shatter any hopes in a minute that digital material is being collected. So what's collected the least is evaluations are really low, submissions are really low so that material that the civil societies are really trying to want to get hold of is actually some of the least collected material. And then what are we collecting in the digital space? Not very much. So you know I think this is an important slide in that it shows nearly 20 percent of organisations aren't able to say what digital material they've got. You know some central number aren't collecting anything. We're way more... I mean of course we have more and larger print collections that would be expected. But very small number of large-scale digital collections and when you do look at what they're collecting some of those collections are more sort of journal articles or their standard electronic material. So we're estimating that the collection costs are about 265 million. So much much less being spent on collection than what this material is worth. So this is probably a little bit dense but it is good to hear the voice of the people responding. So we asked what issues people had and we got 525 written responses which is pretty incredible for an online survey. People were very passionate about their problems that they had. So we coded these. 45 percent have said that finding the right material, finding material is not a problem. Finding relevant material is a massive problem. They don't know what exists. They don't know where to look for what they want. They have to then go through lots and lots of material to find out what's valuable and there aren't enough collections that they can go to or they aren't the large-scale collections. So as the first quote says finding the latest and most relevant research to my topic can be time consuming at best, a nightmare at worst. It's very hard to know what exists and how to find great literature reports produced generally held on individual websites and there's no way to search for particular subjects across separate websites other than search engines like Google but they also pick up a lot of chaff as well making it frustrating. I think that's pretty much sums it up. Accessing resources is another big issue with the cost of journal articles and also market research which can be very expensive. Government content and the fact that quite a lot of government funded material never sees the light of day at all. Maybe the dark grey literature is problematic for 43% of people. So this quote issues around subscriptions to peer-reviewed academic journals. They're too expensive for NGOs to maintain comprehensive subscriptions and most researchers I know rely on their personal university enrollments to access them. We also got a quote from somebody in government saying they employ PhD students so that they can get access to university libraries. So this is a black market in access to journal material by our decision makers. Poor production quality is a big issue for users but it's also a big issue for collectors. So it's brought up by 14% of people. Accessing older policy literature, having to spend significant time searching and assessing quality, lack of publication information, e.g. dates and authors on documents. Those kinds of issues. Multiple subscriptions and memberships are required. Sometimes the information is not peer-reviewed or the extent of scrutiny peer-review is not evident. So I think I just might have to speed along a little bit. So just we had a look about deadlinks and most certainly collectors are concerned about deadlinks. So about half of users said that they came across a deadlink on a weekly basis and the average time spent looking for an item was 17 minutes which is really quite high. So we were able to do some calculations on that and estimate that deadlinks are costing us $5 billion per annum, Australians. So just to come to the conclusion what we asked collectors what we should do about this situation, standard bibliographic information was the number one. Fair use copyright law is campaigning for that as number two. Agreed metadata standards and collaborative collecting was the fourth one. So I think we need to really have a think about how we do those things but I do think we've got a clear understanding that is what people are prepared to do or say that they think needs to happen so hopefully they're prepared to do it. We have a mandate as the politicians say so we asked people did they agree with the idea that libraries and information services should be able to store copies of print and digital material including grey literature for long term access and use and all groups, producers, users and collectors all agreed very strongly that that should be the case and we estimate that if we had better management of grey literature we might be looking at an efficiency dividend of around 17 billion dollars and I estimate that it would cost a very small portion of that for collectors to be able to work together to provide a better service for information users. So as part of this project I've just put out a discussion paper about this last week and there are five recommendations and this is the discussion paper. So great of course. It's not the final report so what we're really hoping is to get some sense from different groups. This is obviously a collector group about are these the right recommendations that we need to be putting forward and also how do we action this from here moving forward. So one of them is to improve collection and curation of policy resources. So these are the things that we're suggesting needs to happen. So increase the amount and type of digital and print grey literature cataloged and collected. Increase the investment in collecting services and infrastructure for digital resources. Improving the auditing capabilities of collection so that we can actually know what we have. Establish systems for collaborative collection of digital and print grey literature and adopt common standards of metadata. Maybe we need a national digital collection strategy. Develop and adopt common standards of usage metrics and citation systems for repositories and databases so that people can track how their material is being used. Work with existing aggregated services like such as Trove and WorldCat, the Internet Archive, Google, Google Scholar. So if we've got various collections, how do we bring it together so that people don't have to go to a whole lot of different places? And develop sustainability strategies around these collections because the funding doesn't always last for the collections either, just like some of the organisations. So how do we make sure that if a resource doesn't continue, that it can move on to some other place? So that's it from me. If anybody has any questions, blinded you with data. We do have a few minutes for questions. Thank you. Tim Jones, a question which I've got right. I mean, you say you don't want to get bogged down in the definition of great literature, but more definition is to me, working in an archive gallery where I just do some very good work. Paper, garbage every time a mail arrives is quite, at a quite a higher level. We deal with tickets and invitations to exhibitions constantly and they have to refer value. And we don't know what to do. That's just an observation that there's an even deeper level of great literature that is equally difficult to learn. Yeah, I think that's true. And I certainly don't want to suggest that other material is not valuable. Annual reports are often trawled through to be able to make assessments on what companies are doing. What I'm, how we've tried to frame this in order to have some sort of limits on what we study is the policy context. So what is actually being used in terms of the evidence base for public policy? Maybe tickets might be relevant to that. You know, if you're a historian, you might be able to get something out of that. But we've tried to bound our analysis by the public policy context also so that we can make a case for its importance in a particular to a particular group of people, which is in this case, government as one user. OK, last question. I think that Brewster was Brewster, did you have a question? OK, you're both in line. All right, Brewster, did you can go for it? Do you have a question? Yeah, it's actually an offer. There's a link rot, said was a major issue. And one of the nice things about the people you're dealing with is they'd like actually link rot to not happen. Yes. And the Internet Archive made a free little widget that you can put on a website in the 404 handler. So if you connect to somebody's website and it drops to a 404, it'll prove the Internet Archives collection to go and say, is it there? Yes. If you have cooperation of webmasters, that's a winning way to go. Of course, you don't always have it. But in your particular context, you might, because you might be able to get it through your government because you actually have a civil government online, well, the United States. But they're doing something like this in Britain for government websites. So you might be able to have a technical act that at least gets around one tiny part of your thing, which is link rot. Yes, I think that's a great idea. I think we'll take the last question and then move on. Amanda, do you have any importance of a lot of this material you're talking about and its value? I'm just wondering why legal deposit is so poorly observed? I think because we are really dealing with organizations that are not professional publishers, most of them wouldn't have an ISBN and not that that makes all that much difference. So it would trigger legal deposit. But one of the problems that many people complain about is you can't even get a date on the publication. So let alone thinking, oh, I should send my document to legal deposit. It doesn't occur to organizations. They might just not be aware of it. And they may not even have the sense that they are being formal publishers. I actually came across a government document that had the only thing information had on it was a title. And it was actually a policy that they wanted companies to follow. But that was all it had. So, yeah, I think it's just the lack of awareness of professional standards of publishing. The matter of education. Yes, yes. We are out of time. So please join me in thanking Amanda. Thank you.