 Okay, very simple outline of the talk so I'm going to describe a problem which is developing in operational metrology. I'm going to then describe solution and what we're doing about the problem. And finally at the end, I'm going to summarize things and ask the question, are we on the right track or not. So, yeah. So first of all, I thought I'd start off by saying a little bit about why, why we need new sources of data for weather forecasting. Well, I guess the main reason is that a lot of severe weather occurs on quite small scales. And there are economic benefits if we can forecast severe weather. Obviously enables us to better target warnings, and the better warnings are targeted, the more effective mitigations are. Now, historically, a lot of small scale weather, we haven't really attempted to forecast. And it's only in recent years that forecast models have become powerful enough that it becomes a possibility. So, an example of that is fog. So this is a map or visibility in the London area and I'm sorry there's no scale on the map but basically the area of the map is very roughly the area enclosed by the M25 and the there are some places marked by crosses including the the London airport. And it shows very graphically that fog is very small or can be a very small scale phenomenon. So this this map is produced by 300 meter resolution forecast model. And you can see that some of the London airports are affected by fog on this day and some are not and this huge economic consequences to fog affecting the London airports, particularly Heathrow, which is running close to capacity all the time. So any disruption at Heathrow has massive economic consequences. Now, as things stand, I couldn't produce a map like that based on observational evidence, because we just don't have the number of observations visibility necessary to construct it. Now what we do have is what I describe as conventional observing networks, which have served as well for, you know, well, since the dawn of meteorology, 150 years. In some cases, we've had rain gauges and thermometers. But nowadays, of course, we've got a whole lot of other things as well, satellites, radars, lots of remote sensing instruments, but all these are sort of, you know, infrastructure heavy. And they're very expensive to both install and to run. So that's that's sort of what we have. Now this is this diagram shows what what's been happening in terms of trends in the resolution of both forecast model and some observation networks over the last sort of 40 years and also looking ahead. So first of all, if you look at the red line, that is the actual that is a representation of the grid spacing on the UK operational forecast model. So when I, when I started my working career, the grid spacing over the UK for the forecast model was 100 kilometers. And today it's one and a half kilometers. So being nearly a two orders of magnitude increase in resolution over over that time. Now, for our conventional observing networks. There either has been very little change, or, or, or where there has been change, it's not been towards the magnitude. So for example, if you look at the weather radar for as possibly our highest resolution observing system, the rest of the pixel size in the late 70s was five kilometers, and it's now one kilometer. So it is, it is actually our only observing network, which is a comparable resolution to the forecast, as things stand. And of course radar doesn't measure everything it measures rainfall, mainly. And also provides some wind data. So that's, that's a graphic representation of a problem that's developing in that the forecast resolution is improving rapidly. And the, the conventional observing resolution, while some, some aspects are improving other aspects are fairly static. And there is a, there is a, an increasing gap between the scale of the forecast and the scale of the observations. I mentioned that there's a gap, but why, why is this a problem? Well, we need observations to initialize the forecast model to monitor how it's doing to verify the forecast, which is really important. And also to inform the model development. Now, if, if the gap is allowed to persist or widen even further. It's a problem. I mean, a very basic level, the forecast won't be quite as good as they could be. But there are other things as well, the, if we can't verify the forecast, then there's a question about a credibility. It means that any investment we make in improving the models is going to be devalued. And it also means we're wasting the huge computer resources that are going into running the high resolution models. So what's, you know, how has this problem developed? Why, why, why are you being able to increase the observations density? Well, I think it's mainly down to the difference between the rate at which computer power has increased compared to the rate at which the cost of observations has decreased. And also observations are the cost of observations has got some fixed constraints, which are very difficult to very difficult to work around. So just to summarize the problem and the first time in the history of the Met Office and operational metrology in general. We now have the ability to forecast whether in greater detail and we're able to observe it. And so I think we need to take action to try and close this resolution gap. And we simply can't do that just by expanding our existing networks. It's not practical to find the millions of observing sites that we would need. And it's not economically feasible. And it's going to require innovation. It's going to require the ideas promoted by the digital environment program. And everything I've said so far is aligned very nicely with the aims of constructing a digital environment program. If you look at the homepage. There's a statement there, which more or less summarizes everything I've said so far. So the alignment between this problem in operational metrology and the digital environment program is almost perfect. Okay, so, so I've described the problem. So, what, what's the solution? Well, there are opportunities for solving this problem arising from a number of sources. There's new technology, preparation of sensors, new measurement platforms and new weather sensitivities. And I'm just going to describe each of these briefly in turn with some examples. So new technology, first of all. Well, there's, there's, I just listed there, some of the technologies which I believe are relevant to solving this problem. I won't read those out. Hopefully it's fairly obvious. And, you know, I think we've got examples of where we're using all those technologies to try and access new sources of data. Okay, as well as that, there is a real privation of sensors in society now, all sorts of infrastructure and devices have sensors in them, which have some sensitivity to the weather. Even the washing machine, even the humble washing machine has a pressure sensor in it. I've no idea whether the pressure sensor in a washing machine is of any practical use to to meet trilogy or not. But it just shows, you know, that there are sensors in everything. And obviously some of those sensors are highly capable. The radar in the nose of every civil aircraft is a highly capable radar. You know, and some of the sensors are fairly dumb, but similarly that but even so there are a lot of them, and potentially they're a value like the light sensor in the top of most street lamps. And there are new measurement platforms. Now, what's really exciting here is that things like autonomous vehicles and mobile phones are, or sorry, I should say connected and autonomous vehicles because it's not, you know, increasing number of cars becoming connected now. You know, before before we get to fully autonomous vehicles where where connection will be essential. But the exciting thing here is these these platforms exist in their in their millions and millions of observations is exactly what we need. And looking ahead to the possibility that there might be significant drone fleets in the future, well drone fleets are very exciting as well because they offer the possibility of measurements above the ground, which is which is very important to weather forecasting. So just thinking about whether sensitive is sensitivities in society. If you have a, if the weather has some impact on a system, then potentially it's possible to turn that impact into a source of observational data. And one example there, if you if you know how much power a wind turbine is generating, then that is related to a greater or lesser extent to the to the wind experienced by the turbine. So if we know the power can we can we turn it into a wind observation. And, you know, my instinctive answer would be probably yes. It's something we don't do at the moment. We tried doing this sort of thing with with other weather impacts. It's normally a lot more difficult than you might think. There is potential there. So, here's another example where, you know, when the wind blows things over and including lorries sometimes. So, you think, well, if we if we had access to a data set which showed lorries blowing whether lorries were blowing over or not, could we infer something about the gust speed. Obviously, the answer here is completely different. The answer is probably not because it's probably a highly nonlinear system. And I think physics is just probably just too complicated to to get back to anything about the wind. Okay. So, I've talked about reverse engineering of weather impacts, I've talked about new technology, and I've talked about new platforms. And it's not the way of looking at it. So, you can classify opportunities in terms of whether they're crowdsourced data, citizen science data, third party data or opportunistic. And each, each of these different approaches brings their challenges. The thing I want to focus on for the rest of the talk is probably the opportunistic approach, the repurposing of data, because it's in this area that we've had actually greatest success at trying to close this this resolution gap. And it probably is going to be the most cost effective area of R&D. So what, what makes, what makes a successful source of observational data? What needs, what are the necessary conditions for success? Well, I think there are three. And the most obvious is that the data has to be useful. In particular, is it going to be useful for helping us to resolve more detail in the atmosphere? Because then it offers the possibility of being able to improve the forecast as well. That's the most obvious condition. But the, the second condition, which is obviously important is, is the method going to be technical, technically feasible? Is it going to work well enough? But the last condition is the affordability of the data. And this experience suggests that this affordability question is actually quite critical to a number of ideas. We've had more ideas thwarted because of a difficulty of creating a business model that works than we have worrying about the technical feasibility. Often the technical bit is actually the easy bit of finding new observational data. It's the affordability which, which causes the big headaches. But you can see with all, with those three constraints, it's quite difficult to identify opportunities which, which are going to be work, which are going to work and are going to make it through, right through from idea right through into operations. So thinking a bit more about the data utility, it'd be nice if we had a requirement for some data. So we wanted more humidity data from the lower atmosphere, for example. And then we, we somehow think of a way of, of acquiring this data. Well, of course, with, with this more opportunistic type approach, I mean, that's just, it's just not never going to work like that. So you have to, you get, you know, this is a world of, of second best solutions, you have to accept what data is out there and try and try and make the best use of it. So it's, it's, there's going to be compromises in, in what you obtain. It's, so I've suggested here that instead of humidity what we can, what is probably easier to, to get data on is actually a refractivity of the atmosphere which is related to humidity. And, and it's something which can be exploited. So these, these, this is just the list of the sort of questions you've got to consider when you're thinking about a new data source. Just sort of obvious, obvious questions really to ask. I think about the technical feasibility. I put up here the, the NERC technology readiness level scale from, from one to nine. The Met Office being a, an operating authority, ideally would, would really be looked to get involved at things when, when they get towards the sort of the top end of the scale towards operational implementation. TRL six and above. But, but I have to say that in this, in this area of opportunistic observations and operational observations. I found that there are remarkably few actually groups in universities in UK universities working on this sort of data. And the Met Office has been, we've really been, been forced to do work ourselves at quite low TRL. Well, which is, which is, well, personally, I've found it extremely, you know, extremely rewarding myself because I think this work at low TRLs is, well, I've said that that's the, that's the fun bit in, in my book, you know, the, yeah, no, there is a, there is a surprising, a surprising number of opportunities to work in, in this sort of area that, that it's, it's not a crowded space at the moment, I would say. And yeah, few words about data affordability, aside from our experience, this is, this is often the most difficult thing. The successes we've had so far have mainly been with either finding data that's, that's either open and published, or data that can be intercepted and obtained for free through, through interception. Right. Now, get to the, finally, get to the, the meat of it, I suppose, which is, you know, I've, I've, I've got to describe some examples now. I think I've got two successes and two failures. When I say failures, I mean failures so far that hopefully the failures can be turned into successes one day, but they're not there yet. Okay, so the first success the Met Office ever had with using opportunistic type data was obtaining measurements of water vapor using the Ordnance Survey network of GNSS receivers. And this became operational about, must be nearly 10 years ago now. So what the idea here is that the, the water vapor in the atmosphere changes the propagation speed of GNSS signals from the satellites. And if you have a very good receiver and the Ordnance Survey have a network of high quality receivers around the UK. If you have a very good network, you, a very good receiver, you can measure the, the, the delay in the signals caused by water vapor in the atmosphere. And so we get these delay measurements from, I think it's about 100 receivers in the, in the Ordnance Survey network. And we turn that into a measurement of integrated water vapor. And that's used, that's been used in the operational forecast models for, for a number of years now. So that was the first success that we had. But that is only, that is only 100 sites that's not, that's not massive in terms of resolution. Now, more recently, we've had a, well, I've described this as a more spectacular success. So all civil aircraft are emitting or broadcasting navigational data every few seconds of various types. Some of it for anti-collision purposes, some of it for, to inform air traffic control of their, what they're doing and what they're intending to do. And amateur plane enthusiasts have been intercepting these broadcasts for years. And there's, there's equipment in the, on the market for them to receive and decode these data broadcasts and display them. And you're probably all familiar, or a lot of people will be familiar with websites like FlightRadar24, which use networks of amateur receivers to aggregate the data and generate plots of aircraft positions and so on. But so what, what the Met Office has been able to do is use these data broadcasts to generate measurements of wind and temperature. So using just five, well, I think we've probably got more than five now, but a network of a handful of receivers in the UK is generating a huge volume of data. We're getting measurements from every aircraft in UK airspace every few seconds. And there's, there's an article about this on the digital environment website and I've provided the link there. So, yeah, as I say, we get about 10 million wind observations per day. And that's more than an order of magnitude, more than from all of the source of wind data put together. So this has been a massive, this is a massive source of data. And it actually produced a tangible improvement in the UK weather forecast skill, which is quite unusual just from just from a change in an observation network to produce a sort of discernible improvement in the weather forecast skill. And if you assume that this is a bit of a gross assumption, but if you assume that the the benefit from UK weather forecast scales linearly with forecast skill, then that that's equivalent about a 10 million parameter benefit to the UK economy. So, you know, I would say the methods promoted by the digital environment program really do work, and they can have a, you know, make a very tangible benefit. So, yeah, but the thing is we need to repeat this trick many times over. And it's not proved so easy to do that. So that brings us neatly on to, you know, some struggles. So, as I say, we need to repeat the trick we've done with the with the aircraft data and one obvious area where we lack data is data is data from over the sea. We badly need to do something equivalent in the marine domain. It's recognized it's not easy because over the sea at low levels, I mean, you've got the aircraft flying, you know, cruise altitude and so on, which is fine. But we need data from lower down as well. And there are not so many opportunities for all the reasons I've listed there. You know, it's not like there are no opportunities at all, because one of the things we can see is that there are lots of lots of small boats and around the UK, lots of yachts. And most of those yachts have some meteorological sensors, in particular, a mast head anemometer. So that's Torquay marina, and there are probably about 40 anemometers in that in that picture. And they're good, you know, they're good quality instruments, because they're normally there to help with navigation and the self steering gear. So, you know, they're well maintained, they're good quality instruments. But at the moment, the data doesn't go anywhere apart from stays on the yacht. And there's a picture of Plymouth marina with a similar number of anemometers. And obviously, hopefully, these yachts spend a certain proportion of the time at sea. And so there's potential for quite a bit of data. So, even better, there's a virtually free way of getting the data back to shore, which is the marine AIS system. AIS is basically an anti collision system. But there are slots in the AIS broadcast messages for environmental data. And in principle, it would be possible to interface the yacht anemometers to the AIS transponders. The AIS transponders then broadcast the data, which is then picked up by receivers on shore. And the data can be transmitted to whoever would like to use it. It's not proved to be that easy to make this happen. Because a number of things need to be in place to make this work. And at the moment, we haven't been able to get a consortium together to get as far as, say, a demonstration project. So, but it looks like it's technically feasible. Data utility, probably unknown at the moment, I think it's fair to say. But again, the main thing that we need to work on is the business model that works for the yacht owners in particular. But we're still working on it. We haven't given up. Okay, so another one where we've run into a problem with the business model is this idea. It's actually an idea that's been around for about 20 years. And this was something that was developed probably first in the UK, actually. So this is the idea that the back call links that the mobile phone networks use. So that's for communication between base stations. They have to use to get the data rates that they need. They have to use fairly high frequencies, which attenuate in rain. And if you if you know the attenuation on these back call links, you can then derive the rainfall, the average rainfall or the integrated rainfall along that link. And because there are so many mobile phone base stations, you can actually generate. If you have attenuation data from all the links, you can actually generate a rainfall map. And you can see there these are maps produced in Holland, where they've they successfully implement to this technique. And so, you know, technically, there's no problem. It's been it's been shown it's been shown to work. The problem is that in the UK, we have failed to get access to the data. It doesn't it doesn't come at no cost to collect these data for the for the network, the phone network operators. And at the moment, we just haven't got a business model that works for both the data users and the network operators. So, yeah, a number of people have had a go at this in the UK over the over the years, but we haven't been able to unlock the door yet. So that's a that's another example where of a failure. I think the situation. I think everybody's given up at the moment on this. I don't think anybody's actively trying to trying to solve this. But yeah, with, I mean, with with the introduction of 5G, when probably the density of base stations will be even higher. And the opportunity, the opportunity is growing. So hopefully one day it'll, it will come off. Okay, right. So, so the last section really is, well, I'm going to, there's the next slide is a summary, but also this the question, is this going to work. So, I said that new sources of observational data are really vital to support future weather forecasting in future. I said that I focused on opportunistic data, but the sisters and science as well. And probably we're going to need a combination of all of these to solve the resolution gap. We have some successes. And as I've shown, some of the successes are quite big and they made a tangible impact on weather forecasting, but I have to say I have to admit that on the current trajectory. We're not going to close the resolution gap. And it's particularly when we move to the next generation of weather forecasting models, which are planned to have a resolution of sort of 300 meters or less, around 2030, this the gap will significantly widen. And so if we're going to change things, if we're going to change that outcome, then then we need to do something now, basically. So, so what the Met Office is trying to do is it's trying to encourage an acceleration of R&D, particularly at this in the low and medium TRL level where we where we think that that we need to do more. So the Met Office has a number of formal Met Office academic partnerships, which at the moment, we partner with about five universities, and the partnership mainly covers the field of weather and climate forecasting. So the partnerships are being expanded to include additional universities and the scope is expanded to cover this area of opportunistic observations. So, I mean, of course, we collaborate with with lots of universities at the moment. But there are additional benefits to these these formal academic partnerships in terms of in terms of funding. So if anybody if anybody listening is in a university and might be interested in in investigating these these partnerships, then then I provided a link there. I think that the tendering process is is open at the moment. So it's a good time to think about it. The other thing to do is publicize this as an issue. And just the scale of the scale of the impacts that that can be achieved in this in this area of R&D, which I guess is what I'm doing today. So, yeah, with that, I think that's my last slide. Yeah. Thank you. Thanks for listening and happy to take questions. Thank you very much, Malcolm, for a fascinating talk. There are a few questions in the in the Q&A, which we'll we'll go through now if that's okay. And the first is when when you're taking advantage of opportunistic or citizens science, is there any limitations to this caused by the accuracy of the data that you get from those. Yeah, the short answer is is yes. I mean, take for example the and sometimes you have to do a lot of work on the data to to get it into a usable form. So, for example, with the aircraft data, you know, as it comes the data are unusable because of issues with the data. For one thing, surprisingly, the aircraft aircraft civil aircraft don't know where they're heading to within a few degrees. So we have to correct for heading errors in the in the aircraft. The temperatures are also not brilliant quality, and we have to agree we have to average do a lot of averaging to get to get usable temperatures. So, yeah, there are nearly always issues with accuracy. It's a constant battle. Cool. Thanks. And then then we had a comment followed by question and the comment was just around the the issue with not enough people working on this is the peer review process and the utility of opportunistic sensing divided the academic community for every decade and say funding has been difficult to obtain. And for the same reason publishing has been a challenge, but that's improving. We're world leading in internet of things and AI but then held back by the old ways of focusing on making ever more precise measurements which as Malcolm has said is just And then I guess the question related to that is, is there potential for organizations like the Met Office to endorse opportunistic approaches, so that they can be accepted more as a standard. I think I think there are a number of reasons why why it's relatively difficult to to work in this area. I think one of the reasons is it's it sort of falls between lots of lots of things you know it's it's not it's not metrology. And, you know, it's a bit a lot of things it's a bit of technology it's a bit of the metrology or atmospheric science, it's a bit of data science. And I think quite often it doesn't have a natural home. And it doesn't have a natural home in URI either because, you know, you're not learning new things about the atmosphere in general. So, so, you know, it's that NERC, or is it EPSRC. So I think that is an important issue. And that's what's really good about digital environment program actually, because it does this work does sit absolutely squarely in the digital environment program. So, sorry, what was the second bit? I guess it was whether whether the Met Office can in some way endorse the opportunistic approaches to so that they can become accepted more as a sort of standard for doing things. Well, I think, hopefully the examples I've given shows shows that we are absolutely open to data from wherever, you know, I don't think we, you know, I don't think we have any. I don't think we place any barriers on where data comes from. That's all I can't, you know, that mean, obviously, you know, what there is a sort of, there are different tiers of observations. You know, the highest, the highest, the very highest quality observations you're going to need for maintaining things like the climate record. But we recognize that, you know, it's unrealistic to unrealistic to expect all observations to get to that higher standard. So, you know, we do, we do promote this sort of dead approach where, where what we need is, you know, what we need, the problem I've been describing is all about very high volumes of data. And even if it's not the highest quality, it can reveal, it can be you process to provide useful information for the weather forecasting, the, you know, the day to day forecasting. Okay. The next question was, are there data sets that are a priority for improving forecasts, for example, as high resolution rainfall data going to be a more impact than high wind data. Yeah, there are, there are priorities. And the high priority at the moment is humidity, water data above the ground. We have very few methods of getting humidity from above the ground. There are very few aircraft which have humidity sensors installed. We have the balloon borne radiosons, which are launched from a few stations once or twice a day, but, but there is, there is a serious lack of humidity data. And it's very difficult, it's very difficult to, to fix that, because it's just not easy, it's not easy to measure humidity. You know, it's either the sensor rather expensive or, yeah, it's expensive and difficult to work with. The next question is, what about, wow, because this could be quite dead with lots of enthusiastic amateurs. Yeah, I didn't mention wow, because, you know, I, my, my, I mean, wow is sort of like citizen science type approach. And wow is, wow is very successful. So, for people that maybe haven't come across it. So wow is a sort of a website for collecting data from amateur weather stations. And there are about 1000 UK weather stations in people's back gardens connected to wow. And there are other port, there are other similar portals as well. You know, wow is not, not unique. And it's, it's really good. It works. Because we've got about 300 professional weather stations in the UK. So having 1000 more amateur ones is really useful. Data is, data is used. It's not, it's not used in the operational forecast model yet, but it is used by forecasters to monitor the monitor the weather in real time. And it, but it, but it is limited to 1000. You know, it's, it's 1000s, 1000, and actually the sort of observations I'm interested in the millions. Okay, one, one final question. So for the last 40 years at least forecast resolution and actresses increased in a predictable way. I'm not convinced any that observations are losing in this race. However, with the end of Moore's law we cannot expect such easy wins for forecast model resolution in the future. So do you think this gives observations a chance to catch up. Well, I hope so. Yeah, I hope we're going to catch up anyway just through our own efforts that's all through the digital environment program. Yes, and it successes, but yeah, I don't know. I can't. I just, I just, I don't know, I wouldn't like to say how it's going to go. I mean, I think people have been forecasting the demise of Moore's law for, for a while, haven't they. Yeah, that's right. I think it has, it has demise. Brilliant. Thank you, Malcolm. In the interest of we better stop there but thank you very much for that fascinating stuff and taking those questions. I very much appreciate it.