 So good afternoon, and thanks everyone for braving the snow to get here We're honored we have a we're honored to have a tag team of two of our EOL staff for giving today's EOL seminar Mike Daniels head of our CDS computing data and software computing data software and Charlie Martin who's also in the group and Charlie's distinguished by having this year received our The NCAR new car outstanding achievement award your number Six in all of you car or something like that. That's not handed out very often Anyway, they'll be talking to us about new ways to collect data Okay, first thing I want to say is I'm the principal investigator of this award It's an earth cube award. And so this is the team, you know, these are the team members So I'm representing the work of all these great team members we have So with that the court stands for cloud hosted real-time data services for the geosciences And so hopefully by the end of this presentation, you'll understand what that means You know real-time data is critical and growing importance. We know that very well in the observational community You know, there there are real-time measurements of floodwaters Tweets real-time tweets that describe the hazardous events and then of course in you know, well We do a lot of these field experiments where we're you know, trying to observe a dynamically changing Phenomenon and we have aircraft we have radars We have surface instrumentation that we're trying to coordinate around that storm team to take the very best measurements And of course real-time data is really key to that. We need to you know, get a good picture of the situation in order to You know assess it same thing happens in the oceanography community for example So they're kind of episodic real-time events like our field programs And there's this sort of measurement of the ambient condition and you know to detect events that occur So I don't know how many folks know about this, but there's this concept of the Internet of Things It's essentially saying that every many many more devices are going to be connected to the Internet in the future You know your watch your phone I'm a fisherman I I decided to replace my 14 year old fish finder and it has a internet connection And it uses it to you know to download maps and to send position data to a crowdsourced map So this notion that many many more things are going to be connected The Internet is is what one thing that's driving the importance of real-time data But of course watches and not everything can be used by the geosciences community So we're really focused on geosciences things And that that might be you know a surprising area You know things might pop up in in areas that you may not expect for example We were at the AGU last December and there's a fellow at Berkeley developing an application called my quake You know the gaming community is is developing accelerometers and really fancy Capabilities in these phones, but they can actually also be used for things like detecting Earthquakes and trying to help put those data into a another crowdsourced model to predict where they'll go So I think that the Internet of Things is going to change the way we think about The everyday devices that might be connected to the internet Dan Exactly yeah in this in this particular case the application called my quake runs on your phone in the background and Reports data, so you do have to have you know have that app installed and running In fact, he told me that one of the early designs of this Had had challenges in terms of the battery consumption, so I'm not saying these are easy Problems, but it is a different domain, you know with all of these devices connected to the internet So earth cube in about 2012 decided to do domain workshops So geosciences domains being ocean in the oceans the hydrology atmosphere or solid earth sciences and Vanda and NSF Asked what what workshop might be appropriate for our domain you know the atmospheric sciences domain And I decided to think about it a little bit differently in terms of a technical domain that I thought could cross geosciences So we got funded to put on a workshop for real-time data in June of 2013 There's about 70 to 80 folks around the geosciences that came and gave us their ideas their projects And we generated a report on some of the highlights from that report are that there needs to be improved community infrastructure Everything from communications from the sensors to the to the internet to on-demand computing and protocols for data exchange There was very little metadata being generated for these streams They just sort of popped in and you sampled them and displayed them and there was no tracking of the provenance of where these Data came from and you know metadata describing the measurements in more detail The the community felt that we needed tools to integrate and display these data from differing time and space Domains and things are coming at very different rates in very different parts of the world and to have one infrastructure that could kind of assimilate that and and Use that in an effective way was something the community needed and Then along with real-time data comes the actual control of the instrumentation at times So in our case we have an aircraft flying and I wouldn't say control is the right word But guidance to you know to where the aircraft flies is another important aspect of real-time data And of course in in many sensors that can be done in an automated way and EOL has the examples of that as well And then we want to integrate some of these real-time data with You know the folks are actually doing emergency management an example I heard of in an at the ESIP meeting in January We're using UAVs, you know to do a sort of a Photographic survey of damaged areas to try to help the emergency managers target You know the areas that really needed attention and ignore others that looked okay So that's a real strong connection in terms of hazards and then things like that and then the whole area of social media apps and crowd sourcing was another Identified need from this community as I was saying you're gonna have there are many more devices are gonna be connected To the internet in the future. So that's the summary some of the highlights of that report. It's all available online I'd be happy to share that with you And as a result of that workshop We submitted a building block proposal and EarthCube now EarthCube if you don't know It's an initiative that covers the Geosciences They have both building blocks and then what are called research coordination networks, which are more like workshops You know to gather feedback Building blocks are actual tools or prototype tools. And so we submitted a proposal in 2014 with these Areas You know we identified that real-time data has some unique challenges compared to the retrospective data And it's really important to get it right as you do the sampling because otherwise, you know The data downstream is of limited use unless you take very good measurements in the front front end We what we wanted to use a cloud-based infrastructure for this and we Understood that there are many real-time systems around the in the Geosciences. You all has some there's an iris group that does seismic sensors Notionography group also has sensors on ships. And so in those cases, we would fork those streams You know stay stay going the data would stream to their existing systems But then would fork to cords so that it could be accessible by a broader community you know less overhead and the simpler access and One of the things we decided to propose is using some standards this open geospatial consortium has a whole standard directed at sensor webs, they're called so these webs of sensors and You know connect some of our streams to those standards and that would provide some metadata Make those data more useful. That was part of our initial proposal and then What one thing that we hear from our scientists here is they would like to get access to our real-time streams to actually do algorithm Algorithms the compute let's say the next trajectory of an aircraft or where to send the the vehicles that are chasing storms and so That you know the scientists community would want access to the real-time data in order to put you know connect those to their own algorithms and Then as a earth cube community developed it had several building blocks some of them You know dealt with models others were you know post distribution Others were like discovery so we wanted to connect real-time data to those services So that you a person could discover the sources of real-time data that are available through you know through the cords mechanism in the geosciences community And then there there's a fair amount of work being done in this adaptive sampling area where As I was mentioning before you can control the instrumentation based on the inputs that you're receiving so You know doing this in sort of an automated way, and I'll give you one use case for that and then finally You know connecting these streams to these decision support Systems and you know making that making them very diverse. So these are the main topics of our 2014 proposal The partners that we chose to work with Chandra is a research scientist and engineer at CSU. He's doing a Project down in the Dallas-Fort Worth area They they have these radars that are very small. They fit on cell phone towers are called Casa Part of the Casa network and so their project is to estimate the precipitation in a cloud and And then send those estimates to the downstream hydrologists who will then predict what what areas will flood so he had a lot of Exposure and experience with adaptive sampling because these small radars actually track the storm as it moves and You know They'll point in the direction of the storm and then another network of these will take over so he had quite a bit of experience in that area We also had it has it a partner Bronco Kierkez. He is working in a group in Michigan He's he just finishes PhD. It's a small group. That's him and a couple Research assistants. They're doing adaptive sampling with water quality measurements and so one of the issues one of the challenges with these water quality instruments is They're solar powered so they they have to have to be careful with their power consumption and and actually They routinely sort of do sort of a slow sampling Rate to sort of save on power But when that when an event comes a flood event comes or something like that They want to sample much quicker so what what he's actually developed is something that goes to weather on the ground to get forecasts To tell his instrumentation, you know in these strains to start sampling faster So he had experience in this adaptive sampling area and he's also an early career researcher. So he was brought in as a partner Frank Vernon is as part of this Earth Scope project These are seismic arrays that are are moved across the United States And so he had the solid earth experience He also had a real-time system already in place that we could use as a as a test bed for forking data into cords So that was his contribution to the project and Then we had the UAH group who was you know very engaged with this sensor web enablement System and connected to many of these other end-user tasks And so they were enlisted as a partner as well. They had the experience with this open geospatial consortium standards, for example Here in EOL our contribution, you know This is an example of what we what we show in the field. It's a situational awareness display of aircraft We have the video live video coming down from the aircraft during missions We have satellite data radar data overlaid with the tracks Intern and we use this to guide the missions one thing about this display. There's a lot of capability We have a lot of different overlays that can be added including models But the data itself the streaming data from the aircraft is not accessible through this display It's more you know like a close circuit TV that you can watch and you can configure But the actual data are not accessible through this and so that was one one need that we had in this project so we Wrote up a couple geoscience use cases one of them is a evaporation of water and Assessment of its quality in the Great Lakes region. So this would be you know a project that would have hydrology instruments needed An aircraft that would measure evaporative fluxes across the lake Buoys that were in the in the lake itself. And so it was a nice case of Geosciences, you know real-time data measurements that would help the scientists understand this problem another use case was in the seismic area about 2011 Frank Vernon came and talked to us about the the time when they had the Earth Scope network out in a in the Midwest and They actually saw a signature from a tornado from the seismic sensors So that's not something that our atmospheric science community thinks about much But there's some there's some interesting, you know data and use cases that can be developed from that fact And so he gave a talk about that signature and could those sensors be something that our atmospheric science researchers use Yeah, it was pressure Right and but they also have you know the the state parameter pressure, you know data as well. Yeah Yeah, actually they have meteorology little meteorology stations that we could you know tap into and in fact They had a lot of questions about about the instrumentation that we use you know to help improve their quality So we did submit the proposal, but we had we got approval for a scaled-back version It's essentially what we what we decided to focus on instead is just a spoke exposing this concept to the broader Geosciences community that you can actually develop a common, you know real-time data system that could apply to a lot of instrumentation across the Geosciences We're we're starting we propose to start adapting and testing some diverse real-time streams beyond just atmospheric science sensors And then we would test this open geospatial consortium SWE infrastructure with these streams and Then then one other part of the proposal is that we would we connect our streams where possible to these other existing building blocks And is part of the earth cube ecosystem So we began by you know focusing on these very small measurement teams As I mentioned here on the left Bronco Krakaz is a is a recent PhD graduate he has three research assistants and You know they don't have a lot of resources, but they're they're pretty good at the measurements that they're taking They're good at building instrumentation a more recent group that we're working with is this group here at Incar and Raoul and Joss who are developing these Weather stations that are printed with a 3d printer all these components are printed with a 3d printer And they use raspberry pi computers to do the sampling of the data You know you can get a weather station like this for two to three hundred dollars and their objective is just put these in developing countries You know as a very inexpensive way of getting some meteorological information But again, it's a small team with not a lot of resources for the you know for the downstream uses of their data so we started by focusing on these kind of kind of users and You know the characteristics of these users is that generally they're experts in in their unique measurements They're they're frequently looking for funding to keep the programs going they're really not able to focus on this complex standards and OGC infrastructure and things like that they're really focused on the measurements and so they don't have time to you know deal with these complicated Documents and standards and they really lack resources to spend on Expensive hardware and software. They're really focused on acquiring the data But of course they'd they'd likely see me a benefit of the more more use of their data So that's the the nice part that we can add with courts is to expose their data to a broader community So here's the general model for courts what we just what we had come up with You would go to something like the Amazon web services a cloud provider You'd see a little acronym that I had to paste this in but this is our vision An acronym for cords, you know getting your own cords instance You would you know download that instance it would be running in the Amazon web service It actually fits in what's called a micro instance, which is free for the first year after that's 50 cents a day Is that right so very inexpensive? They don't have to go to their IT department to ask for resources or create a virtual machine They just do it on the cloud and they configure their own portal with their instrumentation, you know describe it Add as many sites as they want and then start you know getting their their data coming in So that's the the three-step process and actually it turns out Obviously there are steps in between each one of these and the first one to get the Amazon instance It's about eight steps. Is that right something like that and then Similarly when they configure their instrument they have to type in information about what they're measuring the variables and that kind of thing And then they get their cords portal. So that's the basic concept a Couple of the reasons that we decided to use the cloud and virtualized servers is scalability You know if you have a very large network, you can scale that up very easily everybody knows that's one of the benefits of the cloud and And and you know that you don't have to access your friendly ID IT department It's your instance. You can you know configure it at your own will The the basically the the cloud services are ubiquitous. They're all over and so it's something that Yeah, you know is very common and will be commonly used and then there's this notion of DevOps where the developers are also like the sysadmins You know for the for these systems and that makes the cloud services makes that very easy to do It's easy to deploy software in the in the cloud Which is what one thing we found out and then These development environments can be generic enough to work on on many different cloud providers. It's pretty agnostic to what cloud provider We we we would use and and then that makes the software easier to install on many different systems so when you're When you've got your court instance going the next thing you want to do is start sending data to it And so we've given a couple examples in in Python and in W get But it is as simple as typing a web address with the variables and the values in your web browser and of course Most scripting languages can can send an HTTP request. I'll break that request down a little bit so you can see what I'm talking about You know that that first host name is your portal address You have an instrument ID, you know if you have several instruments at your site and then you the wind direction is 121 the wind speed is 21.4. Those are the actual values that That you're inputting into the system So you type this into your browser and you've just entered one point into the system So, you know, what would you what you do is connect that up to your sensors have it start spitting out HTTP requests each second or each Some period of time and and that's very simple for folks to do and this is how you get your your data into the system Yeah, then and in fact We have we have quite a bit of experience with communications But that can often be the weak link, you know weak link if you don't have cell phone coverage Or you know or something like that, but of course you you could use a satellite connection if you want So, yeah, we do assume that there's some basic data communications Capability and in fact, I think we've helped help them understand What's available in some of our some of those cases, but yeah, you do have to have a basic internet You have to have a connection in it. What do you want to say? Sure Yeah But that is a good question you have to have basic connectivity, yeah Okay They're related actually yeah one one one issue is if you just had the URL Then anybody could send you know a point To to your courts portal and so the key is is a security you know mechanism a unique key You know for your portal that you have to include We we just scratched the surface at security, but but we did build in some and we know that that's going to be Something that we want to expand. Yeah It is Yeah, exactly and that's that's part of one of the things we had to do in the scale back version We you know we wanted to do more spatial data, but in the scale back version We focus more on the time series data. Thanks for that question So we came up with this idea on our own we thought we were pretty smart and we were you know in the project and Blown behold there are tons of companies that are actually doing this They're they're really working on it from the perspective of the internet of things, you know the toasters and they're you know Who knows what else and so but there are print plenty of these you know around To highlight sort of the distinguishing features between us and them these commercial groups Is the portal is completely owned and managed by the science team these other services, you know They have a server that you send your data to and yeah, we'll give you your data So they like to centralize, you know the services And and you know our and ours really lends itself to being a distributed network model and not a monolithic server We're not work our software is on github. It's open source. So it's not tied to one vendor or company There's no monthly fee other than this fee I told you about for the hosting of the portal itself, which is around 50 cents a day after after a year It's an open source project So you can actually download the source and add some new capability if you want to And then it implements OGC standards, which are really important from the geosciences community There's this thing called sensor ML which basically describes a sensor It's just an xml object that describes a sensor and cords will output That information so that it can be registered as a as a as a sensor that's available for the geosciences community And then of course we're connected to all of this infrastructure that earth cube brings in terms of You know exposing all the real-time data sources in the geosciences and that sort of thing So we're really focused on geosciences and really the academic kind of environment So work that distinguishes us from these commercial outfits And with that I'll have Charlie do a demo so you oh, sorry. Yes Very good question We have we have a the portal each portal has a database running And it has limitations on the number of points that it can accept. It's a million. No, is that more now? more than that now But so so what we're not an archive, you know, it's not an archiving system But that has been a common request to somehow connect this to something. That's also Recording an archive and so on the on the database limitations. We sort of flush the database Periodically and then start over so it is an existing limitation. Yeah, yeah, yeah, so the portal has a logins logins and passwords, so You know that that's all again within the control of the team They you know, we think we give them a default admin account and then they can add as many as they want after that It's been with going out not in right. Yeah, that's a good point. I'll help you Eric Yeah And the other thing I'd say is We're not I mean that it's open source and it's contained It's a package that can be moved to another cloud service or even a virtual server So there's nothing in the architecture that prevents it from running, you know In other places if that we're to become an issue No, right I yeah, we're headed that way, but we don't at this time Okay, so let Charlie do a demo Thanks Mike that was a great description of the project I want to emphasize too that Mike told you about our initial vision and proposal and then the fact that we really got seed funding for this project and it really is had maybe Six man months of effort to get to this prototype demonstration We know that there are lots of capabilities that we want to to improve on and add to it So but but the idea is out there and and being productively used by by some of our friendly testers This we have a nice website which describes the whole System and and how to get started with it. It's called cords RT cords real-time comm which actually goes to get hub and It this website is a good place to to to get an idea of how the project works and Get ideas about how you would Spin up your own portal and so on Mike was talking about the steps that you use on Amazon to to create a portal and and it really I've got Eight steps here, but most of them are just click-throughs. So it's really pretty easy to do and Once you get a portal Configured so this would be your own instance that you've created on Amazon Then you can go to the new website that it has created. It's created at both a web server and a database connected with this it's built on Ruby on Rails and you'd log into that and This is what the the front page of a portal looks like we call it the dashboard This is kind of the default place to go take a look at what your portal is doing and this gives you different Time span display of measurements the number of measurements that have been coming in from your instruments and The the idea is at a glance you can tell if things are working or not This is not the date itself, but just the number of measurements that you've gotten This is kind of our test bed portal that we're putting some We've got the three and car weather stations going into it plus the weather station in my house and It gives you an opportunity to kind of look look around so the The dashboard is really the first place that one goes to look and see how things are doing Mike mentioned that all of the data goes in via URLs the green banner up here at the top Shows you the most recent URL that's come in with data. It's just sort of a sanity check to see how If your system's working, let me get myself logged in here and the advantage of a login is that then you have administrative Access to the site never Okay, so Now that I'm logged in as a as administrator. I've got more access. There are more options on the left side here We have just a few kind of concepts In the system there are sites which are places that you will have instruments. There's a listing of sites Okay, I can do that. How's that? Is it good more? Okay, good so here's a listing of sites and It's very dynamic at this point because I'm an administrator logged in as one I could I could add a new site here create a new site and Here's a kind of visual map of the sites that we've got running right now. There's the Wyoming Supercomputer there Once you've got sites set up you can define the instruments And this is the list of instruments. They're coming in and some some data about how many measurements we've gotten total on the sites and For each instrument You can go then and configure that This gives you some information about it and you see here. This is the The area where you will define what data streams are kind of come in you define a short name V1 for instance Is going to be what's in the URL that comes in we go back here You can see the most recent URL that came in for this instrument There's V1 equals 9.1 V2 is equal to 80 and so on and they mean they mean different things I'm going to go back to the to the list of instruments And we'll take a look at one that has variables that make more sense So here you go. This is a the weather station at foothills lab And you can see the short names that were assigned to the variables And finally there's a graph at the bottom that lets you take a kind of look at your data and see if things are Looking reasonable or not This is just a standard web Component graphing component from high charts and it'll update automatically one of the nice features we've got is that when a research group is setting up a portal like this They can Run a simulator and For all the instruments you've made it that gives you the simulation capability if I hit this button right here It would start sending simulated data to the portal so you can start sending simulated URLs And that way you can test everything even before you put an instrument out in the field to make sure your portal seems to be Configured properly So that that's a really useful. Let's talk a little bit about getting data out of the portal There's a real simple interface for for grabbing data in different formats like For the foothills weather station, I'll go get the CSV data For the last day did it run Excel? There you go So that's the that's the data that came for the last day You could you could choose different days Function of trimming the database Mike talked about if the database gets too big The infrastructure doesn't handle it very well in these small instrument instances. So It's looking like about three million measurements is what works on our micro instance right now with Giving us a good enough response time They're also on the web pages a little bit of documentation about building these URLs that you would use You can go over here and you can see this is an example of the kind of URL that you would use to put data into the Into the portal and this is how you might get data out of it and let's just try one of these Take this URL and I'm just going to paste that into my browser and There's the last data point that we received So you can do that from browsers, but you can also put those kind of URLs into your code. So you could Be writing Matlab Python or any kind of programming language you want to shell scripting You could write web pages which issue these URLs and grab data for whatever your application is so envision that a small science team let's say a Graduate department has got a project going somewhere and the graduate students could put together our sorts of scripts and activities that could be monitoring and producing analyses and so on Tailored to their kind of research this portal here is as I said, it's kind of the test bed that we're using We have about half a dozen portals going now with friendly users. Here's another one. This is the 3d weather station printed weather station that the RAL folks are are using it for and literally was a very Convenient collaboration with them. They they had a weather station that they wanted to put out in the field their workflow model was that someone would go out in Namibia or Zambia every month and plug a USB stick into it and we said well, why not just do it in real-time? cell phone is everywhere in Africa and That was a great match for them because now their instruments can go online Automatically then get Exposure and distribution of this data immediately So I've been really excited about working with this crowd So I think that's about all I've got to show you. Do you have any questions? So right now It's more governed by how big the database is going to get you could put data in at a one sort of at a one-second rate if you wanted but Right now that that would overwhelm the system right now right now our database is has one row for each measurement and we'd like to Improve the capabilities the for that kind of higher data rate, but right now I'll be thinking of stream gauges weather stations Seismic sensors profiling activities sounding things Yeah Yeah, the low-rate data Yeah Yeah, you could that that's an area that we could we could look into Yeah, this is um data from the g5 from from orcas, okay So so the reason there are gaps is because the aircraft flies And and this is a little bit bogus I didn't have time to organize this this is all 102 variables that are coming down Which I really need to break up intense different instruments Right now having to pick one of these guys out Doesn't make sense But you know it's an example of it's been great to demonstrate how easy it is to get to fork data from other sources and bring it in here So I I didn't have access to write software that runs on the aircraft on our g5 But I knew where the database was here So I wrote a little script which queries the database it gets triggered by the database on the ground and puts data into this Instance and so we've added you know a bunch of different instruments and and projects set up portals for them Absolutely Right, yeah, and actually we've talked to Gordon about putting a Chords hook in the night is what you'd be really trivial to do Oh Yeah development time just staff, you know No, not at all no and and we keep hearing that that they want archive as well You know it was really designed as a communication scheme that let people get real-time Observations out and out of the world on the internet easily and They're saying yeah, we want that and we would like archiving So that's that's something we'd like to put in but it's beyond the scope of the current project I So absolutely you could if you look at this This is This is not a good example If you just send a data without a timestamp it gets Stamped with the time of that that it received it but you can add a timestamp and I'm gonna show you that that field So the point is that you can put data in after the fact or before the fact. Yeah That's right That's right Yeah, I guess this one's not doing that either Yeah, let's go to the URLs I'm just thinking that it wouldn't block you from putting in something that's before the That's right. No, no you can specify the time Forward or back. It doesn't matter. It just goes in the database with that timestamp. That's a good question Director Click on Yeah, and actually Actually, this is connected to the to the UAH Information but we haven't explained that very much. I think what we found in this project is kind of Developing this concept. We got a lot of users interested in sending their data to it And so we configured the portals. We made that very streamlined We we tested the implementation on a variety of different kinds of sensors So that really took most of the most of the time of the proposal But we have connected our streams to some of the UAH OVC Sweet it's a full sweet implementation Yeah, that and there there's really a lot of promise there That's one of the distinguishing features for us is that we we want to develop that more to where a hydrologist can bring up a portal Easily put their data into it and say I can also get it out to all of these other bigger services That I know nothing about, you know, we're not there yet, but that's that's one of our goals Okay, yeah one last question So this is is really the front end and would be feeding systems that have that capability It's not built into this capability and we really wanted to have something that's simple That's really comprehensible for the small researcher and then the ability for it to couple seamlessly to bigger or more Capable systems brings us right to the architecture How do I get over Okay, so Yeah, and that you've hit on the architecture for the core system And as you can see the all the portals on the on the left So these are the things that the instrumentation's teams primarily would be focused on Making sure their measurements are coming in the assessing the quality It's really common for those groups to download CSV files, you know to do some processing But then those are connected to more advanced services, you know, just the just via the URLs that Charlie talked about for processing translation mapping visualization and then Whole set of earth cube building blocks would follow some of these OTC standards So what you're seeing today is just the left side of this graph But really we have to get the data in order for the for these downstream things to happen And we really wanted to prove that this General concept could apply in lots of different kinds of sensors, but that's certainly a development That we want to pursue in the future. Thanks for that question. It's good Here's an example of I did talk about the fact that we were putting our data into the OTC SWE implementation by the UAH group and Can hardly see it, but you'll be able to see some of the Water gauges overlaid with radar data. This is the Geo Explorer application. You may have heard of that follows those standards So I think the next question that we get asked is why should NCAR, you know be involved in this I'd say, you know in a much much smaller scale We can develop services that are agnostic to the types of data just like the FTP and there can be documents It can be data can be images And so, you know, we feel that we can develop a system that's agnostic to the type type of data That's applicable to our data in in the broader geosciences. That's of benefit to us We can by being the involved in this development we can shape, you know what it looks like and you know Make sure that we have some of our requirements into that System development This also makes it easier for our scientists and engineers at NCAR to access our data right now There's a range of ways to do that. Some require database expertise Some have other software that you just download So we really wanted to target this very simple approach of just URLs, you know to get access to the data So it it makes those data more accessible to NCAR scientists and engineers here and then You know, this will this will be a way for us to gain awareness of other Operational systems as we do these field experiments in EOL for example It's very common for us to tap into existing mesonets that are sampling data So this will give us some exposure to maybe these these networks of sensors that we didn't know about But they're using cords the for example and then There are very common Challenges in terms of deploying sensors around the world for especially with communications and and things like that signal processing and so this kind of builds this network of Scientists and engineers that are that are doing that that we can tap into and they can tap into us, you know for actual deployments and Then what we really learned a lot about during this project is experience in how to deploy Software in the cloud and actually the advances that have been made in visualization through a web browser That was something we that Gordon had dabbled in About the same time that this project came up But this is going to prove to be a very valuable tool and skill set for our future displays And I really think this is a development that's appropriate for a national center. We're you know, we're really focused on real-time Measurements that's not Uncommon within the geosciences community and so it's something we can share and lead the community in So I have some that's my presentation. So I just have some questions that I'd like to pose to the group especially You know providers and consumers of real-time data, but What are some other sources of real-time data that we might need or might be useful for for field experiments? Jeff I Was thinking about using this as a way of tracking file names And so that there'd be a time series of our granules people would find that those and That's similar to the use case so the NOAA fellow to ask us about right with the radar data Yeah, that's good. Good one. Oh great. That's a great Other ideas good You've got these sensors in various parts of the world You'd like to cope to locate Data that would be comfortable in space and time Such as for example radar data that was co-located with the precipitation gauge precipitation gauges in your network, but the radar data is also accessible But you want to be able to tie a specific piece of radar data to that geographical location Nice Benefit both sensors That sounds like sort of an OGC It does. Yeah Other Charlie. I just wanted to I forgot to mention that right now. We're strictly focused on time series data, but a few simple Data types that I think will give us a lot of leverage would be some imagery capability You could post an image Modern-sized image every few minutes or something like that That would be a common use and then also some kind of profile or sounding Data type is another very common sort of ray type of data People find very useful Yeah, we had well, sorry And then if the need arises you just turn on the switch and you start collecting all the information for emergency No, thanks flooding big storm like we had yesterday or whatever An instance that would just pop up and then start collecting the data you needed to further advise decision makers, etc I was gonna say we Have interacted with these some of these small research teams are operating UAVs and it's primarily imagery It's what what I can at least the people we've been talking to but And they and they use it for emergency management for example, Chris Okay, thanks Great, thanks for that now on the consumer side You know I what one one thing I would mention here is that Mel Shapiro a scientist here gave a talk about What what if we could take the aircraft in situ data and feed that into a Forecasting model that then predicts in real time where the storm is going to be you're using using some of those data It was an interesting concept and of course some of the some of the constraints were In getting access to the data, but this might open the door for something like that What are some other scientific needs for? algorithm development or models Real curious what people think Chris Going out on a spatial basis Okay draw a box Terry Eastburn Terry Eastburn stem seed and With the idea the course can be a part of the K through 12 science outreach and It provides Which would be cloud and computer science and web Technologies, it would be just perfect fit for high school wait, and we and we were Partnering with this 3d printed weather station group for example so that they the school could get involved in the design and the printing You know 3d printing at the school and connect a raspberry pi up to it And then then have the data and analyze the data with something like quartz Now that I think about it, even outside of schools and such One of our projects, we may know that we're going to have a field project somewhere And without knowing the things that are available out there, we could potentially go out and put out a query And say well, again, if we want anything that's in this spatial domain, show us what we can get at Right. Yeah, that's a good, yeah That's what we envision especially with these discoverability of some of these sources Scott. You had your head up Had any given to all mean like a uber for in-car shuttle I There's commercial applications like this for real-time tracking of buses in airplanes and where they are Yeah, good. Yeah, it's a good one And we're getting we're doing a lot of that in our field experiments with the mobile Vehicles for example, but those data are pretty much in a closed system. Is that right Eric or? Okay at this time. Oh, those are great. Thanks and then I think we touched a little bit on this but We know what are the other what are some other? Data sources that were that would be helpful in real time and useful in real time imagery Was one we we talked about this notion of especially with high volume data, you know It's finding some way to derive products that would be useful To scale it down that in a way that could be more manageable And of course, this is something we do in the in our field projects all the time We don't give people the full Bandwidth because we can't send it But any other ideas on on other types of real-time data? We might look at Yeah, right power spectrum Oh, there you go For products. Yeah, you know, that's kind of a slippery slope about getting into the analysis side of things because then you start I think one thing I'd say is perhaps the analysis teams, you know develop the products, you know Maybe maybe they take the spatial data from the from the cords portals in the area develop the products And then those go out through another cords instance. It may be something like that That's a good idea with an open source I think a real-time perspective and that is that a lot of the things that we're trying to do in neurology is we're trying to forecast what's going to happen in a specific place and You're talking about real-time data if I can access the forecast for the time That is now from different models. I can see which model is is providing me with the most accurate result And that's just a lookup sort of thing You've got things like weather stations or aircraft or whatever But there's a Geographically spaced and they're sampling at a given time a real variable You want to be able to compare that with the model very model of the different kinds of model output to see which models is producing the right output and so this kind of a structure Lends itself to being able to do that for a variety of different locations If you do it for one location and one model you can do it easily for the dozen locations Observations I was thinking of maybe that if there's a time series It's very high temporal resolution that you do like a for a transform break into components and just store the That way you have sort of a time series at a much lower temporal steps That's a good idea What they did with the BAO two decades ago Geosciences that's a great example. Yeah Ideas about how we distinguish ourselves from the internet. Yeah, right the types of things we're trying to do with that data quite different Those are great All right, thanks so much for that feedback That's all I had you know for the presentation to check out cords RT comm and Here's the email address for the for the team if you have further questions, and I really appreciate your your suggestions and Thanks for being here