 Hi, hi folks, thanks for inviting us to this. It's always great to collaborate with like-minded folks, and I've been involved in ESIP for quite a while, and I'm excited to see this collaboration with Australian group start to form and get off the ground. I'm going to be talking about a project called Cords. It stands for Cloud-hosted Real-Time Data Services for the Geosciences. And essentially, it's a project that's funded by an NSF initiative, the National Science Foundation of the US, called the EarthCube. And essentially, EarthCube was formed in 2011. I'll just let you read this description. I'm not going to read it to you. So it was essentially formed to find solutions for software data informatics across the geosciences. And so it's a pretty large initiative. It's been going on for quite a few years, and there are a number of funded projects. Our particular project is focused on real-time streaming data. And so these are things, obviously, there are cases in catastrophes where this real-time information is really important. But also, when you're doing scientific missions, like we do at the lab that I work at, the National Center for Atmospheric Research Earth Observing Lab, we do experiments where we're trying to understand processes better. And so for example, we have a convective system like the one shown here on the bottom right of your screen that we want to position assets around. We have different aircraft that have different capabilities. We have high altitude aircraft, low altitude aircraft with special remote sensors on it. And of course, this storm is not cooperating, and it's moving and changing. And we need to position these assets to optimize their sampling. So that's another application of real-time data. It's also common for the oceans community to do this sort of thing. And so that's really a challenge for us. What we've built so far is a display like this. This is an actual display, just a screen grab. Our G5 aircraft has a video that shows what the plane is seeing. We have these overlays of satellite information, radar information, lightning detection. You can see here the flight level winds from each of the aircraft. And so this is how they're currently guiding the mission. This is from 2012, but the display essentially hasn't changed. The issue with this is it's great for coordinators about the best we can do. But there's really no data here. This is like a closed circuit TV. You watch it, you type in a chat room to try to help direct the aircraft. But there's no data streaming into something like an intelligent system to help them guide these deployments in a much more sophisticated way. So that's where our program comes in, is getting access to the real-time streaming data itself. Of course, we have those large systems. And we've made the case to the National Science Foundation that many times there are sensor networks in place in a region that's being studied by, in one domain, say, atmospheric sciences. In this case, it was a tornado study in the plains of the United States. And at the same time, there was an infrasound network that was deployed in this area that our atmospheric folks didn't know anything about. And it turns out that those infrasound detection can be used for detecting severe weather. And it would have been a great additional resource for us. But we didn't know about these data. And we had no easy way to access them at the time, either. So we've made the case that increasingly, science problems are interdisciplinary. And especially with the expense that we make to deploy all these systems, it's really important that we utilize all of the equipment that's already stationed out in the field to guide the sampling, as I mentioned before. At the same time, there are a lot of these university groups and small groups that are designing their own sensors. In the case of this group on the left, they're at the University of Michigan. They're doing some really cool stuff with water sampling. They'll go out, for example, and they're conserving power. So they'll sample the water depths at a very slow rate until a storm comes. They go out to weather underground, look at the forecast. And when a storm comes, they start sampling faster. So they do this adaptive sampling, really cool and innovative stuff. One of our environment sensing cluster members is part of this. But they're focused on the measurements. And generally speaking, their data is not widely available on the internet. On the right-hand side, we have these weather stations that were printed with a 3D printer. And then the parts are very inexpensive, Raspberry Pi computers and Arduino's and that sort of thing. They're bringing measurements to places that never had them before because this weather station is a $300 system. It's not as sophisticated as a $10,000 net station or anything, but it is giving some valuable information. And again, the folks are focused on the quality of these measurements and not so much the downstream use of them and putting them on the net. And then as Scotty mentioned, there's this whole revolution happening in the IoT world. We did a workshop at Hilo. It was a week-long workshop and people were developing sensors from scratch right there at that workshop and then going and deploying them in the various ecosystems of Hawaii. And so there's just this revolution of inexpensive sensors. There are issues about quality and how you make sense of all this data, but the energy and the excitement from these students is really amazing. And again, they're looking at connecting their data to IoT platforms, but those IoT platforms really are lacking some of the metadata that scientists might use in acquiring and using these data. And so this is a pretty big challenge to bring in all of these sensors. And especially if you wanna try to control the sampling in some dynamic way. And so there are standards that are built for this sort of thing. One such standard is called the Sensor Web Enablements from the Open Geospatial Consortium. And so if you've ever read some of these standards, you'll find out very quickly that it's gonna take you a while to grasp and include some of these standards in your own workflow. So we realize that most scientists don't have the time to do this. They're focused on the quality of their measurements. And so to go and research all of the standards, find out which ones work is just something they don't have time to do. And that's really where our project fits in. We provide a very simple web services interface to what we call a chords instance or portal that allows people to put their time series data in with just a URL. So if you break that URL down, you'll see very simply, this happens to be an atmospheric sensor. You encode all of the information of a particular data point within the URL. So the wind direction is 121 degrees. The wind speed is 21.4. You put all this in a URL. You send it to your chords instance and then one point of data is entered into your system. Very easy to script up for most of these teams. And then that gives you access through chords to some more sophisticated tools like these Jupyter notebooks and on RStudio devices, RStudio environment, excuse me. And so we make it simple for the folks to get their sensors into a chords instance and then get the data out through the same chords instance. We also recognize that getting resources for computers can be a struggle for these small groups. And so we've deployed these instances in the cloud. So basically you follow these three steps, get an AWS account, create and configure your chords portal, and then start sending data and making it available. The very low end chord system fits in an Amazon t2.micro instance, which is free for the first year and then is 50 cents a day after that. So that's for very simple cases. And of course the nice thing about cloud services is you can quickly scale that up to something that's more representative of your sensor network if you have one. And so this is the key. If you put your sensor data into chords, we have this philosophy that your measurements are then born connected. They're born connected to all these standards that you don't have to know the details of, but they're evolving and they're very important for downstream purposes in terms of understanding these data streams. So the ones in black we've implemented, the ones in green are still to be developed. But for example, there's a document called a sensor ML document that will describe what your measurement is or what's being measured, where it's located, those sorts of things. We use standards where they exist, such as GeoJSON for getting the data out. There's also a standard for CSV files that we employ called Geo CSV. And so we try to meet the researchers where they are. Many of our research community uses spreadsheets to do their work. And we don't want to prevent that, but we do want to add some metadata to their data streams that make their data more useful. And then of course, you want to connect these data to very sophisticated algorithms, such as prediction models and archives. It's very important to archive these data. And a lot of the small groups don't have archiving systems of their own. And so we've partnered with some of the larger archives in the earth sciences community to make sure that their streams have a way to get archived and properly documented. One of the examples again of being born connected is there's a Google dataset search now that's been in existence a few weeks. And it uses a technology called the JSON LD or JSON link data and schema.org. You probably, many of you are probably aware of this type dataset. And so a course instance will export the information about your sensor network to Google dataset search. So for example, when you search for real-time weather in Boulder, Colorado, your course instance will appear in that kind of search. And so that's something that you get for free when you use this course instance and start streaming your data to it. I'm not sure we'll have too much time for demo. I'll just very briefly bring one of our systems up. It's these 3D printed weather stations that I told you about. So you can see right here, they're scattered about in the Denver, Colorado area, but also in other parts of the United States and actually some global instances as well. And these are these 3D printed weather stations that are making measurements that haven't been made before. And so we kind of give you a picture, an overall picture of your map and then use what we call a dashboard to kind of give you a sense of the health of your sensing network. You can see that there's some dropouts here and here. And so this kind of gives you just an overview of the health of your network. And then you can sort of drill down and go into the particular site and then look at the individual variables. This uses a technology called high charts. We're all about open source and using open source tools. So you can very quickly get live strip chart plots of your data. We also use a free package called Grafana to do more sophisticated graphics. This particular use is a science museum of Virginia and so they want their temperature and degrees Fahrenheit and miles per hour for the wind, for example. So you can really quickly and powerfully customize your real-time displays. And then you can download these data in the forms that I told you about, GeoCSV, GeoJSON, and so on, very simply. And it can be done with a web service. So that's a very quick overview, but I encourage you to go to the site and play around with it yourself to get more information as I don't have time to go over all of the features. So then I'll just give you a couple examples of where we're using chords right now and the links to the actual instances. As I mentioned, 3D printer weather stations, there are several dozen, a couple dozen locations there. And there are also a group at NCAR that's doing these. These are open source diagrams for the printing of these. So other people can just pick up the diagrams and print them. That's one of the philosophies behind this is that in remote regions of the world, it's difficult to get parts, but if you have a 3D printer, you can print your own parts if something fails. And then, of course, things are very cheap. We have a group that is providing, placing sensors around a volcano in Tanzania and they're streaming their data into chords. One of the interesting things that happens with this group is they have a real-time display of the, these are high precision GPS and GNSS measurement systems so they detect motion in centimeter scales. They have a display that shows continuously in their lab and they were watching the display and they get alerts through chords and notice this anomaly about two years ago that then they explored afterwards. We have another group that is putting hail sensors in the Dallas-Fort Worth area. So if you know radars are trying to, they try to estimate the hail content of convective systems and then these sensors on the ground, they're basically measure hail, hails hitting the ground. And so they're wanting to verify that the hail that hits the ground is actually what they estimated in the clouds. That's a group from the Colorado State University. And then we have this group at the University of Michigan that I talked about previously. Matt Bartos over here on the left is one of our Enviro Sensing student fellows and they did a workshop where they brought in people from all over the hydrology community. And we right there at this workshop, programmed the chips, designed the, not designed, but put together the instrument package. We provided chords instances for each one of these people and then we connected that screen to the archive on this four-day workshop. So from end to end kind of sensor development there to demonstrate the utility and ease of use. So this is kind of the ecosystem we see ourselves in. We see a lot of folks just building their own sensor systems and creating their own instances. But of course, to make sense of this data, you need some sort of services to visualize and map and process these data. And then you want to connect them to other efforts and algorithms and then make sure that these data are archived. So a lot of times we're really focused on this left part of the pink part of our screen, getting more and more use cases of data to prove that it can work across the geosciences. And I would say we're still in the early parts of this right hand part of the diagram. Who are the users of these data? Who are making sense of these data? I would say myself it's kind of early days for that at this time. But as Scotty mentioned, there's really an explosion in the amount of sensors and data that's streaming in these days. So here are some observations and lessons learned from the project. We've definitely learned that there are commonalities of real-time data across the geosciences. You can make a generalized time series interface that really can be agnostic to the type of data that you're sampling. Sometimes some of the hard parts can be difficult to navigate which standard to employ. And in part because each community has their own set of standards. There's one set of standards for the hydrology community that may not match the standards in atmospheric sciences or solid earth sciences and so on. So we've tried to do the best we can to accommodate as many of those as we can. And one really particular example is controlled vocabularies. You'd like to have similar names for things that are being measured, temperature instead of temp for example. And so what we've chosen to do is pick comprehensive vocabularies that then map to these other subset vocabularies. It's, you know, as soon as you start to work with the data providers, you learn very quickly that they have some kind of unique needs. And so you're always kind of balancing their unique needs with what you think the needs of the community are. I'm sure people who run projects like this understand that issue. This enthusiasm among the small sensor community is really infectious. And I mean at this workshop in Hawaii, the students were out there at midnight deploying sensors and ditches in Hawaii and the energy there is just really cool. It's really, really fun to be a part of. And so that's one observation. And then as I mentioned, it's still early days for the use of this sensor network. We have a goal to work with high bandwidth data such as imagery and high rate time series. But those can bring another set of challenges in terms of bandwidth limitations. And so we have some ideas on how to scale that back. And in terms of quality control, it's really this ESIP sensor DAT lab that Scotty, myself, Matt and Renee have put together to look beyond just sampling the data and look at common quality control algorithms. And then the last thing is sustainability is really an issue for projects like this. We have a three year grant. We're in our final year of that grant. We put everything on GitHub and we practice open source philosophies but we need to make sure that this community remains supported beyond the life of this grant. And so that's our project. Here's our team. I want to note that this picture of me is on the left. It was when I had to attend a lot of management meetings and I kind of turn into a hippie. You know, more of a hippie that looks like this. So if you ever see me, I look more like this now. But here's some links to the project and I'd be happy to answer questions and or correspond via email. Thanks.