 Pylos Director of the Historic Preservation Program. And I wanna welcome all of you tonight to this first lecture in our Fall 2020 lecture series. Very exciting to be able to have our lecture series in this new format of Zoom, which is of course has its limitations and we are all in a sense zoomed out, but it also allows us to welcome friends from places that we normally wouldn't be able to. So that's the silver lining. So again, welcome all of you. It's wonderful to see you. Just a little bit of housekeeping before we get started. We will have everybody's cameras off during the lecture and we will ask you to put your questions in the Q&A chat during the lecture or after the lecture. I think it's better if you hold your questions till after the lecture so they don't become a distracting kind of parallel conversation on the side. And then once we turn to the Q&A section then I will call upon some, I will read the questions to our speaker and if we are able, then we can have a little back and forth on the live. So the section, the other thing I need to remind everyone is that this will be recorded. So the lecture is going to be recorded. So keep your camera off during the Q&A. If you don't want to be recorded during the Q&A then please keep your camera off. Okay, so without further ado then let me introduce our speaker for the night. We are so honored and so pleased to have Dr. Deborah Laffer as our distinguished speaker tonight. She's a professor of urban informatics at New York University in the Center for Urban Science and the Department of Civil and Urban Engineering. And she holds undergraduate degrees in art history and civil engineering from no other place but Columbia University. So we're very excited to welcome her back to her alma mater. She also earned a master's degree in civil engineering from New York University and her doctorate from the University of Illinois at Urbana-Champaign. Her work often stands at the crossroads of technology creation and community values such as devising technical solutions for protecting architecturally significant buildings from subsurface construction, undermining, just fascinating. And also because Norman always reminds me that the School of Architecture started in the School of Mines at Columbia. So we'll have a long history to go back to there in terms of undermining. As the density of her aerial remote sensing data sets continues to grow exponentially with time, Professor Laffer and her urban modeling group have pioneered computationally efficient storage, quarrying and visualization strategies that both harness distributed computing based solutions and bridge the gap between data availability and its usability for the engineering community and the heritage community. And her current research interests focus on subsurface data integration with high density laser scanning, hyperspectral imagery and historical data about the built environment and its forgotten remnants as a way to both understand urban spaces and also to manage them. Her work has been funded by the National Endowment for the Arts, the National Endowment for the Humanities, the Ford Foundation, the National Science Foundation and the US Department of Defense. The title of our talk tonight is Historic Preservation through Advanced Remote Sensing Capabilities. We are so pleased to have you here. Welcome to our virtual lecture series. Welcome to Columbia. Thank you so much. We would all be clapping right now. It is always, I think, the greatest honor to be invited back by your alma mater. So I really appreciate having this opportunity as well as everybody taking time in their evenings and what I know is already quite long days in these COVID times to come and share this with me. I have to say that for me tonight is perfect because the two things I like talking about the best are remote sensing and preservation. And here we get to talk about both of them together. So I think without further ado, we'll get started. So we have some time for questions at the end. Let's see, have an advance this here. Okay, so I'm basically gonna be talking about remote sensing tonight. And I've kind of broken up the talk into three groups, the good, the bad, and instead of the ugly, I think the beautiful. So we'll end on a high note tonight. So for those of you who might not be fully familiar what I mean by remote sensing, in my context, I'm talking about line of sight technologies where you have a sensor, it could be active, like a laser scanner where it's emitting something or it could be passive like a camera where it's just capturing what's in the environment. And what we're basically interested in is trying to document things that we can see and making sure that we can see the things we're interested in documenting. And probably the four most common technologies in this area are laser scanning, traditional imagery, hyperspectral imagery, and thermal imagery. And we'll be talking about three of them tonight but mostly about laser scanning. So the good news is that this is sector of the technology that is just taking off or has taken off over the past two decades. Remote sensing in the United States is already a $10 billion industry and it's projected to triple that by 2033. So what does that mean for a small field like preservation? It means that a lot of people are investing a huge amount of resources to your benefit. And a lot of it's being driven by mapping. And what we've seen in the last 20 years is that the equipment itself has gone through this radical transformation. It's become much, much cheaper. It's become much, much lighter. It's become so much easier to use. It has more battery life and you can actually do immediate processing, processing on the fly. So it used to be we would take these big, heavy units out into the field and you'd be there all day and then you'd have to bring them back into the office, run them overnight or even for 24 hours and hope that what you see is what you wanted to see. And now all of that happens completely on the fly. And what I'm showing you here is the Regal VZ400 and this is kind of the Lamborghini of laser scanners. This one does everything for you. It co-registers things. You have an integrated camera. You have an integrated GPS. It is just super easy to use, super fast. And it's got this great interface on the backside. You can hook up your iPhone to it. You can hook up a computer to it. We had one out in the field recently and we were out for about eight hours and we barely got into the second battery. So it just a fabulous piece of equipment. So what is a Lamborghini in remote sensing costs? So this costs about $130,000. And if you think about when I started working this area about 20 years ago, the entry point into this technology was a quarter of a million. And again, super heavy, super difficult to use, no bells and whistles. And what we've seen since then is that there's so much now that's easily available. So we've got our Lamborghini here and now we're gonna see like our Honda or our Skoda. So this is a BKS unit. It's mostly an indoor unit. So this is, but I know lots of people who use it outdoors because it's super light. It's very small. It's got a portable packed foldable tripod that goes with it and just toss it in your backpack. And this used to retail for about $25,000. I just checked it today, it's $18,500. And the prices are just coming down. This stuff's getting smaller and cheaper because there's so many more people that are using it. So as the market expands and technology advances, we all benefit. And then what I think is super exciting is what's happening in the UAV market with drones. So here we see the Velodyne puck. It's been on the market for about 10 years. Entry point started about $15,000. It's down to about 7,000 and it's 500 grams. It looks like a hockey puck and it can go on the, almost the bottom end of the drone market. So DJI Phantom for $400, you can hook this on it. Is it super phenomenal? No, but it's pretty good. So all of a sudden we have things that are within our price range that we never thought about and capabilities that we've been longing for that we never had before. So that's great. Now, the challenges are up to come. So, oh, sorry. So here I wanna show you just some of the data that you can get off of this. This is one of my two study areas. It's in Dublin, Ireland. You'll see the red circle at the bottom. That's where the Irish plan to put in their first metro tunnel and they were going to push it straight north towards the airport, right past Trinity Church in the right and through most of Georgian Dublin. So I moved over there in 2004 to try to capture all this and it was a great experience but they never built the metro. So many years later, I'm back in New York still working on much of the same problem. So here we started capturing the built environment because there's just not drawings of most of these kind of older buildings. When we started in the mid-2000s, the quality of aerial data was extremely poor. So you would get something on the left typically and you can see you really can't do anything with this and certainly from preservation point of view or even structural engineering point of view, not being able to see the facade is not very helpful. And if you've ever taken one of these laser scanners out in the field, on the ground, they're perfect if you just need to do a building, a small site but to take this around the entire city from block to block is a lot. And what we've learned from New York City's recent investment in mobile data is that in a lot of cases, you're not getting what you want because there's so many obstructions on the street. There's scaffolding, there's people in the way, there's large vans and trucks, and then you're only getting the bottom maybe two, two and a half stories. So in my mind, aerial is the way to go. Obviously we have strong restrictions on drones but that may change in this country as well. But certainly the work that my group does from a helicopter from 300 meters or about 1,000 feet and even as far back as 2007, I was able to generate what you're seeing on the right. It's not perfect, but you start to at least get a sense that you're going in the right direction and that you can start to capture the features that you're interested in. So how did we achieve this? Well, coming from an engineering background, we sat down and said, okay, why are we not getting the data we want? Well, part of it's because the scanner is facing down. So the scanner is mounted on the belly of the helicopter or the airplane and it looks straight down. And this was set up originally to do floodplain mapping. And what this means is that the field of view does not capture vertical things very easily. So if you're on the roof, you're on the ground, these kind of areas get captured very well but the facades of the buildings don't. So we started to think about from a geometric point of view, how to minimize the occlusion. So when we develop this overlap pattern where you're coming and you're flying over the structure a couple of times and you say, oh, wow, you're painting the city. It's gotta be expensive. Well, actually the flight time and the cost with that is quite marginal, it's only about 10, 12%. And you'll see that what we get is something spectacularly different from traditional scanning. And then what we did was we imposed our own flight path. And so what this is, and you can see it's kind of a grid. And what it is, it's twisted from the underlying grid of the city. And what this does is it allows you to capture twice as much facade area that you're basically decreasing your occlusions. So why do people fly up and down the street? Traditionally because it's easy. The pilot goes up one street, turns around and comes back down the other. And it was very hard initially to get the contractor to fly the way we wanted. It took quite a while, but after they saw what we could get they were really died in the wool advocates for what we are. But I want to just mention here, if you can see in my mouse, we're actually missing a flight line. And I guess that's kind of the downside of this is that it is easy to miss something like that. And you'll see that Dublin's a windy place and we were up 300 meters. And so these are the actual flight lines of the helicopter kind of wiggling around. So here's the study area. And what you're gonna first see is traditional commercial grade, good quality aerial light are 35 to 50 points per square meter. And it's enough to see some things. We can definitely see the building blocks. We can pick out some vegetation. But for the kind of things I think most people here are interested in, it's not enough. This on the other hand is our data. This is not a photograph. It's not a model. This is simply the point cloud. So each of the little points you see in here is just being represented in its X, Y, Z position in space. And all of a sudden I think you can all put your mind and think about all the incredible things that you could do with this kind of data. And as I said, it only cost us about 12% more than a standard flyover. Now obviously there's some data processing issues on the back end and we're gonna get to those. If you wanted to do very detailed analysis of roof integrated wind turbines in terms of solar usage of your roof, if you wanted to do some right to light stuff, all of a sudden you can really start to see things in a very different way. So you'll see here, we're looking down on that data set. And this is in the bottom. This is actually Christchurch. And again, this is not a photo. This is the actual data. And we've made the point slightly bigger so that they kind of glued together so that they're a little less transparent. And I've got just a couple more pictures of that. But the quality of this stuff is amazing. This was our second flyover. It was done in 2015 and we've completed one last year in Brooklyn and we'll be releasing the data for that. This data set is 350, 330 points per square meter. The Brooklyn data is about 50% denser. So it's almost everywhere. It's at least 500 points per square meter. And you can just see there's a lot of detail that you can pick up on the chimney pots. What you'll notice down here on the streets is all this traffic. So what's happening is you've got latency on these buses. This is not a super, super long bus. It's that it's getting captured multiple times. So it's kind of running together. So obviously the community is often looking to get rid of this kind of data to move the transient data out. And here's another picture of that with the custom house. Some of you may be familiar with end of them. So I'm an engineer predominantly by training. So what do we want to do? We want to put this stuff into computational models. So we have this idea of going directly from the points into the models. And this was a very initial kind of conceptual image of this where we took each of the points and we basically just built a cube around it. And again, you'll see that there's a number of issues with this. This is a completely unclean data set. So you have the double-decker buses coming along here kind of stacked up one after the other. You'll see there's a few stray things in here. The birds flying along that got captured. But you get this understanding that this could be turned into a model that you could use for a variety of things. When we started on this problem back in the mid 2000s, this was the solution. People would take the point cloud that you're seeing up on the top and they would shove it into some type of CAD program. And what you can clearly see and then do some type of fill in. So you can see here that you've got areas that you should be having openings in that you're not areas that you shouldn't be having openings in that you are, such as over here. And it's a big mess. And this was really an untenable solution. And part of the problem was that these CAD programs were trying to fit some type of geometric primitive to this data. And most of these buildings are very complex. They're very ornate. They have a lot of decorative elements. And to be honest, you know, they're sitting out there for 100, 150 years and they may not be that straight anymore, even if they were at the beginning. So we started to work on this problem. And one of the things that I think that distinguishes our group from a lot of the gematics groups is that we're constantly thinking about kind of pulling apart the data and putting it back together in a workflow that's very hone to where we're trying to go with the engineer in mind. So here we're seeing the rubrics. This is the oldest building in Dublin. It's a faculty residence or faculty slash student residence on the Trinity campus. And I guess if you're very lucky, as the faculty member, you get to live there. And you can see it's quite a complex building. So we started with this one. Why make your life easy? And at the time, our data sets, we knew we're not going to be completely sufficient to do this at the level we needed. So we did a little cheat and we combined a ground level terrestrial scan and an aerial scan, just so that we had comprehensive data. We've since gone back and done some of this actually in 3D, but what you're going to be seeing here is just a 2D. So this is the point cloud and it's very, very dense. You can say it's very red. And the question is, how do you start picking out these profiles? I mean, this is really tough. The windows are different sizes. They're different shapes. They've got these rounded elements also in the doors. And so a lot of the procedural modeling, a lot of the geometric fitting techniques that had been developed, particularly in photogrammetry in the 80s and 90s, just don't work with a structure like this. So I was very fortunate to have quite a clever PhD student on this, Iman Solanvari. And he had this idea that he was going to chop up this data and he was going to take it into strips. And he was going to think about the fact that as you come along here, that there's continuity and there's discontinuity. And we can do that in the horizontal and in the vertical. So he basically sliced this building. It's called slicing method and did that. And so that's what you're about to see here. So this is actually real time processing of 2.6 million points. And what he's done is he's taken these strips and then he condenses them into a line and then he harvests the endpoint anytime that line has a break in it. And here we go. And not only do we have a model, a boundary model that looks pretty good, when we put this into a off the shelf commercial finite element program, we get this. So this is the actual meshing of it. And it meshed with no convergence issues, which means we had, we did not at all have to go in and manually mess with it. It meshed completely and we could run this, which is pretty spectacular compared to where we were with that CAD model. Where does this stuff become interesting? So I thought it might be nice to look at the consideration of pedestrian wind comfort. So all of us have had that experience. We've come around the corner of a building and just nearly been knocked off our feet with the wind. This is wind tunnel. And the community that does this kind of modeling generally takes very simple structures like you see in the gold and uses that. And yet we know that a lot of times these smaller features in the environment actually control the wind. And so I've got a PhD student who's doing this and I'm gonna meet Singh and you'll see this initial study that he did just up here. We're looking at just a group of nine cuboids. We're looking at them in plan view and then here in elevation, he's pushing some wind in from left to right. Now, instead of it just being a cuboid here, he's added kind of a typical roof design. So we would consider this as level of detail one and this is a level of detail two. It has a semi-realistic roof. And then all of a sudden we start getting a very different story about the amount of wind and the wind speed coming through and here as well. And now he's gone in and he's made these a little more realistic. He's put on some balconies and some door recesses. And all of a sudden we have a completely different discussion here happening from top to bottom. And so you can imagine that if you've used these very simple models at the top, you are not gonna get a very realistic output at the end. So we're very excited about moving this direction and taking the points to be able to directly feed into these computational fluid dynamic models. One of the other exciting areas is, I think in material identification. So we've been talking about the geometry and now we're gonna talk about the material. Here we've got stand, everybody knows what standard imaging is. So you've got one picture and one set of pixels. In multi-spectral imaging, you'll have multiple openings on the camera and for each of those you will get an image and there'll be different wavelengths. So hyperspectral imaging is this on steroids. So the multi-spectral, this camera has five bands. This technology generally goes up to about 10. This camera cost about five grand. And now you're gonna see hyperspectral. So the hyperspectral unit can have typically up to about 625 bands. And for each of those pieces of the wavelength, you'll be getting this slice of information. It's the same technology that they use in the lab with spectrometers. It tends to be extremely heavy, extremely expensive. When we were working with this unit a few years ago, it was $165,000 and it needed practically two people to carry it. And working with it was really challenging. Shortly thereafter, Coupbert came out as a French company and released this. This is a 600-gram hyperspectral camera that goes on a drone. So this is the equivalent of that Velodyne puck. At the time, it was selling for $45,000. It was the only product on the market. I think it was back in 2014, 15. There are now a number of competitors. They're getting better. They're including more wavelengths. But the technology is still fairly challenging to use because we don't have great libraries to automatically interpret the outputs. But I think that what we saw in the laser scanning the last 20 years is where we're seeing hyperspectral imaging going on. This is really great. The laser scanning's too terrific at the geometry, but not for material identification because you're just using one wavelength in terms of the laser. So here, we've got a brick facade. Obviously, there's something going on, maybe some efflorescence. Who knows? Hard to know unless you're up there in a cherry pick or on some scaffolding. Now we apply hyperspectral imaging to it and we get a completely different story. We don't know 100% what the story is except that these are obviously repointings in the joints and that this is some different brick that brick has been replaced somewhere along the way. But we're heading in this direction. And again, these technologies are getting easier. They're heading in the same direction as the laser scanning. So I'm really excited about this. And it works really well. We did one project with an older structure in Norway and we applied some very, very basic clustering to it and ended up with some very good results. So this was the original building. This was the hyperspectral. This was the training. And then these are the results. And what we found was that the paint that had been used on the lower windows was clearly different than what was finally on the upper windows. But in this case, where we trained on just, I think we trained on just data from the windowsills that it was still able to pick up all the corbels and all the lentils. So that was really great. Now, all things come at a cost. So the bad news is that there's a lot of data. There's an enormous amount of data. The data you saw from the Dublin site is with the imagery and all the metadata is about two terabytes per square kilometer. So what would that be for all of New York City? We're up to a petabyte already. That our capacity to handle this data is not matching the speed at which we're acquiring it. Also, there's not yet full uniformity in the community, particularly if you wanna put different types of data together. You wanna put the thermal with the imagery. You wanna put the imagery with the lidar. You wanna put the hyperspectral with the lidar. There's not a great way to combine them automatically in part because some of them are 2D data and some of them are 3D data, but that's just the beginnings of the problems. You get different resolutions. So the lidar, we maybe have a two centimeter spacing, but for the hyperspectral, maybe we've got a 20 centimeter spacing. So do you scale up? Do you scale down? And then there's noise in all the data. And sometimes you have incompatible spatial temporal coverage. So when we did the building in Norway, we did it with a hyperspectral and a multispectral. And although they were taken minutes apart, somebody in one of the windows changed the shades. And so it really caused all sorts of problems for us. We couldn't actually do the glass classification because of that. Now, this is not my office on the left. It really could be, but it's not. But this is to really help you visualize the data storage problem. So on the right, obviously we have a library. All the fiction is by author, last name, all the non-fiction by duty decimal. We do not have an equivalently great system for our remote sensing data. And it's really challenging because there's a zillion possible ways to store the data. So we started working with something that's quite common in the community. It's called an octree where you take the whole volume of the data and you split it up into these cubes. So you do an initial split across the top in two directions and across the middle. So now you've got eight blocks. And if there is a piece of data in the block, you split it further. If there's no data, it just stays there. And you just keep drilling down. And your storage then is quite efficient because you're really only storing where you have data. And to try to represent this and to teach this kind of stuff, we developed these data flowers. And so basically they're just gifts. But we have here in the left is the original raw data. And now here you see that the storage of that data matches this basically on a one-to-one basis. And with the octree, how many times you go down is your choice. You can have it as a set number that it keeps dividing five times seven times into what they call these children nodes, or you can set some type of objective standard. So if you have a computer and you know that from a processing point of view you can't process more than 5,000 points, then your termination criteria for the octree might be that no cube can have more than 5,000 points. If it has 5,000 more points, it's gotta be split again. But choosing this data structure, whether it's the octree or something else, is not so simple in terms of what works well. So there's a few of them out there that people use because everybody else uses them. And so here we see our octree and it's super efficient. Here's our octree. It also, oops, sorry, does a splitting, but it splits in a less efficient way. And so that you're getting more overlap and so you get a more homogeneous flower. So you can see directly from the representation that it's storing more than it has to compared to our octree. Another common data structure is the KD tree. And what this does is it splits every point from the other. So it splits and splits and splits so that every little box is only a single point. Well, you can imagine that this is computationally super expensive. In fact, to be able to represent this flower, I actually had a decimate this data by 90%. But it starts to give you some insights into this. And I think this is one of the areas that we're gonna see a lot of advancement. I wanna show you one of the issues with noise. Let's see if we can get this to work. No, all right, we're gonna go out and we're gonna see if we can get on to the other screen. Can you guys see the new screen? Jorge, can you see a new screen? No, I think you need to stop there. Stop and start, okay, I will stop. All right, we will try that again. So this was done with that Regal V400. It was done at the corner of Jay and Willoughby in Brooklyn. That's where the 370 Metro Tech subway stop is where NYU has their Brooklyn campus. And we did this at like five in the morning on a Sunday. And yet we will see that we have a huge amount of noise in this. So we have people going through the site. We have vehicles going through the site. And what you have here, because we're scanning and scanning and scanning and the scanner is so fast, this was actually a semi-truck that was actually turning. The data sets quite interesting. It's amazing how much you can kind of see in it. But we can also see some underground stuff. So here we're going into the subway. Let's see, I'm not so expert at this. We'll try to reload that. And we can see in through the windows, we can pick up a number of the floors and ceilings as we come along. So that's all stuff we're picking up inside the building. This was a series of four scans. There was one scan taken from each of the four corners. Now you're going to ask me, what is all this junk up here? And it's not a flock of birds. So it appears to be the reflections that are happening off of the glass and the mirrors of the cars and other vehicles. So that was a bit of a surprise. And obviously this stuff has to be cleaned up. So data cleaning is a major problem for the industry. And it takes a long time unless you come up with a pretty clever way to do it. So we've been working on some of those solutions. Okay, so, but this is sketchfab. This was done by one of my postdocs on Vivo, but this is something that we teach our undergrads and they're really great at it. So it's technically not a hard thing to do. So now we get to the beautiful. So this is kind of the future where I think things are going. So I'm gonna provide us some insights on where my lab group is going, where I see where I think the industry is really going. And obviously people are getting much better writing algorithms for finding things, for connecting things, for cleaning things. This whole data science movement, particularly in the US, really affords a lot of opportunities for that. Distributed computing. I think we're eventually all gonna have to bite the bullet. We did that in my research group when we moved over here four years ago. The data sets are just getting too big for standalone computing. And to deal with that in that environment is not easy because there are not a lot of great tools to work with it. Those that exist, things like Oracle Spatial, IBM pairs, they're really not set up for 3D data. And some of them work okay, none of them work well. And I'm gonna show you some stuff that we're doing. I'm gonna talk about something called per point processing and then talk about some flexible storage options. So we decided since there wasn't anything that we were happy with in terms of a distributed computing spatial database that we would build our own. And we've put a cesium as a front end to this. And one of the reasons, one of the motivations of doing this was not only just the size of the data, but there were things that we wanted to do with pieces and parts of the data that none of the systems would support. So what you're seeing here along the bottom are actually the full wave forms, these little squiggly things, that each of these represents a point. So what happens is the laser goes out and then the energy comes back and you get a distribution of that energy. So here's the helicopter, it's sending out these scans and the points are coming back and you're getting these kind of Gaussian looking waves and then the commercial producers who are Regal or whoever your scanner is are basically trying to harvest that top point. And we know that some things happen that sometimes there's multiple points coming back, sometimes it's missing things. And by having full access to the raw data, so people think about the point class being the raw data as opposed to some type of derived digital terrain or digital elevation model, but that's not true. That the real raw data is actually this full wave form. So we set up this database to start working with that directly. Okay, so we're back looking at the Dublin 2015 data, that data is available for download as well as the 2007. It's obviously very dense. We did a little histogram to look at where that was. So what you're seeing here is that these are kind of the facades we're getting about 35 points per square meter and on roofs and other flat surfaces you're getting about 300 points per square meter. So you're only getting about a one to 10 ratio. So we decided and we've got all this great data and we've got all this great computing power. What can we do with it? So we decided we were gonna do a solar radiation sample simulation and so for every point in the data set, this is 1.4 billion points, we did a solar analysis for approximately I think on average 12 hours per day with two readings per hour for 365 days of the year. So for every time that there was daylight being shown based on records from the local weather authority. And then based on that and this exposure where you're basically doing a ray casting but now in distributed computing, we're doing that on each individual point. So traditionally people would take the data and they would create these models and so you're introducing error, you're losing detail but here we're actually doing it on the point. So for each point, we're assigning it a certain amount of surface area, we're kind of growing it a little bit and then we're doing the calculation directly on it. I don't know with Columbia but certainly there's a big discussion happening at NYU right now about something called embedded PV where they're taking basically solar technology and putting it in the windows in hopes of being able to generate enough energy in that window for all the activities that are happening in that office. And this is a very active discussion that's happening and so something like this becomes very useful. Because we're doing this in distributed computing, we have insights now into this process that we didn't before. So what do I mean by this? So the solar radiation process took four calculations. So for every point it got calculated four different times in four different ways. And instead of throwing out all that data and just kind of using the last piece of it, what we did was we stored all of those interim results back onto the original record. And from that, it turns out that the second calculation that you do is that shadow analysis and we could just take that and do a visualization on it. So again, this is not a photo, this is not a drive model, this is just that shadow casting happening on those points. And all of a sudden from a right to light kind of environmental justice perspective, you can see who's not getting light, what would be the impact of introducing buildings into this area and that on this winter day, so the solar one you saw for a day in June and it's winter one in November that a lot of those small streets really did not get a lot of daylight during the course of this. And one really interesting thing that came out was that we were having buildings that were 300 meters away and only 60 meters high casting shadows into the study area. And traditionally what environmental engineers would do with this was they take the highest building, so let's say the highest building was 100 meters and they say, okay, we're gonna create this ring around this building of 100 meters and that's gonna be our study area, that's gonna be our tile that we're looking at. And yet what we're finding is that again, this is 60 times into 300, that's five times. So it's one and two and a half times greater than what people are traditionally using for the modeling. And one of the reasons that this stuff has not happened in the past is because people did not have the computing power. So I think this will be a great place to stop and to answer some questions and thank you so much for listening. Thank you, Deborah, that was really phenomenal. So I invite all of you to please put your questions in the chat and I will try to read through some of them. And I will start us off by asking you kind of a little bit about just the work, I mean, one can see the implications and especially in the last section of your talk how having this data and asking questions of this data is where it becomes really interesting and you can begin to address all sorts of, as you say, social, environmental, justice questions. The shadows are, there's such a long history of fights about overshadow and shadow castings and preservation that that's just a fabulous way of taking it to the next level. But I wanted to ask you a little bit about your, like the nature of your work. You are gathering the data from commercially available instruments. And it seems that the core of the work is actually computation, trying to compute this data in different ways. And so is, and in some way the automation of computation, some kind of algorithmic process where the data itself can be the site of the analysis. And you're trying to do away with, if I understand correctly, modeling. So the whole process of taking the data and then representing it again in another medium, no, the modeling. So that's where, and I wanted just to clarify that with you, whether that's kind of where you see the future of the field is kind of doing, especially of engineering, going away from the modeling, which is where you began your talk. You began with the discussion about, okay, how do I take this data and make a model out of it? And then you ended the talk with. I think it's more about automating things and about harnessing things that have been available in various communities, but not necessarily for the preservation community. I have a dirty secret to share. Everybody in my entire immediate family has at one time or other made their life as a programmer. And I grew up with this and I wanted to be as far away from this as possible. And so I went and I became an art historian, which is something I loved. And over the years, I've moved closer and closer to computer science. And as I found that the solutions that I was looking for did not exist in the community and people didn't even ask those questions in a community that I had to move my research further and further upstream to create the tools that I needed to have to answer these questions. So where I started with all of this was the problem with tunneling that we didn't have these, we had great computational models, but we didn't have the data to go into it. And it was too expensive to go out and send some surveyors out and model every of the 400 buildings or 1,000 buildings along the tunnel were out. And so what we were producing were risk assessments that were not very good. We were spending lots of money in the tunneling community, monitoring buildings, doing grouting of buildings. And still we were having huge claims. I mean, up to 15, typically, you know, from 5 to 15% of these multi-billion-dollar contracts, in some case. So the Jubilee line in London, the extension there was a two-billion-pound contract. And they spent 25% of that planning for the damage, trying to mitigate it and monitoring it and still they had damage. And then we really haven't gotten too much further beyond that. So is the... I'm sorry, I'd please everyone, if you have questions, go ahead and write them in. So, and I'll read through them. But is the hard work or the, you know, it seems like what's driving it is the undermining, right? You are interested in how soil moves. But you are also interested in now, I mean, where you ended was not how soil moves, but how shadows move. You know, I think academic research is like sailing. You know, you want to get from one point to the other, but you've got to tack a bit. So sometimes you follow the funding, sometimes you follow the opportunities. But, you know, in the end, where we started was that people were relying on a priori data. They were using two and a half D models. There was a lot of manual intervention. There was a lot of parameterization reliance on things that were just not true for historic buildings. And I think we've done a good job pushing the community away from that. And to say you don't need all of this because if you do your data collection right and then you do your data processing right, the information you're looking for is already there. And so they did use our flying techniques when they did the Amsterdam Metro and they used it also on the documentation of Michael Skellig out on the West Coast of Ireland. So I think people are starting to pay attention and starting to open their minds to what the possibilities are of some of these things. I have a question here from one of the attendees. Can you talk about the stakes, risks related to making this built environment data public and any restrictions laws around this new technology? There's questions from Luna Bouganem. So the United States just over the last couple of years started to undertake its first national LiDAR survey and it's much less dense than what you're seeing. But this is a trend internationally. The Netherlands is already on their fourth round of a national scan. All these scans in the Philippines and Vietnam in Switzerland, they're all public access. And to be honest, my 13 year old with his $400 drone can go up and even with his camera can just collect unbelievable data. So it's out there, there's nothing to hide or you can't hide, it's out there. So I don't think there's any issue. The issue is gonna become in the next couple of generations of this a privacy issue. So right now when I scan from the sky, I cannot identify somebody's face and I cannot identify the numbers on their license plate. But the day is gonna come where I'm gonna have to deal with that and that's gonna be a challenge. And how do we, since that is, I mean, everybody's face is getting scanned in order to, or at least identify in order to be able to turn their phone on. And at a certain point, that's going to be paired to these scans. How do we plan for that? I mean, in terms of thinking about social justice and so on, I mean, are we, is there a way to be asking these questions just as a follow up to this? Well, I mean, I think we've already seen some of those impacts with people being able to, document things in real time that they're unhappy with that are happening in their communities. And as these laser scanners get smaller and cheaper and a lot of it's being driven by the driverless car community and that whole industry and that all is running on LiDAR, that how people use this, I think it's gonna be very hard to predict. I think, you know, 20, 30 years ago, if you told somebody that we'd be talking about having areas, at least, you know, some restricted areas where you have driverless cars, they would have thought you were crazy, but it seems to be coming. Do you think that this data, I mean, it being so much data and so dense, do you think there will be a kind of wiping out data that a certain age and then just covering it over with new data? I mean, there's a kind of moving wall of data or, I mean, I'm thinking about archives. Yeah. You know, I don't think that we as a society have come to terms with that. I mean, we even have that problem with webpages now where you've pulled data from somewhere and then you come back a year later, two years later and it's not there anymore. I think these are much bigger questions. I think for us on the kind of preservation side, I think the more interesting questions are, how correct does this data have to be to be useful? So, yeah, and are we at that point where we can pick up crack detection? Are, do we know enough about the hyperspectral imaging that we can not only pull out different kinds of brick from each other, but we can name them? We can obviously distinguish the mortar from the brick itself. We've done some lab work that shows that you can find quite robustly lime mortar versus mortar that's got lime and Portland cement. We've fired bricks and we can pull out the yellow brick clay versus the red brick clay. We've fired them at different levels, kind of an under-fired, a regular-fired and over-fired. And again, we can very robustly pull those out. But what happens when you're not working with virgin material, when you're working with a structure that's been out in the environment for decades, if not hundreds of years, and you've got layers of salt and layers of pollution and things like this. And I think this is where some of the real challenges start to come. In the same way we couldn't use the architectural primitives to describe the geometry because our buildings were leaning over that it's not so easy to characterize these materials from these wavelengths because you've got all this mixing happening. But we're seeing people starting to do this and there's a lot more of these libraries starting to become available, which is exciting. Related to this, there was a question from the audience talking, asking you to expand on how hyperspectral imaging detects the differences in materials. And you're an example of the differences between bricks and mortars. You got into this a little bit. Yeah, it's kind of a lecture on its own. So I'll leave it saying that every material has its own molecular structure. And the molecules vibrate in different ways and at different resonances. And that this produces a signal that is basically being picked up by the hyperspectral imager. And talking about glass and this, because you mentioned the question of in the Norwegian building, you said that the glass, you didn't record it because somebody opened the window. But is it also that glass is difficult to register? I mean, are there certain materials that are not, that just don't register? Definitely, anything that's shiny, the laser scanner doesn't work very well with. At one point, we were taking, when I was back in Ireland, we were taking the solutions of converting the point clouds into these basically watertight models for the finite element, for computation models. And we said, oh, I bet we could do this for 3D printing because that's what they're trying to do. They're trying to go from a point cloud directly into what's really a watertight model to do the 3D printing. And so we started messing around with that quite a bit and had at one point opened up a little 3D printing center and we were quite fortunate to be able to get even a metal printer. And we probably spent an entire summer student intern as time trying to figure out how to coat things like jewelry so that you could scan them and then produce them in the 3D metal printer. And we never succeeded. We tried all sorts of tricks and it was really a challenge. And so when you're looking at your occlusions in your data sets at large scale, some are from this shadowing, the self-shadowing, that you can't see on the backside of a building. Sometimes you've got this, what we call street shadowing, which is one building casting a shadow or an interference on the other. But you also have these holes. And the holes are sometimes because the data doesn't come back from these reflective surfaces, like glass, standing water. Sometimes they do. And there is, we believe that if you're at Nader, so you're exactly perpendicular with the surface, then you're within about plus or minus about two degrees off of Nader. You can actually pick up the reflections relatively consistently. But once you're beyond that, you're not picking them up. Fascinating. So you were trying to coat metals like some dust on them or something so that they would... Baby powder, all sorts of things. I see. Because we knew we could get the geometry, but it was so reflective that we couldn't capture it. One of the questions that often comes up with, like say when people are going to museums and they scan a work of art and make a reproduction of them, is who owns that data? Dad, I'm not gonna go there. But it's gonna come up as a real issue. I remember when I was a student way back in the day in art history, and even back then in the 80s, there was a lot of concern that these replicas that were coming out, even if museum shops were starting to be so good that it was hard to tell in a Truscan piece from a modern piece without doing a significant amount of investigation into it. And we will, we're gonna run into some of those problems. And I think it was Apple that recently released a $200 attachment to their iPhone that's basically a cheap laser scanner. And there was one that came out a couple of years ago for the iPad, and it's basically a depth sensor, which is also what works on the HoloLens. So it's not really a laser scanner, but it's in that direction. So we're seeing this convergence of the technologies. Yeah, I find this, my reading people's questions is a little bit like modeling the data. So I want the data to go directly. So I'm gonna ask Bilge maybe if she can turn, instead of reading your question, Bilge, do you wanna turn on your camera and your speaker and just ask the question yourself? Yes, hi, hello. Thank you very much. Your work is really impressive. I really impressed that huge amount of data that you are dealing with. So my question is like, you are gathering different types of data technologies like hyperspectral imaging, terrestrial and aerial, laser scanning, et cetera. It's a lot of data. And can you tell a little bit more about how we are integrating these different data types together? What kind of programs and technologies are you using? And I was wondering, have you ever integrated this with building information modeling? Have you ever used experience with BIR? So thank you. Thank you. So BIM works really well for new buildings. So about 70 plus percentage of all new buildings, at least all new commercial buildings built around the world nowadays are using BIM. It's perfect for that. Somebody still has not developed an equivalent for historic buildings. So we've seen many people try, but there's been nothing that the community has adopted because these buildings are often they're monumental. So they're already special. They often include materials and decorative elements that are just, they don't fit into the existing, what they call industrial foundation classes, which is basically the structure for the BIM. And so there's just, if somebody wanted to do that, there'd be a lot of work to do. And I remain unconvinced that that's the right direction for us. I think the interesting thing, so our most recent data set that we did in Brooklyn, we paid a lot of extra money and we got the contractor to also fly Hyperspectral at the same time. And so we have, I think the first detailed high density, co-registered Hyperspectral and LiDAR data set. It's not across all the bands that we want. They simply did not have a piece of equipment that crossed all of the wavelengths that we thought we needed to really document the built environment, but it will at least start giving us a place to begin that investigation. The other challenge is the Hyperspectral is 2D. I mean, it's basically a photo. And so what people do is they take these flat photos, these 2D photos and they put them into things like structure from motion and they basically generate a point cloud from that. And then they try to register the two point clouds on top of each other. It works, but it's a lot of work and there's still a lot of error in it. So, yeah. Thank you. Thanks a lot. We have a question from Ximing. One, Ximing, do you wanna turn on your camera real quick and? Okay, okay. Hi, Deborah. Thanks for your lecture. And I'm just wondering, could you please introduce a little bit more about how distributed computing is being used in your multiple simulations like Sunshine and the Shadows? Because in that scenario, I think every point is affecting each other. So this seems very amazing to me how it's being achieved. So as you probably know, because you sound very up to date on this, is that the community's traditional way of dealing with these big data sets is to put them in tiles. And therefore, if you have something in one tile that's impacting another tile, you either have to have a computer that's big enough to handle both tiles or to retile it so that just those elements of interest are being processed on your machine. To avoid those spatial discontinuities, we actually developed a whole new data structure that uses what they call ambient occlusion to create a columnar data structure so that we knew, based on the position of the helicopter and the position of the point, what was the line of sight between the two. And then we did that for every exposure point so that we then only processed those points that might be in the way of that. So we had to come up with a special- Yeah, so we had to come up with a special data structure to overcome exactly the problem that you were talking about about the spatial discontinuity. Because lighter points are dumb. You don't know what they are. They don't know where they are in space. The only thing you have is this XYZ coordinate and you have a similar situation with distributed computing. So let's say you sent one tile here and the other tile here, these two computers right now do not know that these two tiles have anything to do with each other. So we're still at a real baby step with all that. Okay. Deborah, this has just been such an illuminating talk. You've taken a material that is of the highest level and of incredible complexity and have been able to communicate it to us in ways that those of us that are outside of your realm of expertise can understand and understand this long wedge of the future that is coming and how you're in a way leading the way and creating so many opportunities for further research, for our students, for those that are just joining the field, for those that are in the field. It's so exciting. It's amazing work and also amazing to see your own interdisciplinary mind with your training in art history and engineering and preservation, your practice, how you connect the dots, not just the scan dots, but the dots of knowledge and it's just really delightful and so thrilling to see you do that. And I wanna thank you in the name of our program and everyone here for sharing your work with us. Well, thank you. As I said, my favorite things to do is to talk about remote sensing and architecturally significant structures. So this is a great evening. Thank you so much. Thank you. Well, a big round of applause from all of us and thank you so much. Thank you everyone for attending the talk. I see everyone's emotions over here with big clapping. So Zoom needs to add sound to the clapping. Thank you. Thank you, Deborah. Okay. Thank you very much. I'm gonna end the meeting for everyone.