 I do encourage everyone to move closer, but that's up to you, no pressure. This is one of the big science case studies, so hopefully everyone is here ready for some big stuff. I'm very excited about this, I will talk a little bit, I have the remote, I don't know how to use it. I'm going to talk a little bit about who's on the panel, what kind of presentation you can expect, and then I'm going to get out of the way and let the people who did the real work actually talk. So we do have on the panel, Inigo Sangil, Inigo is the project sponsor, he works at the University of New Mexico, yeah, he's a smart guy. Dave Reed, Dave is a senior engineer at Lullabot, Dave is going to talk about technical challenges of the project, Sean Busque, Sean is a senior engineer and team lead at Palantir, Sean is going to talk about agile development process. I'm Ken Rickard, I'm the director of development and professional services at Palantir.net, and I'm here because when Inigo came to us with this project, I got really, really excited. I think more so than anyone else he talked to, because it married the two things that we're really, really fond of, which is large data sets and helping the world. Now that's really, really corny, but it's actually true, and I managed to convince him that we could do this job for him, and we've been very, very happy with the results. So that's pretty much my whole spiel. I will tell you a little bit about how the presentation is going to go. Inigo is the project sponsor, he's the scientist on the team. He's going to give you an overview of the goals of the entire project, including what we're doing in terms of ecological and biological diversity studies. That's really, really good stuff, and it will ground the technical and process parts that go along with it. The basic point, and then Dave reads, you know, talk about the technical stuff and how you solve specific problems in Drupal, and like I said, Sean's going to come in and talk about using agile as a process. And what I'm hoping you're going to get up from this, how many of you are not using Drupal currently? Well, never mind, we're done. We're not done. What I'm really hoping you'll get is a sense of how you attack a project of this scale and the scope, and how we think about, as, you know, developers and, you know, project managers and all, how do we attack these kinds of projects in Drupal? So like I said, Inigo is going to set it up, Dave is going to talk about the stuff that made him go, oh, wait, we have to do what? And then Sean's going to talk about management a little bit. So I'm going to pass it off to Inigo and be back when it's question time. Thanks Ken. That's a great setup for the talk. You know what is going to happen. I'm honored to be here in DrupalCon. This is almost like, not almost, this is a dream to be here in front of you, explaining the work that we do, and embracing the community that we came to love many years ago. We started, I'm going to give you, like Ken said, a little project overview, and especially with some emphasis on the research team goals. After I set up the background, then I'll talk a little bit about how to bring all the project sponsors together, and we'll give a few words about sustainability of the development process, and I'll hand it off to Dave Reed, which will show you really cool Drupal stuff. So the first thing that I would like to do is tell you what we are. We are the LTER, the Long-Term Ecological Research Network. It's an international organization that has, that is in 42 countries. Each country has a network of its own where they monitor ecological processes that go in our planet, and the goals of this organization is provide a scientific community and society with the best predictive knowledge about what is going on with our ecosystem and the ecosystem services that are in the planet, the biodiversity, the water that you drink, the air that you breathe, the food that you eat is so basic. Because I know more about what's going on in the US, in the Long-Term Ecological Research in the US network, and that's what I work in Albuquerque, New Mexico, I'm going to focus a little bit to give you an overview of this network. Congress gave money to this initiative about in the 80s, and now we are 26 sites, that is to say 26 places where we conduct long-term ecological monitoring. It spans from Alaska, the north slope of Alaska to Antarctica, going through the tropics. And there are, as you see, 18,053 scientists associated with this network. And we have 34 years worth of data. That's a continuous data, that is to say, we started recording those data many years ago, and we will keep doing so when we are all dead. Now, there are challenges. Many are legacy challenges. One is that each member of this network receives independent funding. So they act as an independent entity. And the biggest challenge is that we have 26 teams, IT teams, spread throughout the country, West Coast, East Coast, the center, the deserts, the boreal ecosystems, the Arctic. And yet we are doing very similar things. However, there are differences, and we have been doing and reinventing the stuff that we have done over the years, since the 80s, before some of you were born. Now, let me walk you through what a single side of this 26 in the US has to handle or has to deal with. Let's go to the north of the slope of Alaska. You drive from Fairbanks about 10 hours. You take a left on the Dalton Highway, where the pipeline goes, and go to Tulik Lake. Right there, there is the Arctic LTR at John Hobbes, set up many years ago. And there are right now about 122 persons conducting research. This is a very rustic place. And there are 247 projects. A project might have multiple data sets and a whole infrastructure around it. There are publicists right now, 974 data sets. They are online. You can grab them. Our mandate is to put all this data to the public so you can make the best use of them. There are 2,400 parameters. In this one site, one parameter might be the air temperature at 1 meter above the ground. So that is 50 million records since the 80s. Some go back 100 years because before they were LTR sites, they were already collecting data. So now I hope that I give you a scope of the problem and the challenges that we face. It's not that it's large in volume. It's also spreading diversity. We look at many different parameters that are related to the health of our ecosystems, the things that make us leave. Let me go and give you a little flavor of some other projects. And then I'm going to walk to the Drupal stuff, which is the coolest thing. Now let's go to New Zealand and then fly due south and land in a nice round way and then get a helicopter and go to the dry valleys. This is in East Antarctica, the McMurdo dry valleys. It's a little camp at the base of the Taylor Razor and the Taylor Valley. And that spills over a primarily ice-covered lake called the West Lake Pony. And so in collaboration with NASA, we deployed an autonomous robot that can navigate by itself without guidance under the ice in the lake and explore what is in there. This is a prototype to see what kind of stuff go, for example, in planets like frozen oceans like Neptune or, you know. And so it is a prototype also to examine life in hostile environments. So it's not just the LTR does many more things. And as you can see, it also has the long horn in there from Texas. I wanted to put this in. But we do a bunch of stuff. We just conduct surveys in the ocean. We deploy drones. Here is from the jornada. We have our information manager from jornada LTR. We study also the effects of the humans in the ecosystem. I mean, most of the idea of a virgin and lost forest is kind of romantic, but it's no longer the main stream case of our planet. So but as we want to use in a responsible way the services that these ecosystems provide, and we study that. So but our team, the team that use Drupal have a slightly different mission is provide these ecologists and the biologists that study our ecosystems with an efficient and affordable information management system that can withstand the long term the changes in technologies that can survive when we are all gone. So we need to manage the content. That means manage the content that we need to do. Allow you to discover the data and to make good use of it and serve the information that is distilled from this raw data and the knowledge that that can produce. So it's not just about data. It is our news items that are blogs. We have a K to 12 educational program that is outstanding. We have bibliography records that we have to manage a whole shebang. So the first thing that came to mind is how we are going to deal with all this. Did I say that we don't have a lot of resources? That people, you know, we are a non-for-profit. We are based in academia sites, in educational institutions. I probably failed to say that, but we don't have. We are not super wealthy in that way. So this is a nice way about the project. This project had many participating institutions. Most of them are universities in the US. So in all the icons that you see put some sort of support, mostly money, but also time and sweat and tears. And the main sponsor is somewhere over there, the National Science Foundation, but there are many others. There's the UPR in Puerto Rico, the University of New Mexico, the NBL, the Marine Biology Lab up in Woods Hole, and so on. But really, having all these institutions work together, that is not trivial. That is actually impossible. It just only happens because there are people behind it. And in this case, you have all this great team. You have Karina Gris. You have my best early adopters, Eda Melendez, who's in the room over here, who believe that we can do this, that even though nobody believed that this can be done, that we can use the same system to manage all this content and go all hand-by-hand. They supported us. They adopted without blinded. And then we have people in Taiwan, Mayru in Taiwan here, David in Israel, Jerusalem. We have Ken, we are based in Albuquerque, so the guy, I don't have a pointer, but in the lower right, there is a guy with a nice note with a 4-on-1 T-shirt. And that's the guy who got us into trouble. He is the guy who said we should be using Drupal. And we were all using different solutions. And he argued why. So why Drupal, really? It's just because it's open source. I mean, yes, in part it is. But there is many dead ends in open source. This is a healthy community in open source that is working, that is passionate, that go out of their way to make things happen. And it has a forum that the first time is accused, the first time that you visited my sounds like Chinese to you. But over time, those things and being walked by mentors, all those things will make sense to you. So we thought that this would be a reasonable solution. There is, of course, many other ways. Drupal out of the box might get you 80, 75% out there. Then you need to sweat it and make it closer. And if you harness the, you convince your group that this is a viable solution, then you can go ahead and get the best people in the industry and help you to get and nail the details that are so necessary. So this is where our great people here are in the table with me coming to place. There are many, of course, many other reasons why we adopted Drupal. And I'm not gonna go into details because it's just intense. It is a humongous project, it's live, we are still developing, we are going forward. So let me talk a little bit about sustainability of the project. We started this in 2009, Drupal 6, it was beautiful, it was great, but we didn't know much about how to best position the project so it would be easily maintainable and not a mess. So this is one of the things that we learned from this great group and we just put ourselves in GitHub, learn about the workflows. We use Drupal, it's a very sustainable system, a future looking system, and of course, contributions of people like you. We use about 80 contract modules. All the code that we developed is out there, we are transparent, we give back to the community, every patch that Dave came up with, whether core or for a contract module was posted and we are happy to give back because we receive so much from your community. So this is from, well, my community too. So this is what we do, I would encourage you to contribute, even if it is one commit a day, that goes a long way, even if it is less than that, one commit a week or just an issue, a test case, a sprint, that's something that will keep us all going together in moving forward our project. Our long-term goals would be many, I have short-term, mid-term and long-term goals, but basically we keep continuing helping the researchers, the people who can provide the predictive knowledge about the health of our ecosystems. But we can also, we want to add users and developers and if you cannot donate your time or your expertise into helping us grinding some of the issues that we have, but you know somebody who has funds to advance this project, please come talk to us because we really need, and this is a very exciting initiative, it's fun, and it is really helpful for all of us. So with that, I'm gonna give it to Dave. Dave, as you know, is from a different planet, but this is the diversity of Drupal, allows us to embrace people like that, and he has done marvelous things with all of us, and I'm forever grateful for all the help and the many times I have gone out of the way to accommodate the needs of the project for us. So thank you for that. All right, hi everyone, thank you Inigo. So I'm, you know, right in the weeds with Drupal, I'm a Drupal developer, it's what I love to do, and this was a really interesting project. We went to an onsite in Albuquerque and had a great time eating all sorts of chilies and all kind of good stuff, and had a really good chance to be a part of their team and just kind of learn and absorb, and I saw a lot of challenges. I give them a lot of respect for the Drupal 6 install profile they had already made, but as I saw their team and Kristen, one of their information managers, who's like a content editor, essentially a role, worked through the site and what she usually does during the day, I saw just several opportunities for like, maybe we could improve that, maybe we could help make that better, and make this, like I always like come into a project thinking like, I want this to be a joy for this client to use, like I want them to be happy when they're using this after we leave, and so I hope that they, that's true, but so I kind of wanted to walk through some of the problems that I saw and we recognized as a team and kind of talk a little bit more about how we approached them, how what we used in a more specific way in Drupal. So one of the first things I saw is that there's lots of, I mean it's a scientific project, there's lots of interconnected data and they were doing a lot of data entry, I think there's like, there's a data set content type and then there's a data source content type and then there's a variable content type and then there's people as a content type and there's like all these four different things that would have to come together into essentially to represent one complete data set and they would have to enter it in four different forms and then like back reference them, remember what the title that they use, that kind of stuff and it just, it felt like it was a suboptimal process and one of the great things about Drupal 7 is that there's a great module called the inline entity form that if you use user reference modules to kind of relate things together rather than having to type the name of something, you get more of a UI like this where like you get to be able to do it right in line where you can add something or refer to something existing right from the same form and it actually goes really, it goes interesting, like I'll show a little quick animated version here, I think it's animating, yeah I can't just see it on the preview, but like you can go through and edit a person or the data source just right there and you can add new ones and it's a really, it slows things down just a little bit, I will admit, but I think the benefit of having all this from administering right on the same page and not having to go back and jump around, it's really helpful I believe. So that's the inline entity form, that's a really great module to use. You're right. So yeah. Another really interesting challenge that we had, I talk about this variable content type and I'll kind of show, it was kind of complex and it took me a little while to wrap my head around like all the parts of it and everything, like I'm not a scientist by default, I work with Drupal, so but eventually we got a hang of it and I was kind of studying this process and kind of evaluating what we should do in Drupal 7 and so I kind of want to show what the form looked like in Drupal 6. So we had a variable form and it had one tab with like a name, label, all the basic information, maybe like a description or definition of that variable and what you considered missing values. This corresponds to like actual column of data in some kind of collection. So there's like this first tab, there is a second tab, if it was a physical quantity, like it's something that's actually measured, you would fill this part out with like the unit, minimum, maximum, precision, all that kind of stuff. But if it was a date, you didn't want to fill this out, you wanted to fill out the next part with like the format of the date or if it was a code, like a key value kind of thing that corresponds within associative fashion, you'd list out the codes, like you type in the key and then a pipe and then what it corresponded to is like the label. So like, I felt like we could do better here too. And we evaluated a lot of solutions, like we could have made a content type with fields but kind of put them all in the same form. You know, we could have done like inline entity form possibly or made our own custom entity type. But actually we made our own custom field type. That seemed to make the most sense. So we made a deems variable field type. So anytime you have like this complex interaction or user interface, you know, if you have a developer on hand, don't be afraid to make a custom field because this I think really helps streamline things. So I would really like to show how this works. So we have this, the variable type and it's all on one screen now. So you've got, you know, your name, label definition, you can type in a code list, you can pick what type of variable it is and it will Ajax populate like all the stuff you need to fill out based on that type. And it's just a, I found it really, I mean, from a developer I found it easier to use and I hope Enigo would agree that it's much easier to use as well. And I like, I put a little summary thing up above so like you could collapse it and hide that information if you needed to which is kind of important when you're working with a lot of data as well. So yeah, we found that very useful and that worked really well. So don't be afraid to find something custom. This was this fit that use case where nothing really worked out of the box. So we really enjoyed using this and it was fun to develop too. It was a fun challenge. Another fun problem is reusing data. So for instance, you've got a set of data that you're collecting and you collect it for 2005 and then you wanna make a new data set but for 2006 and record it down. You're essentially reusing the same stuff but you maybe just wanna tweak some stuff or add and put in some new data that was submitted but you wanna use all the same kind of metadata about it. So what we did is using the inline entity form module there was kind of a patch that was languishing in the issue queue to add a clone button and we revamped that up and got it ready to go. And so that works really well. So that's just, you can hit clone and it clones that data source and then you can edit it with the inline entity form module and it works really great on the box. Another thing that we developed with the variable field type was how do we allow people to easily reuse variables they've already entered before? Cause again, say you're entering something very similar you wanna kind of reuse a variable you'd already used rather than trying to go back and refer to it it's now a field type so we can't really do it it's not an entity. So we did something really kind of cool and interesting. Our new field type, the deems variable field type we actually integrated it with the core search module so anytime that you filled out a data source that had variables we shoved those variables into search so that you could search by keyword on them and so we actually have this like search field inside our variable widget so that I believe if I have an animated version if you typed in you can easily search and it shows you like a little summary and if you click it it automatically populates all the fields that you had used based on what you selected and that way it's just an easy way to reuse what you'd already done and it was just, I don't know it just it worked out of the box with search. So, okay. All right, another interesting challenge was I mean Inigo talked about having large amounts of data as well and they were not lying like these files that were coming in from the field were gigabytes or larger and if you have ever tried to use a Drupal upload field with a large large file like that it tends to not go well so it was kind of just a small challenge here to figure out how we could best handle that like we could have them upload it separately okay let's go with that but then how do we refer to it in like the Drupal UI how do we pick it and there's this great module called file field sources that kind of has some options like I want to refer to a file that's a remote file or I want to use a file that's already uploaded somewhere and but out of the box with file field sources if you use this like option for I wanna pick a file that's already been uploaded it would still try to move the file to your files directory with a large file that does not work well again so we actually contributed a patch back to file field sources that if this directory you could pick from was already in your Drupal files directory it just left it alone it just said here's all the files that are there and so they could just be field feed upload straight to their Drupal files directory and pick a file right from there and that worked really well and we also integrate with the chosen user interface library for select fields so if this list was really long they would actually see like an autocomplete and they could type in the file name and be able to find it without having to scroll through the entire list so that was really nice and I'd probably say the biggest challenge for this project was how to represent and display this data so there's obviously like the HTML version of showing all this data off but it's a scientific project they don't always share things in HTML and there's actually this format called EML or ecological markup language am I saying that correct? Yes. That they would display all these data sets and it's essentially XML but a subset of XML with added fields that kind of stuff and we essentially need to represent the same stuff in EML and the problem is we have all these four different content types yes we're doing a better job in the UI of connecting them, showing them all on the same page but now we've got to actually display them as essentially XML data so how are we gonna do that? How are we gonna tackle that? And in Drupal 7 there's a new terminology called view modes I don't know if anyone's heard of view modes before a few, okay good so view mode is like just a name like a teaser or full page it's just a kind of a way to build a standard set of how to display an item like a node so like how your node displays when it's viewed as a teaser and that's the teaser view mode and there's actually an RSS view mode in Drupal Core out of the box and so we made an EML view mode and out of this EML module and basically kind of centered everything around that and we made specialized formatters so field formatters for all of our data to output what chunk of XML that would need to do so like our variable field type we had a formatter for that that would output it how we needed to have it in the EML and we connected all together with templates so I'll kind of show just a little bit about that it's gonna get some code, don't be okay it'll be okay it starts out in our EML module with EML entity view so this is the standard Drupal 7 hook that gets called anytime an entity node, user, et cetera is rendered and it gets a view mode parameter there as you can see and so we say if we're going through the view mode that matches EML we are setting a special theme key so rather than using like the node template we're kind of overriding it and say let's use the EML template which actually works out pretty well like Core lets you do this so we had a base eml.tpl.php and then we had an eml that corresponded to a data set so this is our like eml-node-dataset.tpl.php and it's recognizable as XML but we're doing kind of the standard things the Drupal does in templates which is render fields we have some hard coded things but not too many and an interesting thing like down this last line here the field data sources so we're rendering the data set and then once we call this render field data sources we're going to then render the associated data sources within this data set so we're starting to nest down and render and continue down to all those associated things and it's really nice because our form I would say well render the data sources using the eml view mode and it does the same thing so it's a much more flexible way of controlling this output and I really have to say I was impressed with Inigo and his team because they kind of took this framework for rendering these data sets using the eml module and they've actually duplicated it and they're now rendering in another format and they're kind of set up in a better way and to be able to pivot or shift in case a new format comes up or they need to some government sponsorship or contract requires you to be in a standardized format they can easily do that so I think this has set them up for success and in long term success which is appropriate since it's a long term ecological research project so another fun thing was some of the long term stations had set up like a data explorer where you could actually go in and like actually view the raw data like you could say show me all the records from this data set from 2005 to 2010 and you could see it in the browser or you could download it as a CSV file but this one station had done this feature and we're like maybe this is something cool we can have for everyone it sounds like kind of like a thing people would find useful and we discussed with Niko and his team and they agreed and so we kind of it was interesting because this independent station had basically copied all their data into a separate database which was then used to display all this data on their website not in the Drupal database but a separate one so we kind of abstracted that functionality out into a schema reference field so it's just a nice little field that lets you refer to a specific database and then a specific table assuming you have it set up in your settings.php listed as an official database to Drupal so if for some reason they have made this data set in an available in a database they can refer to it and then we did a whole module with like making all of it exploreable using the variables and kind of connecting it all together but the schema reference field made it possible so it's kind of an interesting one if you need to refer to external stuff. Another great challenge was this was a Drupal 6 install profile and all these sites that had installed were working on Drupal 6 but we were working on the Drupal 7 version how do we get all these sites to Drupal 7 because the reality is it's an install profile so everyone's kind of tweaked or changed the way that they've done things maybe added some content types removed some fields there's no real enforcement for what they can do so it's kind of a wild west of install profiles so it's this challenge of how do we migrate all these sites and handle that? Well, our first step was to basically identify all the stuff that was in the base install profile we wrote a migration for that that's kind of our base migration so all the content types that are supported out of the box we use the migrate module which is this kind of a sample of what you would see as a migration so this migrates people in a content type from a Drupal 6 database but you don't really have to understand this, it's okay but that's our base migrations we started with that we had it written and so yeah, we got all of our content and common stuff done and then we actually added a hook, a hook alter very appropriate for Drupal it's what we do in Drupal 7 that says, hey, we've got our base migration here's all the classes and what they all refer to I'm gonna run this alter hook does anyone wanna change it? and this is actually really really useful for all those individual sites cause they can say oh hey, that person content migration I actually have to tweak a couple of fields I'm gonna override that class and I'm gonna do what I need to do but still have the support of that base class so for example, Sevierda is one of the research sites and they extended the person migration and you can kinda see with like remove field mapping a site nope, we don't have field user account we don't actually have that on our site we deleted it but we do have a field user account that should come from this data and they're just able to override it very easily they don't have to copy the entire migration class they can just do what they need to change and it just kind of works out of the box and it works for their site so if you ever have a large multi-site install or an installation profile like this Migrate is a great tool and I would encourage you to add some support for being able to alter it like this and just we also did some fun things with it being an install profile with the setup process it's a nice, Drupal can allow you to add forms to the actual installer so we actually added some steps like oh hey, you wanna do some EML configuration for some of the things that are put in every EML output so what's your station acronym? Each of them have a package ID to kind of identify that data set and we allowed them to fill this out on install and we also packaged everything up using features as well and one of the nice things we kind of did was in the installer we added a step for hey, here's all the optional stuff that you could enable do you wanna enable any of these before you install the site or they could do it later too so like we have the data explorer I talked about our Drupal 6 migration variable field search they could all enable these right here so that was a fun way to enhance it so if anyone knew to this install profile it would be, I think it would be helpful and of course in Ego mentioned it's all, we're all developing this on GitHub so it's open source, anyone can contribute I can still contribute to it and I have occasionally so I would encourage anyone to check it out we also then mirror it to Drupal.org to get packaged as an install profile but GitHub works a little bit nicer for development so yeah, that was a really fun process like I don't know, it's iterative process of finding these problems how can we figure them out that kind of stuff it really works well with our team it was really great to have the onsite and kind of see what their team is doing and I think by the end of the project I considered myself a Drupal scientist so it was great but to kind of talk about that process that our team had I'm going to invite Sean up to talk about that so people who came in a little later after Canada introduced us as Dave said, my name is Sean, Sean Bousquet I work with Palantir.net as one of the senior engineers and team leads and Dave and I were we had the opportunity to work together along with one of our other in-house developers a lot of also side consultants within Palantir to be able to help teams and we also got to go and visit them and so what I'm going to do is I'm going to talk about the process and Dave talked a lot about how he used different pieces of Drupal or technology or programming to be able to solve very practical issues of making data easier to enter those kinds of things before I start actually talking about the agile process which we use I will brag on what Dave architected a lot of what he's shown you may be specific to Deems but honestly this is really a dataset manager with variables that you guys can add control or change so whatever your field may be if you're collecting data and datasets or managing them that kind of thing Deems is a profile to start with and to even look at maybe some best practices really also though the way that he architected it with what is the output the output is extremely themeable so if you don't use environmental markup language you have something else for your own sites your system whatever you're doing it could be easily extended or changed package it in features the way that he did really it's a way that you can choose what you need and or replace what you need specifically for your site and as he said it's up on GitHub and we hope that you will take a look at it so let's see this is the agile process current slide so as Dave kind of you know said there was process to this as well and really when you look at how why do we use agile it helped us solve some some particular problems with all the number of constituents that we had as Indigo talked about a lot of universities two thousand scientists all over the world twenty six sites how did we solve some problems would that come up with that kind of thing I'm gonna cover the agile process that we used I'll highlight a few points along the way in the process maybe give an antidote or two and then we can have some deeper questions later uh... so those of you who may or may not be familiar with the agile software development process I'm gonna read a quick definition for us uh... you can definitely find out more just google agile you'll you'll be able to find a lot of good material I'm gonna provide this definition so we all have a similar understanding because sometimes agile may mean something different to different people and what it was for us is it's an iterative and incremental development approach where the requirements in the solutions evolve through collaboration and there was a big part of this project was a collaboration between self-organized and cross-functional teams and there was definitely a lot of cross-functional teams that work together to be able to produce this as an agency at Palantir but also for this particular project agile helps us meet certain goals or provide certain benefits to us including speed flexibility where we're able to pivot or and or be able to add on to things later as new priorities come up that kind of thing responsibility and we found uh... both on this project and others the right people have the right responsibilities uh... and have that can make the decisions when we need to and the developers take personal responsibility for a lot of pieces of the project and creativity uh... iterations allowed us to try things and not be so afraid uh... you know we could try things early on in the project at a time where the risk was less and uh... we had in fact you know i'm gonna give some antidotes here is one of them was there was a particular feature that an ego was really interested in taking a look at but we weren't sure how much level of effort it was going to take and how long so the agile process allowed us to actually explore that feature in one of our sprints which is one of the iterative processes that agile uh... helps you with each of those circles that i showed you earlier is like a two-week sprint and uh... in the first two-week sprint of the when the idea came up for using this uh... and ego said go ahead and research it if it's going to take longer than x number of hours which for us turned into a level of effort or what are called story points it's going to take longer than that than we don't want it well it was that we were able to explore it try it roll it out because it was actually going to take quite a bit longer however there are a lot of other features that we got to be able to explore and say yeah this is worth coming in so the agile process allowed us a lot of creativity as well biggest benefit that uh... for our company why we switch over to agile and why we brought it to this project is we want to deliver the right value at the right time what agile does is it's not unplanned because there is a plan in place but it is flexible enough and allows both our clients and our developers and the whole project to be ready to be able to focus on a new priority in the next iteration or series of iterations if we need to and there were times that that occurred on this project if there was a key demo that was coming up uh... you know an ego might point that out to us and say you know i'd really like to show this section to people i know that the backlog which was the order of priorities or things say the next sprint is going to be about this but can we move it around so sure because what it that our process of the agile process allows you to do is to be able to to work on what's providing the most benefit at that time uh... we're not switching in between sprints uh... you know a sprint is kind of frozen therefore you don't have to worry your developers you know simply have to move from day to day on what priority was but in a very managed uh... agile process way in the next sprint we always readdress priorities readdress what is the order of the sprint lock which i'm gonna show you a little bit in a second to be able to uh... determine as a team what's the best value for our product at this time and there's a lot we can say about that it's a it's a very good thing in terms of helping to drive your project and provide value one of the things i do want to call out and like i said i'm highlighting different parts of for this particular project how did agile help us and in this case agile help this by one defining a product owner what made it work and what really makes it work with our clients when we can get this kind of uh... activity is our product owner which was an ego was very present during the entire process and that's sometimes hard to get but if that's what made this project an extreme success is uh... you know he would be in uh... different places in the world i think you were in alaska one time and he would show up on our stand-ups uh... that kind of thing and uh... providing that feedback on a regular basis being aware what was going on with the project uh... helping us us to sort out just anything that we needed he was there for us uh... and that really helped also uh... you know there are a lot of voices during this project you know you gave us some good uh... you know feedback we talked with a lot of people uh... throughout the network and who are users of this uh... and some people asked for various things and an ego helped coordinate with the stakeholders uh... and explain to them well here are the priorities right now here's what we're working on yes we'll try to get it in that kind of thing and he just he served to help keep those constituents happy uh... as well as sort of keep the amount of noise we were hearing as developers uh... to what we needed to be focusing on on that time uh... so that was just a great benefit is uh... if you do the agile process which maybe you know hopefully some of you do encouraging that product owner to become that very strong uh... very present person was was very great okay as part of our agile process we have uh... at least on a few projects and we're starting to practice it more and more what we're calling sprint zero what this allowed us to do is sprint zero was before we actually went on site we received a lot of information in prep for going to the onsite in new mexico and and meeting the uh... deems team in the leader team honestly there was a lot to wrap our head i mean around our heads around to be able to understand not just i mean it was hard to understand what they did the scope of what they did was just absolutely amazing and then the technical aspects uh... of really the details of their data kinds of things they needed to do and having the time we we took about two weeks maybe it was uh... four weeks total but uh... we had a what we we had a an official two week sprint zero uh... and that absorbing that information prepared us for going on to the onsite and we didn't we still hadn't known or gleaned everything we could for the project but what it did allow us to do is a lot of times when you go on site a lot of it's just information feed and spooning uh... you know giving you that that stuff i felt like when we came uh... to uh... new mexico as a team we were at least ready to go we understood some of their domain language you know that their custom information uh... so we are ready to be able to talk as a team and do that and having that sprint zero allowed us to do that so encourage others to try better to build that in your process especially when there's uh... a large amount of information and also i think sprint zero's point out that agile is not about not doing discovery agile is definitely about doing a discovery uh... we did it in iterations so it was knowledge acquisition is a typical ticket and they uh... an agile process and then we did an onsite intensive almost like required for a project like this uh... yeah you know we we can do a lot of things for google hangouts that kind of stuff uh... but going and meeting the team uh... having the team present the entire time i think we were there what three days yeah three days it was intense uh... but an ego had grabbed uh... a lot of the stakeholders brought them into the same room uh... we conducted agile training for them and uh... you know i don't know if you've had clients or even uh... yourselves if you've looked at the agile process it feels like there's a lot of information but actually once you get into it suddenly clients begin to adapt it and speak and you know speak in that language and that kind of thing and it really helped us because conducted the agile training and it gave us all a common language to be able to speak in uh... you know particularly with there's a part of agile process called user stories and uh... that just creates like benefit statements from potential types of users we're going to use the site and uh... we began communicating through user stories the day one of the process and in one kind of funny an anecdote for it is uh... we passed out index cards we did use flat straight index cards which by the way again side note in terms of experience you know having the actual index cards and writing them down was more beneficial in it kind of like the communication flow quicker rather than having a sort of stop and have someone enter it into the computer in the computer the big pain afterwards we had something like three hundred user stories to enter in after the onsite but it made the onsite a lot more flexible uh... by just doing it old school and put it on uh... index cards uh... we also have the team uh... carry around index cards during the entire uh... onsite uh... so they had them in their pockets so that they took a break or if they were back at their deaths for a while if we're at the bar having a few beers uh... we wrote several user stories uh... at the bar uh... it encouraged basically that we're thinking about the project the entire time and if something came up they had a quick place to be able to capture that idea and i'll tell you a lot of good user stories came out of those uh... more ad hoc times not only did having this agile process and using user stories and conducting the the agile training gives a common language uh... but it helped us also we began to use the client's domain language itself uh... we learned what they we learned to speak their language a little bit and they wrote those in their user stories and uh... we were able to translate into the technical side can have we doing on time five minutes okay the biggest the reason for the success uh... on the onsite was a lot of the stakeholders were there in present as well so uh... as i kind of mention before we had the user stories uh... kept cards there for the entire time uh... and also we were kind of we had great hospitality and uh... great coffee uh... during the time and uh... that helped us for the success so i'm going to kind of quickly cover uh... the ideal agile uh... two weeks sprint and this is really what we did use on the project as well is uh... starting on mondays we had the backlog in order and if you're familiar with that you're going to encourage you to go take a look at it if you're not having the backlog in order before you start on that monday having the client already or the product owner already prioritize the uh... the needs you know for prior to that week and kind of it stays in order as developers coming in that week we were able to choose new user stories be able to discuss what can we get done this two weeks according to what the the client already set in terms of priority we'd have a pretty kickoff where we met as developers first without the client present uh... and you can do this anyway you can mix it up our pretty kickoff so we're really a place for the developers to come together and do a little negotiating of you know you think we can get the sixteen pointer done or can we move this out and do something else etc after that uh... pretty kickoff we would do uh... an actual kickoff meeting uh... with the team and with the product owner with uh... a couple of key stakeholders although the product owner was the primary speaker for it and we would say hey this is what according to your backlog this is what we think we can get this done uh... get done this week does this still meet with what you're interested in uh... and they would confirm or deny a lot of times they would say oh i just got a phone call from Puerto Rico and there's a new priority uh... that kind of thing so we might adjust it a little bit we also maybe use it as a time to get final pieces of information before we froze the sprint log the next day should i stop here and let some time for questions so and basically we use the general agile process because we are running tight on time and we want to give you guys an opportunity to ask any questions and we're going to stop here and can we moderate uh... a couple of questions for us and we'll use up the last it's like three minutes we have for questions so again thanks uh... these slides will be online you know after this so uh... you'll be able to get the rest of my presentation you know with that so i hope you enjoyed and member deems is out there please download it check it out use it have your developers if you're not a developer get into the code because there's a lot of great good practices thanks so if there are sorry about that on time if there are specific questions there is a microphone we ask you to use it i should also say that there is a survey they're going to ask you to fill out here's the very quick version of the slides and i will also say if you don't get a question answered from us come by the palantir or lullabop with david lullabot we're a palantir we can answer questions there too so please go ahead i said a quick question uh... what were your budgetary constraints for this project budget uh... four million dollars and it's interesting the budgetary constraints of the primary reason we went agile so that we could make sure that within the budget that they had we were delivering the highest value yes are there any uh... current policies that this is plugged into or do you have any goals for having this impact on policies in the various countries the i-l-t-r organization is adopted deems uh... every country is at the different states and in terms of data sharing policies and mandates the u s the united states of uh... here i think has uh... very very generous and very specific mandates about what we should do in our case not only wish we are sharing all the data and all the metadata that we comply with uh... with several standards the iso one nine nine one nine one one five and family of eyes of standards like the eyes of the international standards organization the biological data profile and the ecological metadata language those are all xml expressed so uh... those are constraints that we have but they are all about sharing about getting the most to the public and and other scientists these various for scientists and there are yes there are enormous constraints in other countries it is uh... is uh... a whole ecosystem of of constraints out there one of the things that the system also does is it it validates against uh... the current email standards uh... and it validates the data just to make sure that it's it's that so it's you know if you have a particular service or a set of standards uh... it can be built into the the system added to the system is a feature to validate it validated against those standards all right we're at time i want to thank everyone for coming and do remember to fill out the survey so that the the conference organizers can help plan next year's event thank you very much thank you