 Hello, hello, you can all hear me. Yep. Good Hi everybody, thanks for joining us today I wasn't going to do this, but I'm going to start off with a completely unrelated anecdote Just to give you a little insight into the past 24 hours of my personal life Yesterday I had to leave the conference early after I received three panicked messages from my wife They read There is crap everywhere. There is crap everywhere. OMG. There is crap everywhere And I have an 11 month old baby. So I was like, yeah, obviously this happens all the time But there weren't any emojis in the message. So I kind of figured there was something afoot and yeah, a sewer pipe burst under the house Yeah, so I'm pretty sure my house is condemned We all had to flee the building. It was kind of terrible. So Yeah, as I said completely unrelated Wipe that from your mind If you do see me blanking out, it's because I'm thinking about something else like where I should live now Anyway, it's fine. It's totally fine So completely unrelated. Let's talk about govcmscm It's a module we built with the ultimate end goal of connecting central data sort and seek and seek and providing a Visualization layer in govcmscm seven sites So few logos on the screen Let's quickly run through all of the people companies technologies involved in this one We'll start with the easy one It's me I'm Stuart Rollins a senior technical architect at Acre and the technical architect of the state of the environment report Which is the parent project of govcmscm? My role has been architecting the technical solution and providing oversight to the development team and I like beer I'm looking for a new home. That's pretty much it for me The incomparable Megan Watson from the Department of Environment It's our amazing client Megan Watson is the product owner from the department She's the one constantly scheming up ways to break our brain with awesome complex requirements, and we wouldn't have it any other way Megan also brews beer, which is I think the best hobby a person can have Jeremy Graham, sorry the masterful Jeremy Graham the gang lead developer from our partner doc house agency He's been an absolute champion at churning out quality code throughout the entire project At a rapid rate and doesn't look to be slowing down anytime soon Jess is going to be giving a demo a little bit later to show you all how quickly you can get up and running with Datasets playing in secan and creating awesome graphs and visualizations Jeremy also likes beer, so you're starting to see a little bit of a pattern here And as a side note if you use code your xpmc for your media streaming You should probably buy him one later because he's a contributor to that project GovCMS it's an award-winning program and custom triple distribution tailored for the Australian government It's owned by a department of finance and is beating expectations on all fronts Currently over a hundred sites live over 30 in development. I think and has a ton of community support and momentum It's been really great working besides the Department of Finance on the GovCMS project and a huge Thank-you goes out to everyone who's been involved in contributing to GovCMS We certainly wouldn't be here without all those contributions Of course a man man secan Mexican rap god on his way to global fame after racking up over a billion YouTube hits But no he hasn't contributed anything at all secan on the other hand is an open source data portal it's in use by Many governments and large organizations throughout the world so data. Gov. That are you Victorian Queens and governments They're all exposing data in a transparent and accessible way using secan It's not only a great place to centrally store and share upstream data It's got a data store extension built in which returns Compatible data sets in a queryable format via API Which luckily is exactly what we need Means you can upload CSV files excel documents and it will translate that into nice machine readable Content that you can access over an API Cool. So I'll give you a brief client brief around some of the things that Megan and the team wanted to tackle It comes from a Static report of a paper report. There are hundreds of images There are thousands of Different combinations and permutations of graphs and visualizations that we needed to cater for We need to take those static representations and create them in a digital format Yeah, and they needed to be interactive so some of those many many variations here First of all, we've got a scatter plot with the trend line. We've got labels on the x and y axis We've got an inset label We've got some weird kind of scatter plot thing with a Shaded region on the right and set legends ticks along the top x axis Yeah, lots of stuff going on there Something else some kind of yeah two horizontal lines across this one Shaded areas again. We've got legends and lines and inset labels and It just kind of goes on and on and on and on there's a lot of this one's a good one actually sorry Yeah, so variants between values That one's crazy Yeah, you get the idea. There's a lot of complexity within these things Okay, not only that we've also got to support govcms s Govcms is obviously a highly secure while performing platform We can't be adding contra modules left right and center need to make sure this thing is Well contained flexible and fairly lightweight So we had to find technical solutions that fit within The bounds of govcms. We didn't want to add technical debt or add new dependencies to the distribution Speaking of flexible it needed to be pluggable and extensible So people should be able to take their data and create whatever custom visualizations and charts their hearts desired There was also another projects running alongside this one the Victorian budget site They had their own requirements for graphing and visualization. They wanted to use the govcms module so Not all of their required functionality could be achieved out of the box with the module at the time of development So we managed to align the development processes and have contributions from both DPC and DOE flow into the module in parallel And it was a really nice moment to see federal and state governments align actively contributing to the project at the same time And kind of illustrated the power of open source and govcms So now I'd like to welcome up Megan to the stage She'll give you some further background on the report itself and some of the requirements Thanks, Stu So I'm just going to give you a bit more of a background of what our requirements were We produce as Stu said a paper report. It's traditionally been a paper report This is the fifth version of the report it comes out every five years It's called the Australian State of the Environment Report if you haven't checked it out already check it out in about February next year So every five years we're mandated by the Environmental Protection Biodiversity and Conservation Act to produce this report It's written by a team of or a panel or a committee sometimes as independent scientific experts, so Hence the quite complex requirements for graphs We About it takes about 18 months to write the report and it's full of a lot of great information It covers nine different themes I won't run into what those are but things like land and biodiversity in the marine environment and that type of stuff And so all of those themes have their own data requirements and their own information that we need to put up on the website so It's a mix of expert opinion It's a mix of data sources in the interim period between 2011 and 2016 a lot more data sets have become available We we also have the single assessment summaries which these guys have done an amazing job of Visualizing that data and but that's a whole another A whole another talk In 2011 the report was 932 pages long, so we've got to take all that content mix it up on the site create different content types Make it basically not a static HTML version which it was in 2011 But we also have to produce this report to table in Parliament So we have this parallel production process of we're trying to create content for the website But we're also having to create a printed version So that's in itself a massive challenge So that's what it looked like in 2011. It's it's great. It's you know, it's amazing content It's it's a really good Snapshot of what the what the environment? The condition of the environment at a point in time is but it's book You know this you can't compare from from 1996 when they first came out to To 2016 when that report comes out unless you have all of those books open at the same time on the same page So obviously we wanted to do something a lot better. This is what it looks like on the web currently It's kind of got a pretty boring table of contents You know really static content everywhere Very hierarchical as well So part of the thing that we want to do on the the site is create different content types that can be aggregated or Surfaced in in a different way So if you want to look at all of the state and trends across all of the thematic areas that we report on you can do that on the site when it's launched Currently, you can't you know, you sort of read it in a in a very hierarchical manner But also you need to be able to read it from introduction to Closing summary of that report as well. So you can see there's kind of there's a lot of challenges So how can we easily compare findings and content and information and data sources across Reports and across time and so one of the criticisms that we had was it's really difficult to see what the trends in the environment are Changes over time basically, which is what we're trying to satisfy with this You know, we can't interact with the graphs obviously But we also have to show that snapshot on time We've got a really diverse readership. We have people in in high-level policy areas. We have Obviously researchers. We have people in tertiary and secondary sectors who who want this resource For teaching aids. We have private sector people. We have, you know, you guys general public If you're not interested in the environment, you should be You can come to our Report and read what the key findings are and read what all of the content is read what the findings are that our Independent experts have written about We've got lobby groups that want to get Snapshots of information so that they can use that we've got, you know, journalists that want to say so what is the condition of Australian environment? You know, what's happening with Colson gas for example so It's it's really complex We also when building the site we need to ensure that the 2021 content can fit into this model as well So then we start to get that picture of we'll have 15 years worth of environmental reporting that we can then show in a really nice interactive comparable manner But also at the moment, it's just me in my section doing working with these guys on this stuff So we don't have we have one Drupal expert in our Whole department and we've got about 3,000 people in that apartment. We have multiple websites So we needed to have a solution that was not going to rely on an in-house Drupal developer But somebody something that we could you know I could kind of look at analyze make sure that all of the the content is actually getting served up as it as it should Be but we can't be running security patches. We can't do updates when you know to droop like that kind of thing So what was the solution for us? Well, it's it was govcms It was no brainer for us because it just provides this great platform where we can use software as a service I can liaise with the govcms people. I don't have to know anything about you know updating any of the code base Most of the stuff that it could do was good for us, but we realized it was a bit of a gap with the data visualization We also want to use the whole Notion around open data open source applications that the government's developing so Where we like to think of ourselves at the moment as opposed to child We're going to try and get out there and talk about the benefits of using govcms But also using data.gov.au to host all our data So if you don't know what data.gov.au is jump on have a look The idea is that you know all government datasets will be available there online to download So again, that was one of the criticisms from the 2011 report is that we you know Nobody knew where the data was coming from you couldn't download the data We want to empower people to get environmental data and to do what they want with that combine it with other data sources come up with Findings that we not might not necessarily have had We're also using national map to we have a lot of spatial data as well So not only graphs, but we have a lot of coverage as you would imagine across Australia That's of a spatial nature. So we're going to use national map again All of our data sets will be up on data.gov.au So govcms was a perfect fit for us. It allows us to use SAS We can also create lots of different content types that we need to serve our data up takes on a means obviously to identify we we Clearly report by year through 2011 2016. We've got all the 2011 data up there We report by different themes. So we need to be able to tag our content with that We also have the sync order reporting framework, which is a bit if you think of it in terms of this is sort of how people How our experts have looked at things like they look at the condition of something they look at the outlook for that particular environmental issue Ricks risks to it. What the pressures are on it and that type of stuff So we have to combine all these things to actually create our content in the way That's going to be meaningful for our really diverse range of users Obviously aggregate content and that type of thing and filter stuff But also we want to develop the interactive graphics. So as I said one of the criticisms was we didn't have In the past we haven't had information that people could download people want the data They want to know where it's come from. They want to know that that data is actually Authoritative data and it can be trusted to give that snapshot of the state of Australia's environment. So Data this is what we're really here to talk about. We're here to talk about data in the secan module. So We as an author as an author on the govcms Drupal platform I need to be able to access the data plate data sets so that I can display them. I need to be able to Choose what data sets I want to show Suppress some of the others so we might have a really Massive data set the CSV file, but actually we don't want to show all of those data sets in there So as govcms author, I can then choose which data sets. I want to put data series within that data set that I want to display Stu showed you a couple of the graphs that have these Horizontal vertical lines on it. So they're not being drawn in our secan module They're not being drawn by the data set itself. So there's something an author can actually put on and say show me a trend line at this particular Access I Won't go into too much more detail because I think jazz is gonna You know talk more a lot more about the the power of the secan module But there's a lot of stuff that as an author I want to be able to do with the the graphs and also then as a user of the site as a viewer of the data You want to be able to hover over, you know, it's it's pretty common kind of graphing Interaction available on on the web now So you want to be able to hover over those data sets and see the values download the data download a Snapshot of that data so a PNG file if you want to or an SVG file So our print production layout people have been using the graphing module to create their print version of the publication So they download a CSV file run it through Adobe products layout out and in design And also people that have assistive technique not technologies or don't want to see the graph They just want to see the the table of data. They can see that as well They can also just click on a button say download the data takes you to the data.gov.au resource page You can see all of the metadata for that data set So there's no question now of where that data has come from and you can download that CSV file if you like so That was in that was the kind of boring looking old grass This is now they still look boring, but Jess is going to show you how awesome and shiny and new they are so as I was saying these are the tabs along the bottom where you can Download various bits and pieces So jazz show some magic. Thank you Yeah, so everyone loves grass, so I get the fun job of showing you them all Just need to switch over All right So Yes, I'm going to start off with some fun stuff actually showing you the graphs and then I'll kind of work I guess a bit backwards of how you can actually make those and use them in your own site I suppose a lot of what we've talked about so far is very much govcms stuff, but This will all work very happily on just any Drupal site. It's it's not govcms specific So and that modules up on Drupal.org as well so This particular data set actually came from data.gov.au It's a pretty complex one It has got a giant amount of data in there sort of as we scroll across we can sort of see the records and Because it's interactive sort of we can like turn on particular sets And sort of it it's sort of really nicely dynamically changes everything It's using a library called C3.js, which I'll talk about a little bit more, but I guess that's One implementation. I guess the way we've used this is That's you can use any kind of graphing library you want to really to really make that work Pretty much all this stuff happens client-side and Sort of we've also been able to do some cool stuff like download as a PNG which will save that It basically converts it to an SVG Then saves it as a PNG and you've got something you could put in a book or as an SVG And you've got a nice vector of it as well So all that stuff kind of happens client-side with a couple of JavaScript libraries, which I wrote so As you can see it's pretty powerful and you can sort of have a bit of I don't know interesting sort of fun graphs like space launches versus scientific Scientology, so sociology doctorates Correlations ice cream consumption versus shark tax and then I use this So I'll just quickly talk about How a lot of this is made So like I said, it's using D3. Sorry C3, but C3 is a library that's built on top of D3 and D3 is kind of like an Extremely powerful thing when you see sort of some of the stuff it can do This is basically a show real of just a bit of a demo sets showing what D3 is kind of capable of Unfortunately our graphs aren't quite there, but it's certainly capable of it And it's got a plug-in system where you can really quite Easily plug into it and sort of make another module even make these sort of things in your theme and Yeah, which is really good for something like gov CMS where you don't have the ability to create a module so C3 Basically makes doing all this stuff easier is it's got a library specifically for charts D3 I guess is all sorts of visualizations that C3 sort of really pushes on the the chart side of things and it gives you a really nice clean easy to use API So making all these sort of visualizations that I've got good documentation if you just follow their steps that it kind of Comes together quite nicely So I'll go into a little bit of the technical side of sort of how The module works there's many parts to it and I guess you could use just little parts here and there depending on on how you Wanted to use it The first thing is well, I guess working backwards if we're looking at a graph like this How it actually works is it uses a Javascripts jQuery plug-in which I wrote which is I called it table charts It's a unexciting name, but what it does is it turns a table into that chart So what's actually happened here is we've just rendered this table to the DOM and we just put a whole pile of attributes in the table Tag and that defines sort of like the access labels sort of colors Any kind of settings which we want to define for it So if you wanted to you could use this on a non-drupal site You just put a table in your site and add some attributes It's kind of well documented in the plug-in itself, but not really anywhere else But it's it's works really well, and I suppose this satisfied a really great need for To a as well because it's very accessible if you get to the site and you don't have Javascript You've still got all the data there. So looking at sort of one of our Simple ones It's just yeah a table of data We've got a column heading and Heading up the top as well, and it uses both those to convert it together Any questions on that little bit yes I'm not sure what level it is, but it's essentially It could be used by screen readers. You don't have if you don't have Javascript It still works perfectly so it kind of ticks all the boxes I'm sure it would fit into that category So my my little plug-in which I've made Reads that table and then it parses all those settings into a nice sort of ordered format and then sort of handballs it off to another plug-in So that second plug-in is one which I've made specifically for C3 So that's sort of one of the points which you could jump in and create your own visualizations You could just add another Javascript file It's just a class which gets handed a nice pile of settings and it can then deal with those settings And obviously it also gets handed the parsed data which is Reading that table into a sort of a nice array of rows and sort of passing these through as headings as well So I suppose that's the one place you can sort of start to extend it And I'll go into a bit further But there's also C tools plugins where you can sort of do it a more deeper level of integration So that's all well and good. I suppose up to this point. We haven't actually needed Drupal We we can actually just do this just with grabbing a bit of code, but We look to satisfy. I guess it's it's a complicated thing to do if you're a content editor And we don't want that and obviously we wanted to get all the data from data.gov.au so that's where secan has stepped in and like Stu and sort of Megan have discussed a bit secans essentially a great place and way to store data So I've set up my own little secan server whoops a bit close to the mic Just as Example of I guess showing you what a author would be doing from the government to upload some data Once once it sort of installed it sort of looks a lot like this and Essentially adding some data. I've got Just this one here So I'm going to upload this CSV So it's this has been the whole module was mainly built using this start this one CSV to be honest, but Obviously a lot of testing has gone on to us. It's one of my favorite ones to test with So yeah, just I don't think any of the numbers are accurate But we'll just call our packy to stupele south make it public next Cool and then the next step is just uploading a CSV so finish and So at the moment secan sort of yeah, then then kind of parses that and turns it into Something that's the API I can read as a nice table so at this point how we kind of start getting that Into Drupal is we use the media module. So I don't know if anyone's familiar with media Internet But media Internet you basically give it a URL and it'll turn that into a file record. So we Converting sort of this remote Data set into a file which we can use in Drupal. So I'll just step through how an administrator would Go about adding this particular data and turning into a graph. So let's just copy the URL from the top here With step over to the website and hop into files And add a file And so we've got the web tab when we're using media Internet we've got a new resource type which is secan data set and it basically knows that it should take ownership of Specific URLs which it gets past. I'm just passing the URL. It's actually the one of the CSV on the end and At this point The website has queried secan. It's actually grabbed a lot of the metadata if I had given that a name I should have given it a name, but I didn't so it's called unnamed resource We see that has actually come straight across and it's it's picked up that metadata At this point we choose what sort of graph we want to have So this is where the second I guess plug-in part comes So these are all just see tools plugins. You could happily write your own one Which would end up in that list. You just need to I guess follow the same format and there's some great examples in the module So we'll just go a bar chart And at this point it sort of starts giving you all these settings which is specific for that type of graph that you want So I won't go into this in too much detail because there's a lot of things here Kind of every time we kind of get a new weird graph come request come along It's like we okay. How can we work it into here in obviously a generic manner? So it'll work for everyone, but in a nutshell these are the the bars we want to show or the The columns we're going to have on the graph. So we'll choose those ones We'll say what's going to be describing The label which is going to be that sort of side column Actually, no, that's going to be the data sets at the bottom But it does get confusing when you're dealing with so much data coming in and out because you can display that also in many Different ways, but I'm going to run with it. We'll just see how it looks at the end Just with this particular one We actually had a lot of repeats here, and it was actually repeated because there was multiple years So one thing we can do is we can split the whole graph on Yeah, and that should you actually give us two graphs. So We'll keep it simple and just go all the way the bottom and just save Now we should have our graph So it's a pretty simple process. That was just a CSV got uploaded We got to choose a whole pile of settings for it And now we've got it available as sort of an interactive graph, which is now a file So I guess in Drupal world we can use this as a file field which is probably my preference and Sort of embed that wherever we want, but you can also use it in a Sort of a bit more of a generic way. So we've got a page, which is just a basic bit of content. I found lauram hipster as my text generator and I'm just gonna use the media browser And right here. I could select web and go through that whole process upload a new Data set, but I'm gonna just use one from the library When I created earlier, let's go ice cream versus shark attacks And we've got all those settings available. So we could we could tweet those settings if we wanted Let's put it in the middle of the documents Save and You've you've just used it just like an image. So extremely flexible And I think sort of once you start throwing in sort of the possibilities and analysts with these visualizations Which which can come through it's sort of far Yeah, it comes together really well So I guess that's about it for the demonstration is any any questions on any of that so how that works Would you like to contribute? I guess one thing I'm throwing is we'd like to probably get this into Drupal 8 at some point So I think probably one of the biggest things that's holding us up is sort of the media module getting in there. So I think I'll definitely put some more work in once media gets there, but I don't know if I'm committed enough Get media into it Yeah, so we've also got this up on github as well That's sort of where all the development goes. So obviously if you're spot any bugs or anything to contribute and it's a great place to go What's that sorry Yes, absolutely so out of the box you've got a bar graph line graph spline scatter chart and just a table So Uses anyone familiar with writing a C tools plug-in It's it's very much say the same thing as writing maybe a panels plug-in But it's a case of making a plug-ins folder inside your module or your theme You create one file. I think which is just correctly named It's Probably a little bit of formatting you need to define but essentially you define the renderer and you define the settings for it So you can inherit a lot of settings which are all predefined and you can add in your own custom settings as required and That render renderer then actually calls the JavaScript to pull that in so Absolutely, yeah for sure. It's just a lot more work. So if you yeah D3 has got a huge Array of stuff. It's just harder to write. So and we had all the time in the world We would have gone down that road, but um, yeah But yeah, yes Thanks So the difference between private and public is You need to use essentially an API key and that API key is linked to a user and they need to be authorized So the way I set it up just to keep it simple is I didn't use a API key So I had to make my data set public for me to be able to use it with this But yeah, sort of if you're dealing with private data, you've got an API key as that uses permission You know different data sources which require sort of you know different sort of connects between them reason I'm asking is we're actually trying to Produce some complex charts, which some some data talks about life expectancy, then there's data on Accessibility to education and so we have to bring all different sorts of data Yes Which I'm not sure would be possible Yeah, it was something that we I did kind of talk to these guys about in terms of combining different data sets into the one visualization But we ended up realizing that just for our needs we could in fact get around that need to create different bar charts and line charts together by actually Tweaking module to allow that. I didn't look at Tableau, but I think Because the government is a little bit more difficult and also Gulf CMS was the platform So we wanted to actually contribute something back in with creating module So we were sort of tight tight in but not tight in this you know what I mean, so I Think yeah into the future we'll probably have a need to create multiple Visualization Seconds really powerful with how you can use and what sort of data you give it Like it's not just limited to set its face I mean it could be but not coordinates and sort of all sorts of complex data sets So I guess about the moment what the module is doing is it's it's doing I guess what we needed it to do is pulling in sort of I Guess the basics I suppose which is still quite a lot But certainly I guess with a plug-in architecture you can interpret that data however you want like with these visualizations it just gets handed this data and It goes do whatever you want with it and you've got settings to define however you want to set that up as well We use that data, so I think on the secant side It'll handle most of what you want to throw at it and on the module sides It's expandable enough to handle that We Can't one-size-fits-all But there are certainly some things An endless list of my today list never finishes Ah Yeah, I guess we're a little a little bit limited I suppose by the project which we're on and sort of other stuff that needs to get done So I suppose what I guess probably a lot of our vision is is that the next person who Requires this and has some special requirement is going to build on top of it But I think probably the the biggest logical next step is maybe looking at something for droopa late But I think in the meantime. I'd love to see a module like a secan Visualization pack or something like that would be really cool. So obviously not something we could really use with Possibly I suppose Yeah Right One of the things I'd like to see Sorry, it's on being able to have use of findable time-series Notice some of our bigger datasets might go from, you know, 1901 up to present day So, you know, sometimes you don't want to see all that data on one We may not, you know, get over that one before we go live but Certainly something I want to sleep to the future as well as being able to flip between 2011 Retrofit Yeah, so we've already got Moment the grass any images figure-based content will still be static unfortunately because the beauty of this We know where that does come from and my natural to all of our authors and people on the team It's no doubt in my graph, you know, you're not getting a graph unless you give me the data Think we've kind of You know, sometimes you have to compromise but I want to You know start that process between now and the next before coming out going get and seeing if we can Get the 2011 datasets and put those up as well You know Yeah, yeah, yeah, so it's really it's actually really complex to get the data to get these stuff as well because the way our Allsons work is that they go and find different reports from all over Australia different academic research They take a screenshot of the graph that they want We have to go back to the data custodian and try to get it to create a comment All that content is going to be created in the comments as well So Yeah, I guess you probably could I mean is is Drupal behaviors are supposed to trigger building of the graph I Guess obviously it's reading a static table It's probably a way if you change the table trigger to rebuild then that'd probably be a non-successful I suppose it's also a way of it sort of you've always got a table behind it. So Some people is We have a government department that is strong money and resources into the development of the thing that they're just now doing away from the distinction. As a Dribble community, that's normal for you guys, right? You're doing this all the time, you're coding for nothing, and that's why you're caring, it's good for you. We've now got this happening, this is a new challenge. So I personally think that's really exciting, and that's doing good for the community. And we do that with the community, that's what causes all of this. So I find that these guys have done an absolutely awesome job. Any more questions? Yes? I've just put two in the model page, and it's only available at the San Francisco stage. It shouldn't be. Yeah, it's definitely... I don't know if it's fully up today, is it? Yeah, okay. I suppose we've got a developed branch, sort of in GitHub, which has got some of the newer features. Do you want to keep up with the latest development that you get? And I suppose we choose as well over Dribble HQ. It's probably a better place, it's probably GitHub. But yeah, I guess either. We'll look at both. Yes, any other questions? Who wants a graph? Pie charts. Actually, just on pie charts. Pie charts are built into C3, and I was going to... I was going to do on stage how easy it is to make a visualisation of whipping up a pie chart visualisation, but I thought that's throught with danger, coding on a podium, so... But it's a good challenge. It's built into the C3 library, really simple. You could literally just pick up one of the, maybe the bar chart, plug in, copy that, and tweak some things. Wouldn't be difficult. All the hard work has been done already. Yeah. Are we going for time? It's all we've got at the moment. Got to use some of them around mid-February next year. Go check it out.