 So, welcome to our talk on Better User and Editor Experience Exploring Open Data on GovCMS. Hi, I'm Matt Peroni, the web manager from the Independent Parliamentary Expenses Authority. IPI is a small agency where we're only 64 people and we primarily, our primarily function is to audit and report on parliamentarians and staff's expenses. And I'm Julia and I'm an Account Manager and Project Manager at Morft. Morft is a digital agency located in Sydney and we specialize in building sites on the GovCMS platform, setting the scene. So, providing some context and giving a bit of background and some general notes before we dive into our presentation. So, an alternative to this talk could have been how to cram a square peak into a round hole. The site has not been launched yet. It's due to be launched on the 9th of November and Matt and I have been focused over the last few months in delivering this site. Luckily, we're still friends and we can be in the same room together. It has been a challenge. The project has pushed boundaries on the platform and it could not have been delivered without some help. So, I think we want to start off by having a big shout out to GovCMS. Salsa and Amaze for their incredible support in helping us with this challenging site that we've been building. As I said before, our primary function is to audit and advise on parliamentarians and their staff's expenses. So, one of the major parts of the site is to display all that data and it's been quite a challenge to get it to work in its current form. Our current website, so we're currently on Drupal 7 Pass. The website was built approximately five years ago and we've included the addition of two custom modules and multiple other modules which are currently or which are not available on D9SAS. The website, well, the critical area of our site is around the reporting of the expenses. The current process that we undertake to update those data is a very time-consuming process and one of our key deliverables for this site was to enable our users or our internal users the ability to update all the data as easily as possible. So, whenever we started a project and I think this would be normal for most digital agencies, we go through a series of discovery sessions and in this case we had several stakeholders that we had to consult with and we recorded lots of feedback during those sessions and I've got to say that probably in this project we did much more discovery and much more documentation than we normally would have done. We recognised that the project had a significant issue to resolve and this was primarily around the uploading and delivery reports that contained complex data sets. As a result, our solution architect Murray Woodman spent a lot of time planning and mapping out the content model and the IA and you can see here sort of the mind map that we came up with and we'll talk a bit more about the different content types that we delivered in this project, but effectively we came from a D7 site which had five content types and we ended up with 18 content types. And this implementation was really key to our success as we were then able, as you'll see later in our presentation, to use solar and indexing to only query the data we needed for key reporting pages. We were also conscious that there were features that would be hard to reproduce on the GovCMS SAS platform. I don't know how many of you in the room are familiar with GovCMS, but the SAS platform is a locked distribution and we were coming from a D7 site which had some custom modules and they were not modules that we could introduce into the SAS platform and there were also other modules which would have been so useful in this project like Views Data Export, which we could not use either. We planned in a number of ways and one of the key things that we really wanted to demonstrate very early on was the fact that we could do some data imports into the site that could be actually controlled by content editors. So we did a proof of concept very early on to basically demonstrate that we could deliver that key deliverable. So the key challenges and project risks. We were aware we had many challenges to solve in this project and we were not 100% confident that everything would work and there was a level of risk in the project. We aimed for an MVP and knew there would be some unresolved features and functionality that would not get delivered in the initial launch for November. So from the discovery and defined sessions, we could clearly articulate the core KPIs for project success. So feedback from our initial consultations with our users really highlighted the fact that we needed to improve the information architecture, the search functionality and the findability of our content, which was a lot of the time, it was very varied. So one of the big improvements was the need for improved search. So our strategy was to introduce more content types to support the reporting data sets. We created essentially what I now look back on and see is for very distinct different types of content types. So the first group of content types were centered around the data imports. There were 10 of these content types. These content types were specifically reserved for the importing functions, the solar indexing, reporting pages which are rendered using views displays and then other content types that were also lazily created which enhance the SEO and future extendability of the site. Probably the most important content type in this set was the introduction of the person content type which became a key focus for the new website. We then had two presentation content types and these were for collections and sections. So collections were pages where we were displaying the views data, the reports essentially with the facets and the filters that we introduced. And then the sections I guess you could see as primarily like a landing page and they were generally sort of edge-to-edge displays. Within our third content type was a classification content type and I think this is one of the things I find most hard to explain but effectively if you imagine that you have a series of articles like blogs and they're classified in a listing page which is its own content type called blog. Does that make sense to people? Yeah? Also, scored that one. So the beauty of doing it this way is that you as a content editor can create as many article types or publication types that you like. So you have a classification content type and then it's the other one. So traditionally in Drupal you would use a taxonomy to do that. The way that Morph does it is that we actually create content types for that and the benefit of that is that you get your listing from your view display mode but you can actually edit the top area yourself without going back to a developer whereas when you use taxonomies and use like a view display straight off there's no way that you can actually manipulate the top part like the banner and all of that kind of stuff. And it's one of the things that like you know as I've been working with Morph I've kind of really grown to kind of appreciate this innovation. So the final group of content types is for general content like you know about us use items, fact sheets are kind of the kind of singular unit. The introduction of the content types has allowed Ipyo's content to be more discoverable or flexible and has highlighted many of the shortcomings of the current website. The key KPI for this project was to make all our content more searchable and findable. So the architectural design and content modelling at the start of the project served us really well having well defined content types that served a specific purpose delivered many positive outcomes for the project. The initial define and design work fed into every part of the build and was intrinsic to delivering both the requirements and the desired improvements for the website. So our next KPI. So the main goal was for our internal auditors to be able to manage their own content but also the upload of our data sets. The current process for our content editors is to create a feature branch upload all the data, the CSVs, PDFs etc. all into that feature branch and then with the assistance of a third party to have to then export the database and then re-import it into the production site. So that's been one of our issues that we've been trying to resolve because it's very timely and very time-consuming and very costly. So Morse's approach to this challenge was to use the media entity and the migrate module which are both available within Guz, CMS, SAS. So we created media entities for each report and a content editor can then upload each of the reports or each of the new CSVs into these media entities and all up there were five data sets they need to upload for each reporting period. Once the CSV is uploaded the content editor then imports the data using the migrate module. Before going live the data needs to be verified and that's quite easy for the smaller data sets where you can we've built out these content lists in the admin area where they can actually visually check and the smaller data sets might be 8, 15 rows but we had a particular challenge around the detailed report which could have 25,000 plus rows of data and that's what really did our head in on this project. This, a capacity to verify that data is actually currently unresolved for us and as I mentioned before like if you're in the wild building the strickel site outside of Guz, CMS you would turn to probably something like useDataExport as a solution but we don't have that at the moment but we are working on it. So the current process is for the management of the site or for the upload is for all the PDFs to be uploaded via CFTP the import of the CSVs via feeds import the database dump via Lagoon. This process on average could take up to seven hours of time and that's with the assistance of a third party. So this process or this project we've morphed we've tried to reduce that down by 50% to allow us to carry on doing other tasks. So the solution now empowers the IPA team members to manage the reporting data inputs themselves without the need of assistance of a third party and the new site takes two to three hours to upload and publish the data sets for each quarterly reporting period. So part of this requirement was to allow our site visitors the ability to view our data and the ability to search filter each data set because typically where our data gets reviewed by journalists etc looking to see what our parliamentarians are doing so the ability to have the data to be able to slice and dice has been really good. So the current site was struggling to deliver all the required reports for each period sometimes the site was stalling and pages were taking several minutes to load and I don't think that that is a mark of how the site was developed I just think when the site was developed five years ago there was no concept of how much data needed to be actually held in the system for the reports to be delivered. So our solution used a mixture of solar filters and facets as well as the introduction of new content types. Moving from Drupal search to solar provided more flexible indexing of content. When you visit the current D7 website report section the CMS is querying all of the data for each reporting period. We changed this with a new build so that solar is only querying portions of the data set required for the particular report a user is viewing. So if you go back to the start of our presentation where we showed you all of those content types that was key to the solution that we came up with. As a result we are using less server resources and are able to deliver the required content to the site visitor more quickly and efficiently. So we also worked on improving the user experience so we introduced filters and facets that allowed better interrogation of the content. So for example on the old site there was no way to compare a parliamentarian's expenditure across different reporting periods on the same page. So with the new design that we were able to implement on the site Morft was able to deliver this feature so you can see here that you can actually search by you can see here that the solar indexing and how that's delivering these different facets which we can now use on the page. You can also select a name and a state and see all parliamentarians that meet this criteria. So this was something that we only really sort of uncovered as we went through the early stages and the mid stages of the project was and it took a while for the penny to drop that a parliamentarian could have two reports for the one period and it took some time to uncover this requirement and so what it is is an MP if they're in parliament or they're working in their electorate have two distinct reports and they have to be shown at the same time. Likewise if you think about it you could start a reporting period with a prime minister and as we've experienced in previous years that prime minister can become a former prime minister and require a different report. So I think that was one of the I don't know how much we talked at the beginning but we still talk another month or two to uncover that issue. So as Matt mentioned the previous site relied on a PDF for each parliamentarian within each reporting period. This entailed the uploading of over 200 PDFs every reporting period. More often introduced the concept of person to replace the requirement for the PDF and when visiting the person page the expenditure reports are shown in summary format for each category. So what we've done is we've based that on the accordion component so when you land on these pages you'll see there that everything's scrunched down into an accordion with a summary and the summary shows the total expenditure for each one of those categories and then someone can go in and expand each one of those accordions to actually see the reporting detail. This page also at the top includes a dropdown menu so you can jump from one reporting period to another. So one of the other really nifty things that I feel like Mort has done on this project and one of the things that really challenged me when I was first looking at this site was this requirement to be on the what we call the aggregate report and to then be able to link through from that to a detailed report. So for this example you can see we've got selected the January to March 2019 period and you'll see there that we've got Bill Shorten and we've got this amount for travel allowance of $13,290 and when you click on that it'll take you to the transactional data with pre-selected the facets to deliver that particular information. We required a solution that was going to be stable delivered better, more flexible provided more flexible reporting including the slicing and dicing of our information so that our users could explore and extract better information from our site. Mort was able to deliver a structure that was better suited for the amount of data that the site needed to serve. We believe the solution we have come up with is highly flexible and will serve IPA well now and into the future. So another requirement for us was to ensure that all our content the website etc. met all WCAG requirements and inclusivity and accessibility but also to be available and available on all devices as well. So IPA and Moff worked together to introduce specific features that supported the display of general pages but also specifically the reporting pages which is the focus of our talk today and the tables were a particular challenge because we were dealing with many columns of data. So some of the things that we looked at and implemented was like the font selection so we introduced monospace fonts for numeric characters and this made the data within the table structure more legible. We introduced filters and facets and these... I'm just a bit lost here but sorry. So yeah and these could be collapsed so when you visit these pages they load expanded so that you can see them but then you can crunch them down so that you can see more data on the browser window. Part of our design included the display of reports to be edge-to-edge. So this was a specific challenge as it made sense from a content perspective but in terms of visual appeal it did on initial viewing make the page look like it was a bit broken. We worked around this by making the search background edge-to-edge which you can see the faded green area in our slide there and this visual element remediated some of the concerns that Ipia had. We did comprehensive testing and tweaking to make best efforts to minimise side-to-side scrolling on laptop and desktop computers. We also introduced this stackable view for the tables on mobile devices and this again removed the need to scroll side-to-side to inspect the data and it provides an easier way to view the reports on smaller devices. So one of the biggest improvements that we've had has been the ability to search by name over any reporting period. This has allowed the user to be able to compare any report on any specific parliamentarian over any selected period of time. Consideration of the report displays took some time on this project. It was important that we made best efforts to improve legibility of data for the broadest range of users. We achieved this by considering the font we used and the reporting tables utilising the maximum amount of space on the browsers but also configuring that space to minimise scrolling for different devices. Due to the size of our agency of 64 people there's no internal resources that we have to be able to maintain all the Drupal updates on pass. So we relied heavily on third parties to maintain the current websites which became a very expensive exercise for IPIA. The key part of the project was to deliver a functional website on GovCMS SAS eliminating the need for any third party. So GovCMS SAS was a good solution for IPIA as they provide all the maintenance services for the CMS. As mentioned previously GovCMS SAS has provided the solution we need and was the core requirement for our project that it removed our dependence on any third party suppliers ensured the platform stayed up to date provided significant cost savings for our department which has been very much a big part of what we wanted to deliver. So GovCMS SAS made a lot of business sense however it did present many challenges for the redevelopment of the site including the limitation on modules that can be used. And we've mentioned the views data export module and really it provided the challenge of how we were going to import and export data and empower the content editors to achieve that. So we obviously had significant challenges in this project and we just thought we would finish up our talk with just a few slides on a couple of the largest ones and then some pointers on how we unblocked those challenges. So we were dealing with complex data sets. Imports on our local environments were working but on the GovCMS SAS platform the imports were failing. That made it really difficult to troubleshoot especially since they were failing silently so we were not getting any error messages no feedback we were going through logs nothing was coming up. We were able with it and it was really just the biggest one with the 25,000 plus rows so all the other smaller CSV files were just like on song just importing perfectly. So we tried splitting the transactional data into three separate CSVs and uploading them and that worked but that was not acceptable to IPIA because they felt like there was too much room for human error in that. So and because we're getting no feedback from the CMS or from the logs we were really quite unsure on what was actually happening but we were 90% sure the issue was around the limitation of processing and server resources so when you're on GovCMS SAS you're on a shared resource platform. So at the moment GovCMS and AMAZY are currently working together to come up with a solution for us which we thought it would happen but it hasn't happened yet. So that was with the importing so then the other challenge we had was the exporting of data and it's absolutely vital that the IPIA team have the capacity to verify their data is obviously public facing they're mandated to deliver this content to the public and it has to be 100% accurate. So on the current site they use data export module to do this and then once they've validated it they then put it onto data.gov.au so it's available there as a resource. This issue is still unresolved and we're currently working with GovCMS to resolve this issue. So Matt and I have been aware for some time pretty much from the beginning of the project that this was going to be an issue so we thought we'd just finish off because it's been really useful for us to work out how we actually communicate to GovCMS and how we work with them to unblock issues like this. So really number one is to talk to GovCMS and find out what the possibilities are and the way we did this was I think Matt you were talking to somebody at GovCMS and you got wind of the fact that they were already looking at views data export so I would recommend anyone working on the platform to join the GovCMS Slack channel because that's where I got to straight away and started to ask some questions about what the issues were around the module that we needed to use. I found out very quickly that there were a few bugs that needed to be resolved and that was something that I could do something about so I got my team members onto some bug reports on Drupal.org and they started to work on it to unblock those issues. So that's really the kind of the core way that we're doing it and we're sort of slowly progressing. So from a client's perspective I would recommend getting in contact with GovCMS submitting a business case outlining all your requirements as to why but also then to be patient sit and wait for a long conversation because it has taken us quite a while to get to where we are to hopefully get it resolved before the 9th of November. And that's our presentation. Do you have any questions? Yes Helen. 37 questions. I want to hear every single one of them. So with the views export does that allow external users to export your data from a view? So we would not recommend that you can use it in that way but it would probably overload the resources on the services potentially so what we're trying to do is do it internally so as an admin you can go in and export the data so if you had a very fixed requirement where they need to be able to input the data to the CMS but then they needed to export it and then verify it by a third party system to make sure it matched their other system so as an admin they can do that. So comparing the two versions to make sure that the imported data was exactly the same as what it was originally. What do you want to know from the book? Yes. And the other question was with your content type so the person content type so that's a content type or a taxonomy content type. Content type. I might take that offline and have a bit of chat with you about how that works. I'm going to be hovering around the wall of stand and we've got cupcakes so if anyone wants to come and talk to me afterwards we'll be there. Thank you Helen. I just wanted to know how much code do you have to write in that theme to get it running? What do you think that is to do it all through views and config? Most of it is absolutely through views and config and really that query which Murray and the team developed between the aggregate and the detailed data like I know views pretty well and I didn't know how they were going to work that one out so when I saw how they did it I was quite amazed. I was sitting on the edge of my seat going do they really know how to do it? The only other question I was going to ask which is actually really relevant what I'm doing was did it turn out to be the CSV format of the import being the issue like if you'd been able to do a postman API request into a data source would you have had the same limitations as the CSV was causing? I think the biggest issue with the CSV was the size of it. The sheer size of the scale because we have five of them four of them not a problem. But the 25,000 rows. And they went up to 40,000. So we're really pushing it. It was really the multiple joins into the content type that Drupal's doing when it goes to import. I don't even know if it's exactly that because last week because I was running all the test migrations we had a beautiful day where it was just doing it and I was just going oh this is amazing we don't have a problem anymore. Came to work the next day and it's just failing every thousand records it was failing. I was just sitting there all day doing other work. So the day it worked I had a sleepless night because I knew I had to give this talk and we hadn't got the data in there. I was up at 4am and I went from 4am to about 9pm and I got I think almost three years worth of data in during that period. It would be interesting if you could control the import to the point that we get to 500 records we pause, we come back you know half an hour later we do the next 500, they're all automated. So I think we'll know by, so stay tuned if you're interested drop your business card off to me because I reckon by next week we'll know whether the additional resources is going to unblock us on that one. Yeah. Thank you, that was great. So quite interesting for you said from certain common types to actually increase common types of numbers and you also mentioned about don't really use text on me which is quite interesting to hear. So my question is I think from your business requirement you think you want to actually group the data by content type and write them a text on me. Yes. I'm certainly seeing a text on me have some problem as well. My question is do you recommend use content type and write them a text on me? Oh that's just for your business requirement. So I think I'm just going to bring this up because I think it's a really interesting one. So these are the two classifying content types here. And so before I worked on more I would make them taxonomies and that's how I would tag my database. You know we didn't keep the database put pretty small like that. But I can see the beauty of this because it means that Matt can go in and he can create a blog. Use release media or press release I don't know what else you do there. Is that a taxonomy the type of the article? So it's a content type and what we've gotten there embedded is a view of listings. So a view block. But an article type is the content type. What do you use to define a blog so we create an article we'll call it a blog and then when you create a blog article you tag blog in there and it's doing an entity reference to there. And then we've got a view block display that just lists it as you would with a standard sort of view. Does that make sense? Yeah sure I understand right now. So I just want to understand the use case. The use case is that I think from use content type more flexible for editors they can type you on a view straight forward rather than target a taxonomy. So it makes sense. And probably publication type is probably the better example because if you think you might want a page called annual reports to there but then next week you might go oh we've got a whole lot of fact sheet and you don't want to go back to the developer to create all of that this way as a content auditor. Yeah I think the question that I come from because I'm from development I'll always ask her best know why you want to contentize because people come to us later we want to contentize. And the first question I always ask can you do different way that's the question I come from. So come to us still and talk to me. I need the brain to go away. One more question with paragraphs of work to suppose the taxonomy or content. So that would be if you were manually building out a page this dynamically built that. Oh no no so you've got a reference to an entity reference to the article type but would it have been any simpler to just entity reference a paragraph type with the same theme. I need to know a paragraph story to be able to hear this. I'm just curious about the footprint. Will because you still worked on the actual model being fully considered. Because I know a part is working on it. There they are. We're all pushing together because I think that there's a whole world out there that would love to see that module and the content to explore. Absolutely. Thanks so much. We'll just keep on talking and being friends and we'll get there in the end. Thank you.