 All right, they closed the door, so I'm going to go ahead and start. Hi, this session is called Creating a Single Face of Government Across Multiple Websites and Technology Stacks. My name is Brian Hirsch. I'm the Deputy Chief Digital Officer for the Commonwealth of Massachusetts. I work for a department in the Executive Office of Technology and Security Services called Digital Services. And our mission is to use the best technology and information to make people's interactions with government fast, easy, meaningful, and wicked awesome. Now that our new Maths.gov website is live, which is the foundation of our new digital platform, our next frontier is creating a single face of government. So before we dive in, first, thank you all for being here. I'm so grateful to have the opportunity to share our ideas and experiences. As the saying goes, most of the smartest people work for someone else. And so, you know, we're solving big, gnarly problems and really appreciate ideas and input that anyone has to share with us. I also just want to set expectations that we've got a lot more questions than answers. So I'm excited to share our work. This is by no means a solved problem. But those are the caveats. Let's dive in. Okay, so let's start with a survey of the problem space. Why create a single face of government? Are there too many faces? If so, is that actually a problem? Maybe you think that things are fine the way they are today, right? So here are three stories that represent the status quo. Problem number one, you need a driver's license. Where do you go? Where do you go? DMV, in Massachusetts we call it the RMV. Now imagine you've got your driver's license. You want to become a ride share driver for Lyft or for Uber. Where do you think you go to get approved as a ride share driver? No one's going to guess the RMV or the DMV. Okay, first time that's ever happened. So usually people guess that you'd go to the RMV for drivers, you know, for getting approved to be a driver. In the state of Massachusetts, you need approval from the Department of Public Utilities, which is under energy and environmental affairs. So until recently you had to leave the RMV website, which is called massrmv.com on the left, find your way to the EEA microsite on mass.gov on the right and then down to the transportation network company division page of that microsite to figure out how to become a ride share driver. Putting information about how to become a ride share driver is like taking an orange in the grocery store and putting it in the nuts and snacks aisle. You know, having RMV and DPU both interface with drivers is one example of many faces of government. I'm going to call this problem the orange's problem. Problem number two. For this one, I want to watch a recording from a product test that we did with a tax preparer interacting with the brand new mass.gov site that we launched in the fall. Which I'm assuming they did because that's where most people go. But at the same time, you know what's missing from that row? If I need taxes, I can go to taxes. But let's say I know I need DOR. There's one that says, department, what all the departments are. Because sometimes you don't know where you want to go and this is beautiful. And sometimes you know exactly where you want to go and you have no idea what button to push. So it's tax season and she wants to find tax forms. She was really nice about it but she's also frustrated. And she's also spot on. The idea behind the new navigation menu is that you shouldn't have to understand the state bureaucracy to figure out where you need to go. So for many visitors, this is an improvement. But for power users like her who come to their website more frequently, who know exactly where they want to go and what department to find it in. The new design has actually made things a little confusing and cumbersome. How many people here have shopped at IKEA? So at IKEA stores, visitors walk through this long winding pathway, right? For shoppers who only go there once in a while. The long path, the installation displays are really user-friendly. But if you know exactly what you want and where you want to go to find it, you just want to go in and out. It would be really frustrating if you couldn't take those staff shortcuts, you know? And she feels like the person who knows exactly where she wants to go, but she's not allowed to take those shortcuts. So I'll call problem number two, the IKEA problem. The IKEA problem is more nuanced. We don't have 300 microsites on mass.gov anymore. So the problem here isn't too many faces per se. The problem now is an expressionless face. An experience that doesn't acknowledge the different circumstances in different relationships can be just as frustrating as a bunch of totally different faces. Problem number three, this is a true story. In 2009, a young entrepreneur registers his small business as an S corporation with the state of Massachusetts. He faithfully pays his taxes to the Department of Revenue on the left. For 10 years, he pays unemployment insurance to the Department of Unemployment Assistance on the right. And eventually he decides it's time to shut his business down. He's never shut a business down before, so he calls all the different state organizations that he's done paperwork with to figure out what he needs to do to shut down his business. And do you know what he found out? For several years, he'd failed to pay a registration fee with the secretary of the Commonwealth in the middle. As a result, for years, he'd been paying taxes for an S corporation without having the benefits of limited liability protection. And that was the whole reason he got incorporated in the first place. So for people not familiar with the details of starting a business, this is like an airline ticket selling a boarding, excuse me, an airline selling a boarding ticket and a seating ticket separately. What if you went to the airport and you presented your boarding pass and then you find that you can't actually fly anywhere because you didn't buy a seating ticket? Why would we sell these things separately? Having DOR, DUA, and Secretary of State interface separately with businesses in this way is confusing to say the least. So if you'd been sued, his family, his home, his life savings were all exposed, he had no idea. I'll refer to problem number three as the plane ticket problem. So there's a survey of the landscape. Can we all agree this is suboptimal? Yes, okay. Today, more than 76% of the people in Massachusetts interact with state government online. That's way more than any other medium. In 20 years, we've gone from zero to 76%. And we're only just beginning to understand how this is changing people's expectations about their relationship with government. Let's look at these analogies in the context of a relationship. So the orange problem is about look and feel. Look and feel sets expectations. When an interaction doesn't meet expectations, it can be jarring. It can be confusing. If the grocery store puts oranges in the snacks aisle, what other produce might be scattered about? Am I in a grocery store or am I in a flea market? What else is in the produce section? Is that really the produce section or is that really the big bin section? I can't help wondering what else I don't know. Am I even in the right place at all? The IKEA problem is about navigability, which turns out to be about more than usability. Usability is key, but navigation is also about being seen. It makes me feel like this website is for me. Or, decidedly, this website is not made for me. The plane ticket problem is about success criteria. Success metrics have to shine a spotlight on alignment or misalignment between what our platform provides and what our constituents need. If a passenger needs both tickets to fly, it's not enough for the platform to just make sure we're selling boarding passes and seating tickets. We let people down and we create distrust every time we fail to sell the same number of both. We can't see this with our eyes, but we can see it in the numbers. These are the kinds of success metrics we need to surface and track to make sure that we're meeting people's needs, not just setting their expectations, but also meeting their expectations. This is how we'll build and sustain a trusting relationship. When we talk about creating a single face of government, what we really mean is fixing the dysfunctions in this relationship. So now I'm going to talk about work in progress on all three fronts. Some places where we've solved problems, or we hope we've solved the problems, some places where there's work in progress right now, and then some unsolved problems we see on the horizon. Look and feel is about setting expectations. So, I know where I am, I know where I need to go, I know what I need to do. I know the oranges are in the produce section. Before our project started, Mass.gov was a portal website, a site that points to lots of other websites, including 300 microsites that were hosted on Mass.gov. Portal websites were all the rage ones. Who knows what we call a website that points to other websites in 2018? Search, Google, right? So, in the portal world, the rideshare driver needed to find her way to the TNC division of the DPU under EEA's microsite, like finding an orange in the snack aisle. That's up at the top. This month, MassRMV.com is going to shut down and redirect the RMV's organizational site to their new organizational page on Mass.gov. So, it's going to look and feel like the rest of the state organizational pages in that bottom row. Driver's information of all kinds will be found here with one look and one feel, and since the DPU's background check process is done by email, reorganizing the content in this way is all we need to do in this circumstance to really present a single face of government. Presenting a consistent look and feel across web properties goes a long way to setting people's expectations and building trust. Here are a few applications that tried to make the leap to a common look and feel by extending the visual design concepts from Mass.gov with their own custom implementations. The RMV just replaced their mainframe-based system Atlas with this system that you see here on the right, and this is the first big transactional application to dip their toes in with us. Next, MassTaxConnect on the left updated its look and feel to be more consistent with the Mass.gov colors and typography. There are also several who have not started down this journey with us quite yet. A few of the big ones include UI Online, Unemployment Insurance in the back left there, HHS Virtual Gateway for Food Assistant recipients, Food Stamps, that's in the back right. The Secretary of State's website and Business Portal website in the front. The interest and the initiative that we see from RMV and from DOR on the previous slide is really terrific, and some of the folks on this slide are also very enthusiastic. We just haven't started yet. There are also a lot of really terrific people here. At the same time, all the HTML, CSS, JavaScript, and images, all the stuff that makes up the visual designs on the previous slide is being created and maintained separately. So at best, over time, it's gonna be a lot of work to keep all this stuff in sync. And at worst, with this approach, the applications can still evolve in really different directions, and we can end up with things that look and feel really different, a really disjointed user experience, you know, sometime down the road. So wouldn't it be great if we could package up the design system that we made, the Mayflower design system, into something more reusable across different applications to make it less effort to get everything in sync and then keep everything in sync? Look at all the websites that are using Twitter Bootstrap, or Google's material design. Packaging up our design system in different categories to be reused across web properties like these could be a major accelerator for web development projects. It would shift the default to keeping the state's digital ecosystem in sync, evolving the visual design and user experience together, and that's what we think the future should look like. This fall, we're gonna begin collaborating closely with agencies that maintain transactional applications to start implementing a consistent design that hopefully requires less custom implementation and works more like Bootstrap or material design or other modern, successful design systems. But before we do that, we wanna validate our approach. So we're eating our own dog food, we're testing our recommendations out on separate applications that we manage and maintain, and we're working our way through our own recommendations through the bumps and making revisions first so that when it's time to start making technical recommendations to other government organizations, those recommendations have been field tested, proven, and ideally we have reference implementations that we can point them to. So, here's what that's looked like. Mass.gov's theme is using markup from a component library that we call Mayflower. This is Mayflower. Mayflower is built with Pattern Lab. You can check it out at mayflower.digital.mass.gov. In theory, it's platform agnostic, but before we recommend others use it, we're eating our own dog food by trying it out on other applications that live off of Mass.gov, or at least off of the Mass.gov Drupal CMS. Our first standalone Mayflower app is this wait time widget you see in the top right of this Mass.gov page here. This is actually a simple single page application that's hosted on GitHub pages, and it's being iframed in to be embedded into this page here. So, setting this up, including all of the assets, walking through all of the steps that a developer would have to take to reuse Mayflower helped us clarify all the moving parts that need to be explained or simplified for it to be reasonably reused by someone else. Next, we wrote quick start documentation and created some reference examples to show people how to make simple standalone applications like the wait time app that can be embedded into Mass.gov with iframes. And those gave us a great opportunity to figure out how to version and publish our assets on a content delivery network, and then include them into apps in a way that many people are familiar with, including libraries like jQuery, so they don't have to download and build Mayflower all by themselves. Next, we started making sites. First, we started with a simple static website for the governor's new proposed FY19 budget, then a React-based single-page application, which is live now at search.mass.gov. Search is our strongest reference implementation. Visually, it's generating Mayflower markup, so it looks and feels perfectly consistent with Mass.gov down to every link, every button, on every screen size. And then if we peek under the hood, it also clarifies some of the ways we foresee the whole in this new ecosystem being greater than the sum of its parts. So I'm going to talk through a little bit about how that works. The search results that you see here come from 68 different Massachusetts websites, and the results are being powered by Google's search API. The promoted results in the right on the top are filter and the filters that you see on the right example on the left are on a soon-to-be-released news tab, so you don't see this live yet, and we'll have similar functionality on the laws and regulations tab coming this summer, and these are being fed by APIs published on mass.gov. So here, we're just starting to dip our toes into decoupled content management. The Massachusetts sites that we're searching across all these search results, if they publish metadata in their HTML, according to Google's specs, that metadata gets surfaced through Google's APIs to present richer search results. So on the left, it's circled, I don't know if you can see it, they're promoted results, they're promoted links that give you the most frequently trafficked links for the page that that is showing. So it gets you a much faster path to what you probably want to get to if that is the result that you're looking for. So if you follow our metadata standards on any website in Massachusetts that is getting searched by search.mass.gov, you have these specs you can follow to search richer search results in the search interface. So one day soon, anyone implementing one of our component libraries will get the metadata that we're talking about here included by default in their markup. And then let's imagine a situation where we make updates to those metadata standards. We start evolving our rich search results and we want to do something newer and richer. We decide to do five star ratings like most people have probably seen in recipes search results. So let's say we want to do five star ratings. So if we make this change and then a bunch of custom markup has to be changed on separate applications to actually bring those five star ratings into existence, changing the HTML, changing the CSS, making all of that work over and over again in custom implementations is a lot of work. There's a lot of overhead there. But if you have a modern build process and we release a new minor version that has no breaking changes and your modern build process rolls out that new version of Mayflower and included in the new components, there's an empty attribute which just needs to be populated to include five star results that could be fed to search here. We've really dramatically lowered the overhead. We've made it so much easier, so much faster to roll enhancements like that out into the ecosystem. So in a landscape that looks like this, look and feel is about more than making all of the websites match a Photoshop file. Now we're talking about consistent look and feel including interaction design. So whether it's rolling out something shiny and new like a five star rating or improving usability of touch interactions on all of the accordions on all the mobile devices. This is our vision for building an ecosystem that we can sustain, that has a consistent and evolving look and feel. So that was all of the work that is solved and then in progress. Now here's a quick look at some unsolved problems. So we're still figuring out where the boundaries are in terms of promoting a consistent look and feel. So before we started, anyone on mass.gov could have their own look and feel. They could have their own brand. The pendulum has now swung in a very different direction where everything on mass.gov has a very similar look and feel. We like to say consistent but not uniform. We're not saying everything has to look exactly the same. Presumably Mayflower will have sub-brands and there will still be some different organizational identities. But we assume there are times when it does actually make sense to have a separate brand. GovUK has clearly identified and documented one really good use case for this. If you have a marketing campaign, it's a six month project. You launch a new website. It's got its own little promotional materials. Maybe that's a really good example of it's got its own brand, it's got its own look and feel. There might be other use cases where it makes sense to have a separate look and feel when do you evolve from being like Amazon.com on the left there to having something that actually makes sense for the user experience. Not just a request that someone hatched because they thought it should look different, but when does it actually add value for the user to have a separate brand? We're still figuring out where to draw the lines. If anyone has ideas, I'd love to hear about that afterwards. Onto navigability. Navigability, as I said earlier, is about more than just usability. It's about making people feel seen. Not only do I know how to use this website, I know the website I'm on is meant for me. It knows what I need and it's trying to give me the things I need to get the job done. If I know the shortcuts, Ikea gets it, it lets me take the shortcuts. Mass.gov is the center of the Commonwealth's digital ecosystem. Its purpose is to get people to the thing they're looking for, to provide constituents and businesses with the fastest, easiest path to whatever thing is that they're looking for. On the portal website, all 300 microsites had their own navigation. Navigation mostly mapped to organizational structure. If you only visit mass.gov for work and you only go to one organization and you know how to find your way to the exact department you need, this wasn't a bad arrangement, but for about 80% of our visitors who don't necessarily know the department they want to go to, who may only be passing through Mass.gov on their way to a final destination, which might be some other web property, this was confusing. Here's a 2x2 that depicts the basic landscape as best as we understand it today. About 80% of the traffic fits in this bottom left quadrant. These are people who come to Mass.gov fewer than 10 times a year. They want to get in and out. Many of these interactions are transactional. They get a driver's license, a phishing license, apply for unemployment insurance, apply for mass health. None of these transactional interactions happen on Mass.gov. They're finding their way through Mass.gov. Sometimes they want to find information, but usually it's simple information. Find information about a park. Find contact information. So the new website's navigation was designed, tested and optimized for improving navigability for the users in the lower left quadrant first. And it's made things better for a lot of these people. We used a tree called Treejack to test our information architecture that is represented now in the main menu of Mass.gov. Navigation also includes other kinds of wayfinding, not just the main menu, right? So we shifted the whole focus of the website away from being about organizations and organization-centric to being about the services that government provides. That's how we're making it constituent-centric. Our data models are content types, revolve around the idea of services, finding a service, understanding a service, and helping the user take the next step for the service that they need. These steps are usually explained in what we call a how-to page. Many of the how-to pages walk you right up to the doorstep of the transactional application that you need, and then it leaves you there with the application or whatever it is you need to do. To be clear, our job is not done, but people on the bottom left feel more seen. They can navigate more easily, satisfaction has gone up, and when we walk them to the doorstep of a transactional application that looks and feels right one day soon, it's going to get even better. But what about the top right? While the power users at the top right were not our main focus early on, we're trying to just make the biggest impact with the fewest number of changes early on, our mantra for people on the top right was we want to do no harm. We want to improve things elsewhere and we want to make sure we're not making anything worse for the folks over there. So one thing that we underestimated is that this group, the more frequent users, the people who use Mass.gov as part of their job, have the most muscle memory of the old site. So change, even if it's an improvement, is disruptive. So right now, we're in the midst of research and design and some development work has begun to better understand the needs of this group, minimize disruption and make sure that they're really getting improvements, major improvements to their workflows as well. Most of that so far is all about navigability or mostly about navigability. Here's a screenshot from the recording that you saw earlier. Navigability is about more than usability. It's about making people feel seen. Right? So when our tester concludes that Mass.gov, the main navigation on Mass.gov is not for her, she checks out. She doesn't realize that the link that she's looking for is actually right in front of her in two places. So instead, she feels like an IKEA shopper who's been told that she can't use the shortcuts. So to be fair, we need to own the lack of performance of the content here, right? We need to test some improvements to the text, see, you know, she obviously expected to see the word departments and we're using phrases like state organizations and state offices and that didn't land with her. But relationships are a two-way street. We need our users to say to themselves, this is for me. It anticipates my needs. If I can't find something, maybe I should read more of the text, maybe I should keep looking. So we're still doing research, we're still doing design testing, but we do understand enough about some of these problems that we've begun work. So on the right, you see some of our user research. This is where we segment this top rate quadrant and define power users. People who visit the website 10 times or more in a year, that makes up 25% of our traffic. These users are unlike the bottom left quadrant, people who want all the technical details, they want long form content, not just the simplest, fastest, smallest bite-sized chunk. In some cases they need hundreds of pages, chapters, they need to navigate forward and backward through all of that long complex technical material. So we just released, after the recording that you saw, at least these three new features that you see on the left, these represent three different feature enhancements, all related to long form content and navigability of different kinds of long form content. Long listings of building codes, many hundred page documents that have chapters in them and tables of contents. So hopefully some of this is beginning to meet some of those needs. Coming down the pike, we also are very much working on improvements to the way search works, so we know we can make a major impact in the power user's experience with search by enabling them to scope their search to just focus on content from a single publishing organization. So here you can see that on the left in the new news tab that's about to come out on search.mass.gov. On the left in the filters, you can filter by organizations. We're also experimenting with possible designs to put a similar filter in the search bar so that you could get it in the insight search in the header across mass.gov. So then moving on to unsolved problems, there are other ways that we are talking about search, excuse me, about improving navigability. There are nothing conclusive here, but this sort of extends to the range of topics that we're discussing and questions we haven't totally answered yet. So on the left you see amazon.com. If you go to Amazon in the upper left, you can actually select from all of Amazon's departments, and then you get a little sub-nav below their main navigation menu that lets you navigate just that department. So that's something we could consider. We could consider having a sub-nav that extends across our main nav. Even amazon.com, which includes a huge breadth of content and departments in that one navigation experience, even amazon.com has some sites that don't fit onto amazon.com the domain or into the same navigation experience. So on the right you see two examples, services.amazon.com, which is for people who are putting content onto the retail store on amazon.com, and the other one is aws.amazon.com, so that's their main navigation experience. So we're still figuring out exactly where these boundaries should be drawn. When do you draw the lines to define a separate site? And to what extent do we want to follow this kind of a navigation pattern? And what navigation elements, if any, belong as part of our design system in the look and feel that ripples across the ecosystem, if not in the main navigation in the header with other wayfinding queues. Let's talk about success metrics. Success metrics align what our platform is designed to provide and what our customers need. Excuse me, what our constituents need. So we can't stop at setting expectations. We have to meet expectations. When we don't meet expectations, we lose trust. A person wouldn't expect that it's even possible to buy a boarding pass without getting the seat, right? When transactions happen online rather than between human beings at a ticket counter, we need more than common sense to monitor for nonsensical situations like this and then course correct. To proactively identify where our digital platform is or isn't meeting users' needs, we need success metrics. All content has a purpose. When content comes into contact with participants, it either lives up to its purpose or it doesn't. If we know our success criteria, we can use web analytics to identify where we're meeting people's needs and where we're missing the mark. Has anyone heard the saying you can't manage what you can't measure? Some nods? So two years ago, we were measuring very little about digital interactions with people through our websites. Optimizing to meet users' needs was really difficult and it was all ad hoc. We had big blobs of unstructured content that could be used for anything, for all of the pages on mass.gov, and so those pages were masters of nothing. Today, on our new platform, only recently launched and still under active development, we have not solved the problem of the young entrepreneur who accidentally let his business registration expire while continuing to pay taxes. It is still possible to buy mass and ticket separately. But we hope that we've put in place a new foundation that will enable the DOR, the Secretary of State, or any other state organization in Massachusetts to see more clearly what the user's experience is, where we've left people at dead ends or without seats and where the platform is serving constituents in the ways we'd expect. Before, gleaning these insights and then iteratively improving the constituents' experience was impossible for most state government organizations. Now we have tools and we can be methodical and we can be data-driven. We can articulate performance targets and then we can test, try, try, test, and iterate until we hit them. And then we can raise the bar and we can keep improving. And we can rely on data and real verbatim feedback from users to let us know if we're actually hitting the mark. So being data-driven and being user-centric sounds great, right? But what does it really look like in practice? Getting from Google Analytics or Google Analytics out-of-the-box dashboards to data-driven, constituent-centric government product development seems like a really big leap, right? So here's our approach. Early on, we built a proof-of-concept analytic, we built a couple of proof-of-concept analytics dashboards that can be embedded into the Drupal authoring experience. So you can see on the right that these are analytics that are actually getting embedded. That's a real dashboard in a real Drupal no-dead-it screen. The dashboard-building data visualization platform we're using is called SuperSet. It was originally built by Airbnb and now it's an open-source Apache project. These dashboards that you see right here were mostly for internal use partly validating that technically all the pipes were connected and then also exposing enough data and information to our internal team that we could start running early content performance experiments. Then last fall, mass.gov went live and between October and November, excuse me, between October and December, we did a series of content performance experiments. We set out to identify underperforming content, improve the content and then improve the improvements with data and then come up with repeatable recipes our customers, the government agencies that use mass.gov, the government organizations that use mass.gov can use to improve content this way by themselves. All in all, our experiments were pretty successful. 14 out of 15 content experiments demonstrated improvements. In this example, traffic to request unemployment benefits doubled after we made revisions with language in their titles and their short descriptions. And then we also promoted this page from some related content where there were no links linking you over to this related unemployment page. On the cancel your vehicle registration page, we made some edits clarifying that canceling your registration and returning your plates means the same thing. After we did that, that significantly improved the number of people reporting, yes, I found what I was looking for on this page. So in aggregate 73% of our performance experiments increased positive feedback. 67% of our improvements increased page traffic and 26% increased the outcomes rates on the tested pages. Outcomes rates are the thing the page was made for, if it's an applied page, people click the apply button, for example. So this is a start. So now the question is, can we package up this information in a way that makes it possible for our partners, the park rangers, epidemiologists, experts in fields other than web publishing. We'll understand how their content is performing and then be able to take specific steps to improve the performance of their content. In the next quarter, so this past January through March, we ran a pilot to find out. Think of this as a design test like the way you might test information architecture or wireframes, but for our analytics dashboards. So before we write ETL scripts and code dashboards and embed those new dashboards into our content management system, we want to prove with ad hoc spreadsheets and charts, that we know how to produce performance indicators that will actually be useful, usable and used by our content managers. So here's how we did that. We took the nine different charts that emerged from our experiments and we grouped them into categories that our customers will find understandable. So those four categories are findability, outcomes, content quality and satisfaction. So we grouped all mine indicators under one of those four categories. Findability means our people finding what they're looking for. Outcomes means all our pages effective at enabling people to complete the task they came to complete. Satisfaction is the constituent experience fast and easy? Are they saying, yes, I found what I was looking for or no, I didn't. Content quality or are we providing content that is easy for constituents to read and understand? Are there any broken links on the page? Then we supported four agencies, Department of Revenue, Department of Public Licensure, Department of Unemployment Assistance and Mass Wildlife in running 150 content performance pilots to validate or refine the proposed content performance indicators. 80% resulted in improvements to their content performance. So this quarter, now that we've validated those indicators, we're in the process of building out those ad hoc indicators as dashboards that we can embed into the CMS beginning in the new fiscal year on July 1st. And now we're going to use those indicators for FY19 for statewide content performance improvement targets. So we're really excited about that. Here are the wireframes that these are not all the wireframes, but here are a couple of the wireframes. So outcomes this is where we're measuring if a page is resulting the outcomes that the page is meant for. If it's an apply page, are they clicking apply? If you're supposed to download a form or people actually downloading the forms. Content quality is measuring on two indicators. Is the writing here a sixth grade reading level? And are there any broken links? Broken links has a huge impact on people's satisfaction with the page and usefulness of the page. Nose per 1,000 page views. So this is one of the indicators for satisfaction. So are people saying, no, I can't find what I was looking for? If so, can you make improvements to your page to drive that number down? Last but not least is traffic. Is anyone looking at this page? So for each of the different indicators we give some tips on best practices, things that you can do to improve the content. And then hopefully hopefully that will lead to content performance improvements. If we can get 100 of our 400 mass.gov content managers to interact with performance dashboards like these to help us hit our FY19 page performance goals, this will be a huge leap in the right direction. But there are still a few pieces that we need to put in place to really make this platform to really make this a platform that's equal to the plane ticket problem. So this year in FY19, our performance targets are going to be all about page performance. Getting all the content right in the content types, getting all the content into the right content types, making it easy to read, living up to the design that was intended for it and tested will have a big impact. But in the long term, we need dashboards that measure performance across multi-page and multi-website user journeys. So, here's how we intend to start chipping away at that iceberg. This summer, our user experience researchers are setting out to map full user journeys for each of the top 20 at least one user journey for each of the top 20 traffic services on mass.gov. And then zero in on one of those to work on really specific improvements. And hopefully one of the results of that research will be recommendations about possible indicators that we could follow to measure performance of user journeys. Then in the fall, our content team, the same content team that last fall did experiments on page performance. Next fall, that same content team will work on testing out indicators for journey performance across multiple pages or sorry, I skipped one of our summer steps in parallel to the user research that we're going to be doing on journeys. We're also going to be doing some technical exploration to figure out the best solution for cross-domain tracking for analytics. So depending on what pieces we have in place for cross-domain tracking and the indicators that we have to experiment with in the fall, we'll spend the fall working on content performance experiments for journeys. Then we'll take validated indicators if they come out of that. And in the winter between next January and March we'll do another round of pilots testing, effectively user testing. Are these indicators useful, usable, and used by our pilot customers? And if we can get to yes next winter, then next spring we'll be building out the next major iteration on our content performance dashboards to include indicators that measure the performance of the user's journeys. If all that unfolds quarter by quarter the way that I just laid it out, the platform will be humming in FY20. And if not, I'll have to report back to you next year about whatever dragons we've learned or lying waiting for us in that neatly charted course ahead. The mission of digital services is to leverage the best technology and information to make constituents interactions with our state government fast, easy, meaningful, and wicked awesome. In our day-to-day work, we talk a lot about fast and easy. Meaningful and wicked awesome are a little more elusive. Single face of government catapulting people's relationships with government into the 21st century. This is where we're working toward meaningful and wicked awesome. Number one, setting people's expectations. Number two, making people feel seen, acknowledged, and respected. Number three, understanding people's needs, knowing where we meet them, where we don't, and then continuously improving. And then reiterating on government for the people. That's wicked awesome. If any of you have ideas or suggestions or code that we can collaborate on, or if you know smart people that we should hire, please come talk to me. I think we also have 15 minutes. So if you want to hang out, I would love to hear questions and have some conversation here at the end of the session. I know that I'm standing between you and parties that have started at 5.30 already. So I won't be offended if I, if you don't really love to hear anything you have to say. I have two slides I need to show. I want to remind people about the collaboration sprints on Friday. Also encourage people to tell us what you think about the session at the URL here. And I really want to get to questions, but I also am just realizing at this moment or at the end of my presentation that I cut a really important little chunk of content. So I actually want to just say it out loud. Because I talked a lot about the plane ticket problem that the delta between the boarding passes and the seats. So there actually is a really good historical reason why things work the way that they work. And so I didn't mean if it came off in any way as Cavali or taking swipes at some of our wonderful colleagues in those departments. They're actually really important historical reasons. So in the case of Secretary of State that's actually a constitutional office in Massachusetts. So we specifically separate out record keeping from the rest of the executive branch and there are good reasons for that. It makes it really hard for a governor to cook the books or rig elections when you separate out that piece of government. So part of putting the right platform in place is about building the foundation that we need so we can build digital experiences that are meeting our constituents needs and not to take a sledgehammer to the oldest constitutional democracy in the United States. So I just wanted to make sure I said that because I feel like that was an important bit of the story that somehow got missed in the last revision to those slides. So thanks for playing along with me on that one. Please we have 14 minutes left. Does anyone have questions or comments? Have we contemplated using something like glue on to make connections our experts aren't making? Is that the question? I would love to hear about glue on not familiar with glue on actually could I trouble you to come up to the mic just because then the recording will grab our conversation as well. So you're getting and like data across all the sites now starting to I assume we're starting to right you have experts that know things like someone right common sense experts or deep experts that know that someone with it that's wants to get a share drive permit also needs a driver's license and so those are two separate kind of avenues to make connections one is what is the data that people are doing one is what is common sense people say that people want to do right since you have you're starting to get data you can do things like you could throw it into some type of Bayesian analysis or do some right process the data and start making let the AI start making assumptions like that that experts don't know enough to make right like for you know you're talking about the guy who has the S corp needing you know three different things and not only knowing what two of them or whatever there's probably 1500 5000 different sets of things that across multiple departments that you don't ever get in front of an expert to find out about right so once you start getting data like you are right you used to have we just had 300 microsites so you're getting now a view across 300 different little fingertips that you can start doing some deep learning to find that someone who is you know I know this isn't statewide but someone's getting a burning like a building permit right also needs to transport hazardous material or you know things like that you know like that you'll never be able to review all the experts that would need know to make those associations and because you're not like you know like a Kroger's where you know everybody knows to put the oranges produce you know but the nuts and snacks you're not limited to physical you know physical connections you can start making digital assumptions that for some reason someone who is visiting you know some some department in the state that has to do construction really it wants to view hazardous material it doesn't know it yet in fact not only like you might think he's looking for and hasn't found it you might know he needs to find it even before he knows to look for it you know so then you take your data and you start doing it you start using the AI to make those kind of connections that you'll you'll never get from experts because you just can't interview all of them yeah I'll check it out that sounds that's a great suggestion thank you the two places that we're using machine learning today are in automating away some some wrote tasks so we had 150,000 files in our old CMS that we needed to migrate into our document repository and tons of that needs to be searchable for public records compliance but didn't have any meaningful metadata on it so we built these text classifiers the librarian that we worked with at the time estimated that we can now do in some number of hours with the machine what would have taken a librarian 36 years to do with metadata tagging and then the second place that we've been experimenting with machine learning is in tagging the user feedback that comes through mass.gov so we get something like 4,000 comments on a given day and you know trying to figure out what the critical mass of comments is you know about a thing that is broken online or offline you know we theorize that machine learning can really help us zero in on insights like that so we actually decided to start with just manual tagging at first until we get a better training data set and then once our users get used to the tags on the feedback we'll probably turn that back on please so this is a very good talk because it's very relevant to me I work at a university and we kind of have a lot of the same issues especially with like departments you know you talked about how does somebody find the information they need we actually paid a company to come in and do what was called like student focus groups and what we discovered is our site actually works very well for people that use Google the way that they interact is they'll go to Google search for what they want hit a page get what they need go back and that's the process so now it's kind of like changed our approach to like redesign because a lot of people look at our site and it's kind of ugly and crude but it's actually functioning very well and now we're kind of mapping out like the user journey of like what do we actually need to provide these people you know as an experience so thank you for sharing what you've done and I would encourage you like to do that where you just ask people how they're using it with Google thank you we actually found really similar information 70% of the people were landing on an inside page skipping the home page of math.gov mostly coming from Google and I've done a lot of work to try to optimize for that and continue you know working on that it's a journey jump back to kind of the first interfaces you looked at with the navigation and the power users versus the 80% to what extent did you guys consider personalization strategies for changing the whole experience for those different types of users is that something you looked at and said no we don't want to do that because it's going to bring unintended consequences whether maintenance or just confusing the users that kind of stuff but that's an interesting question I think philosophically and practically. For sure. So I'm personally very interested in personalization. I think there are two main categories of obstacles for us there right now. The first is just as a matter of practicality thinking about where can we have the biggest impact on the most people with fewest changes going from where we are to personalization when we started and even still today we have pretty limited information about the different segments of users and so our ability to actually personalize is limited and you know hopefully a lot of that's going to change really soon but additionally there is a real human component to doing successful personalization you have to write a lot more content to do personalization well and so I think from a content perspective we're still getting the right content in the right content types is a really important starting point. Once we're really great at writing an excellent how to page then let's personalize the how to pages or an excellent X then let's make the variants of X. So we're still in you know sort of crawl on the crawl walk run toward personalization once we get there so we talk about it we see that there's a really exciting opportunity there it's definitely something we need to be sensitive about because it's the sort of thing that makes people feel creepy about government and so once we do get there and actually I think before to do it right before we get there we need to talk a lot about how we're going to be transparent with the users how we're going to be respectful of their privacy and at the same time you know let them know how we're leveraging this tool so there's a lot of unsolved stuff there have you had success with personalization that we should know about or think about I'm afraid not I'm in Drupal modules we've tweaked content for different audiences but I wouldn't claim to be an expert there it's just always been something that's fascinating and this is a perfect use case in my mind we have so many different constituents across the state there's such different types of information I don't think I have a silver bullet I mean companies here have products people have written modules and stuff but I don't think I've solved it by any means okay thank you if folks want to just sort of walk up to the mic I think we don't have a clear sense of how many people want to talk and I will gauge my interactivity accordingly and probably find people afterwards I was just a little bit curious about the dashboards for the analytics that you're leveraging there how difficult was that to set up I mean because that I think would be fantastic if every single page you know content editor can go and look at the edit screen and see analytics about you know how their page is performing and what they could do to improve it you know we've looked at what is it Yoast I think they created a module for Drupal to give you some SEO tips things that you could do to improve your content but I'm really interested in that analytics because I think people they want to make really good content but they don't know how to do it and it would be good to have that dynamic information so I'm interested in that could you talk a little bit more about how you guys set that up sure the very hardest part is not actually technical it's deciding what the page is for so those dashboards are only meaningful if you're able to go into your analytics and then distill have you guys all interacted with Google analytics it's really exciting at first but then you say what do I do with all this information right it's a beast we use Google analytics or we've attached it to our site let's say that but I know we're not using it to make good decisions about how to improve our content yeah the hardest problem is not a technical problem the hardest problem is not how to get the dashboards in the page even though that sure is fun to zero in on and figure out but the hardest problem is deciding what is our content for what is this content type for and how are we going to measure whether it's successful or not if you can answer that question with real clarity of purpose everything else can come into focus a lot more quickly so in our first iteration we used RShiny that was not ideal for us because we want for people we don't want to have to go to the data scientist every time we want to make a dashboard we want to be able to configure the dashboards ourselves the way that a site builder would build a content type so that's how we discovered SuperSet sorry not now a patchy project it is a dashboard making data visualization platform it's got a GUI tool that you can use to build analytics we are piping data from Google Analytics to SuperSet so in terms of all of the plumbing it is a place where there are a lot of moving pieces and an unsolved problem that I didn't call out is testing to make sure the pipes aren't broken and knowing when they break a major breakage that we couldn't anticipate or fix but it's one of those things that keeps us wondering exactly how this is clearly a problem we need to solve so to just walk you through the constellation of interacting pieces our markup is coming from Mayflower which we then bring into Drupal so our markup then Drupal Drupal and Mayflower basically both have a hand rendering in markup information that Google Tag Manager knows about so that's what the events are keying off of and that's how intelligible data is ending up in Google Analytics is through Google Tag Manager there's a data layer module for Google Tag Manager that we're using for Drupal in Drupal 8 then we have an ETL script extract transport load script that takes data out of Google Analytics with their Google Analytics API and then transforms it and puts it into Postgres and Superset is running against that Postgres database and then Superset lets you sort of monkey with CSS so with the dashboards we're embedding into Drupal we use their CSS tool to strip out the Chrome and then embed the dashboard into the content management system and we've got a module we wrote a module to put the tabs into the content types so that you can decide either at a per content type level or at a per node level I think there's another one that I'm forgetting but you can configure what dashboards show up with what content types and that all gets stored and config we'd be happy to share that with you if that's of interest to you I think that's sort of like the smallest piece of the puzzle but that's the constellation of pieces that would be wonderful it'll be interesting to hear what happens next year but a wonderful blog post that details all this information would be grand thank you thank you hi my name is Kim I'm from City of Tucson and we're we're in the planning stages of going through this exact same process moving from a department portal site into a service based website the question I have is about your content editors the lines are going to be blurred in who provides what information have you had any problems with that have you had any resistance and rumblings and how did you address that people anticipated huge problems people thought the sky was falling and yes so we did hear a lot about it we actually haven't had any problems so we when we started this project we were pretty early in the life cycle and so it was a pretty clear decision for us the engineering effort that was going to go into making organic groups or groups work to try and draw boundaries around content but then at the same time trying to get people to collaborate on content where they actually should be collaborating because there's overlap was too mind bending and so what we said to our users early on was let's put this question off it's going to be an editor or an author so if you're an author you can draft content but you can't publish it if you're an editor you have privileges to approve and all authors can propose new drafts to anyone's content and see other people's content and all the editors can publish or unpublish any of the content so presumably that's a responsible individual so big presumption big presumption right so we haven't blown our toes off yet we have seen a few places where this has been a major major benefit so the places where this is having a huge positive impact so far are really outweighing the concerns which are still mostly hypothetical we've had a couple places where people stepped on toes but then the authors talked to each other and worked it out pretty quickly there was one place where something got deleted now we have set things up so it gets moved to the trash it doesn't get deleted we can take it out of the trash we set up a notification system to notify people watching content of changes so now for the first time ever DOR and Quartz who both are very much involved in child support they used to have their own canons of child support information and if you were a constituent trying to figure that out it was really confusing now we have one child support page and they can both edit it and they get notified for each other's content changes and we've recently put in place two more mechanisms that made people feel a lot more comfortable with this one is restricted access to nodes and then now a counterpart to that is restricted access to files before the content's published so by default it defaults to open so if it's unpublished any author editor can see it but if you have something that actually needs to be restricted you can say only these three people should be able to see this thing that I'm working on until it gets published and when it gets published the restriction comes off it gotcha thank you hi my name is Lois Nielsen from the state of North Carolina I want to thank you for your talk this has been very enlightening one of my challenges that I've been trying to deal with has been how to help subject matter experts who care about their content and maybe don't care about the web so much how to help them to see why they should be reading their web pages why they should be interacting with them and I really don't have a question I just want to thank you for and I'm totally planning on stealing everything that you've said about the findability outcome content quality satisfaction and these seem very inspired and helpful for folks who again who really have no idea about web communication so I guess I do have a question I need a little bit more information in terms of what the actual dashboards are I'd be happy to show them to you I'd be happy to share the code with you too can I ask you a question so we went live with our digital commons in 2015 so NCGOV and 10 agency sites we did not go the route the UK.gov route or the mass.gov route and we have a platform we started with 10 in 2015 now we've got 35 or 40 folks coming on that's everybody wants to come on our platform which is amazing but we're on seven we're not able to think about eight right now so we're still figuring it all out but what we don't have what we had in a previous administration was the impetus for a unified look and feel what we have with our current administration is eh we don't like what that governor did and we don't like his logo which got bad press by the way so that didn't help and so what we have what we're sort of dealing with is a governance problem where we have the IT part of it figured out but communications area they all want their own unique identity so we're actually building more variability into our templates rather than less and I could have many opinions on that but it doesn't matter so thank you just a quick one from a high level I'm Cory I'm from Habitat for Humanity and the governance are a lot more bureaucratic than we are but we're pretty bureaucratic so how did you sell the organization on the idea of really diving into data driven kind of decision making and then I looked at your homepage and like how do you determine what gets space on kind of featured space areas because I see like be a lifeguard, file your taxes or hey we have an opioid economy like that's very different things and you've got hundreds of sites how do you mitigate various stakeholders with different levels of strength and bureaucracy and all of that what's been your approach to that as a department we have a good fortune of having people all the way up to the top that are really into data I'm pretty sure the governor is really pushing our secretary for data our secretary is very very much into data driven and our chief digital officer who is also our chief digital officer is very much into affecting change to make government data driven and constituent centric so it's more like we need to have a really good reason to not be using data and to not be getting other people to use data if we're not using data yeah it's true thanks for the plug okay I think we're a little past time so if people want to stick around and talk I'm in no rush and I really appreciate all the comments thank you so much