 My clock says 2.15, so I'm going to go ahead and get started here. My name is Brian Hirsch. My name is Brian Hirsch. I'm the product owner for mass.gov at Massachusetts Digital Services. And I'm here with Moshe Weitzman, who is one of our technical architects on the project. I think we have a bunch of digital services people, as well as Palantir and Valir, who have been helping us in the room. So I'm here to talk about our journey to becoming a user-centric, data-driven organization. And before I get started, I just want to see a show of hands to get a sense of who's in the room with us. Can I just see how many people here work for a government agency? Wow. Vendors, contractors for government projects. And anyone who does not do anything with government? OK, great. Thanks for being here. So we want to be data-driven. We want to be constituent-centric. But this is new territory for mass IT, and we haven't figured it all out. It's definitely still a work in progress. But here are some vignettes from our work, and hopefully some steps in the right direction, and some lessons learned and code worth sharing. When Governor Baker took office, he identified IT as a trouble spot. Agencies across the state government echoed his concern, and so his administration hired McKenzie to give us some advice. Here's a slide from the report they gave us a year ago. Actually, now that we've started, could I trouble someone to close the door over here for the noise? Thank you. So here's a slide from McKenzie's report one year ago. 76% of constituents interact with us online. That's way more than any other medium. Phone in person, thank you. Phone in person, mail. Meanwhile, user satisfaction is really low. On the right, McKenzie scores, mass.gov. McKenzie found, among our users, mass.gov scored 32 in user satisfaction. That's 25% lower than the best state, which is Texas. But Texas is not our competition. Our competition is the private sector websites that set our constituents' expectations for what interacting with organizations online should be like in 2017. And the best state only scored half as good as comparable private sector websites. Why is satisfaction so low? Here are a couple examples. To register a new business in the state of Massachusetts, in addition to registering my business with the federal government, the IRS, I need to interact with three different state entities, the Secretary of State, the Department of Unemployment Assistance, and the Department of Revenue. Three different state websites. If I want to find tuition assistance information, I need to find my way to and through seven different state government websites. If I do a search, I'm searching 250,000 different pages and uploads on mass.gov spread across 300 subsites, plus 400 other sites. And mixed in with the information about tuition assistance for Massachusetts residents is information about tuition assistance from the federal government to local school districts to provide special education, for example. If I find the content that I'm looking for, too often that content is confusing or frustrating or hard to understand. We want content to be easy to understand. We want our constituents' experience to be fast and easy and wicked awesome. We don't want constituents to have to understand the structure of the government bureaucracy to find the things that they need. So fast, easy, awesome. This all sounds good, right? So where do we start? So one thought might be we redo our visual design, figure out how to reorganize our content so it's easier to find, rewrite some of the things that are not easy to understand. Let's say some posts will take more time, some will take less. Let's say it averages out to about an hour's worth of work per post of unstructured content that we're interacting with to rewrite. With 250,000 unique URLs, that adds up to 7,000 business weeks worth of work. That is a paralyzing number. There's no way we can do that. So after the McKinsey report, the governor brought new leadership to Mass IT and they formed Digital Services, the team that took over the Mass.gov project right as it was about to kick off. And they gave us two focusing requirements to jumpstart the project. The first one was start at the top. So there's this tried and true management rule called the 80-20 rule. In economics, it's called the Pareto principle. And the idea is 20% of the content probably generates 80% or more of the value. So they instructed us to go study our data and figure out what the top 20 is. We were skeptical, but here's what our data scientists were able to figure out. 10% of our content serves 89% of our page requests. They also looked at the different paths that people travel through the website using Google Analytics. And each of these paths aligns with one of 20 different clusters. And the clusters map pretty cleanly to 20 different categories of government services. So now we have a sense of top services, top content by traffic. We have data on helping us understand what top is. The second thing that our leadership did to get us focused was to narrow the scope of the purpose of Mass.gov and to clarify who our target audience is. Mass.gov is the Commonwealth's flagship digital property. Its purpose is to serve constituents, to get constituents as quickly and easily as possible to the government services that they're looking for. So if there is information for government employees about procurement procedures or HR policies or a service that one government agency provides to another government agency that is distracting or confusing our constituents, that content belongs somewhere else. And if there's content that no one looks at on Mass.gov, that content also belongs somewhere else. So there's no single list of all of the top services or of all the services for us to draw from today, but so far we've identified 400 services that fit into these top 20 categories. And within those, there are 56 that are what we call G to C and G to B, meaning government to constituent or government to business. The rest are G to G, government to government. And then there's some overlap obviously with the top 10% of pages. The further we get, the clearer this Venn diagram becomes, but we think it looks like this. And at the very middle where you see all that overlap, that's the tippy top of our 80, 20 pyramid. That's where we're gonna have the biggest impact with the smallest number of changes. That's where we're starting. That's where we're staying focused until we really get it right. So great, we've got a starting point. What do we do next? We know we need to change things. We also know there's a lot that we're not gonna get right the first time. So we are, we're also working on a cannot fail project. So we're trying to get our work in front of users as fast as we can so that we can fail fast and we can learn fast and create as many opportunities to adjust as we can throughout the project. So we work in four month, excuse me, three month project-wide, tri-test iterate cycles broken up into teams in two week sprints. In our first cycle, we launched a pilot website and that first cycle ran from August through November and the pilot website brought together three critical components to get them working together. Number one, our pattern lab based atomic design system which is suitable for use in both Drupal and non-Drupal websites so that we can encourage consistency and reuse throughout our new mass.gov ecosystem. The second is a Drupal powered website and then third is our first attempt at easy to read constituent centric content written at a sixth grade reading level and following a repeatable template that can be reused by varying types of writers. So we launched the website and then in cycle two between December and March, we began to show the results to real users and test and iterate and here's what that looked like. For the 56, G2C and G2B services at the tippy top, we've been trying and refining different approaches to bring together these different pieces of content that our constituents need in a single place. We call these service pages. A service page on the right is basically a repeatable template for pulling together all these different types of content even when it cuts across different agencies or even different branches of government. So this gives the constituents everything they need to know about that service whether it's registering a business or tuition assistance or anything else. Here's an example of the latest design for what a services page will look like. The top of the page is on the left, the bottom is on the right. This is in development right now and getting here and figuring this out and figuring out all of the different types of content that go into a page like this has taken a lot of testing and a lot of iteration. So here's a look at how we've gone about soliciting different types of feedback and evolving from a proof of concept to something that's more polished. The AIGA, the American Institute for Graphic Artists hosted a public design critique. First with their own panel of experts which included the Drupal community's own Danny Norden from Pegasystems who I think is here. And then they opened it up to the audience of over 100 people which included graphic designers, brand designers, user experience designers and they gave us a lot of helpful candid feedback and actually if you've noticed that there's no logo on the pilot site today that's because we've really taken a lot of their feedback to heart and we're still working on that logo to refine it as a result. We used a tool called Treejack to test if we've actually made the website easier to navigate. The test results from Treejack, many of them came back positive. Here's one where we had lots of room for improvement though. In this test we asked users a question. You're looking, excuse me, you're thinking of moving to Massachusetts with your family which includes two children ages seven and 12. Where would you go to find information on schools? The pie at the bottom left represents users who found their destination in green and users who got lost in red. And then if you look at the pie tree on the right you can see the paths the users traveled to find their destination. So we put information on schools under a top level menu item called learning. But almost half of the testers started at living and many of them then got lost. So happily we figured this out in December and so that gives us lots of time to tune things and get this right before we go live later in the year. We partnered with the Perkins School for the Blind to test for meaningful accessibility above and beyond basic Americans with disability ADA compliance. Here you see a user interacting with a page that's WCAG 2.0 compliant. That's the latest, newest standard for ADA. Blind users who were navigating the site with a keyboard and a screen reader rather than mice and touch screens, excuse me, using the keyboard to tab through the screen and screen readers rather than mice and touch screens had pretty positive reviews. Drupal 8 makes it a lot easier for our developers to control tab order with four people who are navigating with keyboards and using the new Drupal announce API. It's also really easy for us to provide audio cues to users using screen readers when there are changes on the screen like new search results appearing or things changing. What compliance does not tell you and what we saw in this test that you see here, here you see a low site user who is navigating the web, zoomed way, way in on their screen and she never makes it from the main content area over to the right sidebar where there's critical information. For product testing by sighted users, we use a tool called Validately. Validately is a good fit for us because it recruits the testers for us and it lets us specify demographically diverse and representative groups of testers. We use Validately to test how the pages perform when a user gets to their actual destination. We wanted to know is it fast? Is it easy? Is it intuitive for people to figure out what to do to change their address or to respond to a summons or to start an application? Some interactions were confusing for both sighted and low site users and so that helped us zero in on high value opportunities to make improvements to the page layout. So here's an example where we revised the placement of the quick actions call outs that used to be on the right sidebar and now you can see they're in the middle of these how-to pages. Here you see the evolution of the how-to pages parent page. We used to call these subtopic pages. They were basically just topical groupings of individual transactional services that the state provides online. Now we're reinventing these as something more usable and more meaningful for our users, which we call service pages. And here you see related updates to the parent of the service page which we call topic pages. So hopefully now these are gonna be better at giving constituents the information they need to zero in on the actual service they're looking for and get there more quickly. Another focus for us has been the search to get people to their destination faster without confusion. 70% of our users start at Google and then go straight to an inside page. They skip the homepage. To support a better search experience, we're following specifications published by Google, Microsoft, and Yahoo at a website called schema.org. This enables search engines like Google and also other sites like Twitter and Facebook to reuse content from our website in their applications. And then here in the left in Google's knowledge graph, we were actually able to use a tool that Google provides called Google My Business to manually enter some of this information and get some quick wins to provide some of these experiences before the schema.org compliant content is even online. But once that content is online, our expectation is that more and more things like this are gonna happen automatically. But to do this, we need to make the leap from unstructured data to structured data. In our legacy CMS, everything is a page and all the meaningful content is mushed together into one big body field. And it's sort of mind bending for our authors to get out of this mindset of creating pages and think about creating chunks of content. And making this leap and doing it successfully is critical for search. It's critical for responsive design. It is important for changing something once that we wanna appear in many different places. And it was a breakthrough for us to realize that addressing this shift in mindset explicitly with our authors is actually a really critical thing for achieving this in our trainings. And so actually to that end, here's a 45 second video that's hot off the presses that we're incorporating into our author trainings beginning next week. If I can get the play. So we learned yesterday at the Gov Summit that sharing materials like this is something that we can do right away to start collaborating with other government agencies that have similar needs. So it's pretty cool to be able to bring that and be able to start sharing it. In terms of data and analytics, you can't manage what you can't measure. Online marketers talk about conversions and to close the gap that we see between user satisfaction in public and private sector websites like in the McKinsey report, we need to start thinking this way too. Originally the word conversion described the point at which a visitor converts to becoming a paying customer. Now we think of conversion basically being any event that we can monitor in a browser that indicates that a web application is successful at doing whatever it's supposed to do. So a form gets submitted, a link gets clicked, material gets downloaded. Sometimes the service that we're delivering is information and so in that case, the page gets read. So we are reimagining everything in our ecosystem as a path to a funnel to a conversion. And all of our content is about getting constituents to some destination, some endpoint where a service gets delivered and a conversion gets tracked. We use Google Tag Manager and Google Analytics to capture web statistics. Our data scientists have built some dashboards to help us understand the data and it's too early to understand right now with the limited pilot data that we have. What insights we will glean, but what we expect is that when we've got more data this will show us which services are performing the best and which ones aren't. Which ones are the most effective at driving users to forms, applications, et cetera and which ones leave users feeling lost and confused or bounced off to call centers. Data is also emerging as a critical tool in our change management toolbox. So much of what we do today is driven by someone's gut feeling or by inertia. It's just the way we've always done things. And so this has given us empowering tools to help all of the people interacting with content on our website challenge our own assumptions and make decisions that are better for our constituents. This is Photoshopped, but it's coming and this is our next step to embed analytics right into Drupal so that our authors can see service by service how their content performs and that can help them make better decisions about how they can improve their content. So we still have a lot to learn. We're definitely getting better at soliciting input and feedback, but if we get this right hopefully soon the Commonwealth will be addicted to this kind of data and user feedback. So if you're wondering what about the other 80% in our 80-20 pyramid, we would love to talk to you about it. Now that we have made substantive progress and we feel like we know where we need to get to and the work is in progress at the top of the pyramid we're letting ourselves think about the rest of the ecosystem. So we've talked about item number one here, G2C and G2B services. If you wanna hear about two through seven please come to one of our BOFs later in the week. We'll have a slide that tells you information about that in just a second. But before that, I wanna turn it over to Moshe Weitzman to talk about OpenMass. So you guys heard about all the amazing journey mapping that was done to figure out what's important to the users and found out about the rigorous design work that's been done and lots of good software development, all available and visible on pilot.mass.gov. And it's available to all of you, okay? We have made this software open source. So you can have a version of pilot.mass.gov running on your own laptops, show it off to everyone you're working with, see how it works, okay? Yesterday at the Gov Summit, we made this all public and open source. So let me just show you, then it's there. Yeah. I think I'm there. There we go. Okay, so we're looking at our public GitHub repo. You'll see at the upper left, you can find that at massgov slash openmass, okay? Feel free to take a look there. I've navigated on the screen to the releases page. This is where you get your own copy of the product, okay? We're up to alpha two. And so what you'll do is click on that alpha two link. It will download to your laptop and then it says follow the read me for install instructions. The read me is super simple. There's nine steps. Follow those. If you get stuck, just contact me. I really want people to try this out and give us feedback. So feel free to ask me for support. And for anyone who wants to join in and help out, there are public issues there. You can write issues, which we'll respond to. You can make pull requests for the software there. You'll find quite a modern composer-based project there based on Drupal 8, okay? Okay, so just some links for you guys in closing. Pilot.mass.gov is live now. That's where all this experimentation is open to the world. Please take a look. We have some information on Medium. We have a form for you to collaborate with us. Okay, so that's that bit.ly link. Please fill that out and we'll get back to you. And then the OpenMass slash releases that I just showed. All right, those are the key links. Okay, there are three buffs coming up. We'd really love people to join us in working sessions. We're gonna go over the OpenMass roadmap tomorrow. We're going to do a workshop where we help people install, take a look at the pattern lab design system that we've built called MadeFlower. And also SuperSet and other tools like that that we've mentioned. And finally, we're gonna be sprinting Friday morning if anyone wants to join. Okay, we'd love your feedback on this session. Let us know how we did. And I think we're open for questions just for a few minutes. We have five minutes. Okay, five minutes of questions then we'll turn it over to the next presentation. I see a hand here. If you could come up to the microphone, that's ideal. I know it's a little bit of a pain, but. That'd be great. I was looking at pilot.gov if you could put the home page back up again. This is here. I'll just go here. I noticed you have your popular searches there. Talk a little bit about how that content is being chosen and how it's being put up there, if you would please. Sure. Today, that's being curated manually. So we're keeping an eye on Google Analytics and we're linking to popular items. Yeah. Someone's gotta have a tougher question than that. What's behind the searches? Today, this is powered by Google and there's a module that we use that got us pretty far, pretty fast in improving the internal site search results but it's being discontinued in the next 18 months. So at some point it'll be something else but for a while it's Google. Sure, so we're providing a system that will prompt them to enter structured content. They can't create pages anymore and that's because we live in a world that's more complicated than pages. The page metaphor is sort of outdated. It made sense or it made some amount of sense when we were designing for one screen but now we're designing experiences for so many different devices and interactions and so when you think about responsive design and making one piece of content work at many different viewports that's one reason that you want the content to be more structured but then when you think about syndicating that content to many different places whether it's Google, Twitter, Facebook, and Bing all showing different mixed and matched chunks or it's many different web properties across the broader Mass.gov ecosystem we want addresses and phone numbers and hours to update. You have to break those bits of content into their own chunks. Having it all in one big body field literally makes it technically impossible to do that sort of thing. So when you go to mass.gov there's not an obvious link that I saw to get to the pilot site so how are you directing users to come here so you can get real constituent testing? Up to now we've been promoting the site to targeted audiences at places like this to generate a lot of our testing feedback and as soon as the governor announces the site we will start broadly disseminating links to the promotional video that some people saw before the presentation started and hopefully put links on mass.gov in a more visible place to get a broader group of testers. Thank you. Thank you. The question was what was the biggest barrier to doing user centered testing in the big government bureaucracy? Procurement was a huge barrier for a while just getting the tool. So first getting the buy-in is a barrier for some people. We have a leadership that are really interested in this kind of testing and development but then just getting our hands on the right tools and then getting in front of the right users, getting in front of a representative group of testers is not trivial and so validately getting to actually procure that and then let them recruit the right group was a real breakthrough for us. All right, we have to cut it off there because the next group needs to get on. Thank you so much. All right, thanks everyone for coming. We'll see you at the Bobson Sprints.