 Hey, okay. I got a whole lot to talk about. I'm Brian. I guess briefly before the news, I was an engineer, did that for a long time, and then I went to journalism school and became a news nerd, became actually an editor, which was kind of weird. So for a long time I ran the news applications team at Chicago Tribune doing data and politics reporting and investigations, and then ran it, started a team called NPR Visuals, where we did all sorts of fancy visual storytelling projects. These days, I'm the vice president of Productive People. Like you said, it's Spirited Media. We're a 25-ish person news organization with three websites in Pittsburgh, Philadelphia, and Denver. We are a for-profit company, and we are trying hard to be a profitable company. Because local news is a difficult business to be in. We're small, we're online only, so we don't have a lot of overhead costs, like, let's say, a newspaper. But that also means we don't have the advertising revenue that a newspaper or a TV station can generate, and we've got to find our money elsewhere. So we do events, and we have advertisements on our website, and we do some consulting here and there. And those are all fine, but they are still not enough to pay for the work we do. And it is important the work we do. I mean, we may be for-profit, but we are very much a mission-driven organization. We do this work because we want our cities to be better places, and we want to create and inform the public. We are here to enable democracy. Journalism is not merely a job, it is a calling. It is a system of beliefs. And that's why, in my humble opinion, journalism is best off thinking of itself like a church. It's for-profit, not for-profit news. It doesn't really matter. Ideally, a church and its congregation are working together to make good in the world. They share the church and its congregation, share a system of beliefs, and the church is supported by its congregation. This particular church is a weed church, which is something I didn't know existed until I met our team in Denver. Apparently, point all this, so six months ago, we launched membership programs in our three cities to stretch the metaphor out. We're the church and our members of the congregation, and we work together to make good. And so our membership pitch is pretty simple. People pay us because they want us to exist. They want us to continue doing the good work we do. And when you give us money, you don't get a tote bag, you don't get discounts at local bars and restaurants, or any really substantial benefits. The benefit you get is that we continue to exist, and we will work with you to make our cities better places. So, for reference in the context of WordCamp, if you saw Eric's talk yesterday on his spectrum of how you make money, we are on the contribution end of that spectrum. I guess it was on this side of the slide. You're not paying us for access, you're paying us because you want to help us do our jobs. Or to use Austin's framework from this morning in his quadrant diagram. We are a high equity, low scale operation. Although we operate a slightly higher scale than a lot of other local sites because we have three local newsrooms. We've got the same engineering and design team supporting three newsrooms, so we have small economies of scale. Anyway, so to make all this happen, to get people to give you money, it is not quite as simple as merely putting a donate button on your website. You can't just put a button there and expect people to show up. You have to persuade them. You have to introduce the idea and you have to make the case. Like Austin touched on this morning, your user experience must reinforce your business model. And so to get people along this, it's often a multi-stage process. Marketing people might call this a funnel or a ladder or whatever. Our primary path to membership is pretty simple. People read our website and they decide they like us enough to read our morning newsletter. And once you're getting the newsletter, we ask you for money. It's mostly that simple, but it's not quite that simple because if someone is going to be a member, they don't just have to gain value from your service. They have to love what you do, right? And the love happens, in my opinion, when readers identify with your point of view, right? When you and your readers agree on what's important, right? And I know point of view is sort of treading close to a third rail in journalism, but I'm not talking about politics. I'm not talking about being partisan. We're not endorsing candidates, right? But our point of view is expressed in how we think about the changes facing our city and we decide what to report on and what to not report on. I guess I'll say this on live video to the internet. Journalism is not objective, right? Journalism is subjective. We choose which stories to write and which stories not to write. We will then report them objectively and fairly, right? But there is a point of view involved in how we do our work. And that actually sounds a lot like good old-fashioned news judgment for the people who've worked in newspapers. But the thing is that having good news judgment, reporting good work, writing good stories isn't enough. The work we cannot, this is something we all, the news frequently fails to do, is we expect the work to speak for itself, right? And really, if we want our audience to love us, we have to tell them where we're coming from in lots of other ways than just, here's a story I wrote, give me money. We have to express our judgment. We have to express our point of view on the world. Not just our stories, but in our user experience. So without further ado, let's talk about why we ditched AMP and other UX choices we made for launching membership. We're talking too fast? All right. All right. So one thing we had to do is we had to build the donation form, because you've got to ask people for money. And on our donation form, our point of view in donation form and our membership messaging, the emails we send you, we're pretty explicit about our point of view. We love this stuff. Do you love it also? Please give us money, right? But most of the user experience changes we made to our rather plain Jane normal news websites that we had before are much more subtle than just the explicit ask, right? The implicit messaging that you have on your site is what sets the mood, right? It's about showing that we care about the same things that you do in lots of tiny details like our tiny little tagline, which I adore, Denver writes the Denver site. Because you got to pay attention to the details because we can't keep slapping more shit on the page, right? Our users are mostly using phones at this point to experience our work. And it is my opinion that every word and every tiny detail is an opportunity to express our point of view. And so we must consider every single pixel, especially on smaller screens. It is our task. It is our challenge to always be intentional about our design. So let's talk about navigation. Navigation on a website, it serves two purposes. The obvious purpose, the reason we all think why should I build a navigation system, is to help people find shits that they are looking for, right, to help them find stories, right? Most people, as far as I know, people don't use your navigation system a whole lot. I've never seen strong numbers on them, right? And so, forget that idea. I mean, that's sure, that's useful for a handful of geeks, right, to find a specific story and a specific topic. The not so obvious purpose of your navigation system is it is an implicit statement of your priorities, right? This is a list of things we care about, right? And so what does, I'm picking, I'm picking on the Chicago Tribune a little bit today. One, because I used to work there, and two, because we're local. Sorry, Toronto guys. Love you. So what does the Chicago Tribune care about when I visit their website? Well, they care about the e-newspaper, and they care about death notices, classifies. I used to work in that newsroom. I know those reporters. And I am damn sure that no one in the newsroom cares about most of those things until we get sort of to the bottom of the list, right? So this navigation, the Chicago Tribune's navigation, and I'm just picking on them, this is a endemic issue. This is a phenomenon that is not unique to the news, but the news seems to always do it when your homepage or your navigation, your sort of structure of your site frequently, does not reflect user needs, right? It's when it instead reflects your internal structure, right? Or worse, who has the most power, right? Bob said I really had to get that thing on the nav, and Bob's important, so I'm going to make sure that gets on the nav. I think a lot of us have received those requests in our careers. And when that happens, I am told by my designer friends, that it's called showing your corporate underpants. That your underpants are showing when your UX is about you and not about your readers, right? On the right here, we've got Denverite, one of our sites. We are keeping it pretty simple. Denverite reports on lots of things, but we are not putting a link to every category of information on our website. Our navigation is here to express primarily what we care about, what our point of view is. And even within this pretty short list, I think we make some subtle but important distinctions, like homes and cranes is pretty different than saying real estate or construction, right? I mean, we're looking at this kind of news from the point of view of the development of our changing city. People ask what are those cranes about, right? So we're not writing stuff for commercial readers, right? We're writing from a human scale perspective. And so this kind of philosophy isn't just limited to the site structure. It's in our headlines, and it's in lots of copy around the site. I sort of cherry picked this one from Andy a couple of months ago. Look at this headline in subhead. Andy really cares about libraries, right? Libraries are great. Everything is great, right? This is not just telling you about the story. It's telling you about Andy's point of view. Andy loves libraries. That is Denverite's point of view. They are dorks, right? Similarly, at the end of the story, another detail I'm especially proud of is how we do our newsletter sign up forms. To the best of our knowledge, there is one place that is the most successful. I'm certain many people in this room know this. The most successful place to ask someone to take action when they're reading a story is right at the end of the story, right? And that's why everyone does tabula links or promoted stuff. That's why that crap is sitting right there. This is the opportunity to ask someone to do something new. And so that's where we put our newsletter sign up form. But we don't just say, like a lot of sites, a lot of sites literally have a box that says get our newsletter. And a box and a button. And the button says submit. And the word submit has got to be my least favorite web user experience word of all time. That is demanding a lot of your audience to submit to you. We say, hey, you're like us. Do you like our point of view? Would you like to get more stuff like this that we do? You're going to love our emails. And the newsletter prompt is also an example of us trying our damnedest to make a membership experience that doesn't feel like a bolt on. That doesn't feel external. Like pop over prompts like this have taken over the goddamn web. And as far as I can tell, it's because everyone's just too goddamn lazy to properly remodel their user experience. These subscriptions, donations, anything you're asking your users to do, they should feel native of your site. Primarily because they shouldn't piss me off when I'm trying to read. We want people to love us and nobody's going to love you when you keep punching them in the face with pop over windows and all the other crap. Now the pop up vendors, there might be a couple in the hall. They will tell you that these things are very effective, especially the ones that like pop up as you're trying to leave. You're trying to move your mouse out of the corner. They are. But you know what else is very effective? Putting a box at the end of an article. A non disruptive, very effective tool. So how am I doing? Let's look at an email. This is Denver rates morning newsletter. It begins with a great original photo and starts with an introduction from a person signed by Ashley. And then it flows into stuff you must know. You must, must know the weather. And then follows with sort of interesting and fun things. This newsletter unlike a lot of, I mean like, you'll all see newsletters like this. You've all seen newsletters that are literally just a list of headlines. The reason we are, the people who are writing newsletters like we are, that are clearly authored by a person. When we're authored by a person, we're reminding people that we live here too and we're people that need your support. And we organize this newsletter in a way that reflects our point of view. It's pretty simple. It's just sort of an ordered list of stories with sections that are predictable. But that hierarchy really matters. The newsletter, so when we're putting things in order, grouping them and labeling them, that's pretty similar to how you might arrange the front page of a newspaper. You expect the weather to be on this page. You expect the sort of predictability to it. And there's importance. Front page is important, back page is less important. This is pretty similar to listening to MPR in the morning or the front page of a newspaper, this hierarchy. But for some reason, the news perennially fails to present the same kind of hierarchy on the web. Like we suck at having good hierarchy on the web. We might put our top story in the top slot of the homepage for a couple hours or start sending out breaking news alerts when something super important happened. But our websites, specifically our home pages, are rarely really good at making sense of the day. The sense of an entire day. And I'm not going to waste too much more time picking on the Tribune. But look at this very typical newsletter homepage. We have sort of a top story, an elephant in pants. I scroll down, I get to a grid of multimedia articles, which I don't know about you. I don't actually come to a website looking for a format. And these aren't even formatted similarly. They're not all like videos even. They're all quite different. And we've got another grid of stories. They're just sort of a mix of things. This homepage and lots of other ones, not just Chicago Tribunes, are a wall of information. Apart from the top stories block, that thing at the very top, there isn't a real hierarchy. There isn't a real expression of this is what we think is important today. And again, I don't understand their point of view. I don't get what they care about. I mean, if you were like an alien looking at this website, trying to understand what Chicago Tribune thinks is most important, the most prominent element on the page is either pants or the St. Louis Cardinals guy who's been over the top of all the articles as I've been scrolling down. Chicago does not like the St. Louis Cardinals, by the way. But to be fair, I mean, managing a page like this is hard. It's hard work. I have dear friends who are web producers around home pages for news sites. It is a difficult thing to do. I don't think anyone's really cracked the nut of having a great home page that expresses priority as well. But that to me is what highlights the value of having an email newsletter. This is a completely self-contained news product. It is better than any of our home pages. This is much more like getting the newspaper or listening to NPR in the morning. It is an opportunity to express our take on the day, our point of view in a clear, concise, hierarchical fashion that is really difficult to do on other pages, especially the home page. Because you're always just chasing the most recent news and your Facebook feed can't really express priority because it's just a feed. So when readers find our website and read our stories and want more, when they want a whole package of our point of view, which is what makes love, our point of view every day, that's where the newsletter delivers. So in that way, I think a lot of us have this upside down thinking that the newsletter is sort of an accessory to the website. It is not that. Our website is an accessory to the newsletter. The newsletter is a better product. My team sitting over here, they will tell you, we have spent very little time fiddling with our home page in the last year. It's just not as important as other elements. I'd love to fiddle with the home pages, but we're a small team. We've got a lot of shit to do. So by being really great at expressing our point of view every morning, that's why these newsletters are such a powerful tool for creating committed loving readers. So we've got tens of thousands of newsletter subscribers at this point across our properties. And like I've said, going too fast. So the subscribers, I think I've mentioned, are the people that we're chasing after to become members. Those are the people we want to give us money. I forget how many members now. Of our members, I don't believe there are any, or maybe I could count on one hand, people who are members of our sites that aren't newsletter readers. It's very few. But when one of those tens of thousands of people is visiting a website and they've got scrolled to the end of the story, right now what they see is this. Subscribe to our newsletter. That's kind of a waste. Right. So this is our, I'm going way too fast. I got to slow down. All right. I'll explain this in detail. So this is a feature that we still have sort of in the works for kind of halfway done, but I'm geeked about it and I wanted to, I thought it'd be fun to chat about it. So if we know you're a subscriber, we should be prompting you to become a member. And if we know you're a member, we should be inviting you to that member's only event, right? Or we should be asking you for more money, right? $10 a month. That's super. Make it 20, right? And, but how do we do that? Like everyone's, I'm hoping that everyone's engineer gears are turning in their heads. How would you build a website that would do that? And the sort of, at least to me, the obvious solution is you build a login system, right? But nobody wants to log into your damn website. Ever. Ever. Also, like it's weird because we're on lots of different browsers and you're logged in over here and you're not logged in over there and you're certainly not logged in when you're in the Facebook web view and all this shit, right? Like it's, I mean you have to give someone a damn good reason to log into your website and I don't think we have a good reason enough. So that's out. We're not going to ask you to log in. So it turns out there's a pretty simple solution for us. Is that when, so we have newsletter readers and when they click a link in one of our newsletters, we know they came from the newsletter. So we can write it down. We can write down a cookie, right? And so we're just using MailChimp, right? So MailChimp in the query strings, passing over Unique identifier. I can use that Unique identifier to off cycle, not on the load of the web page but off a little JavaScript somewhere to look them up and see if they're a member. And if they're a member, write it down in the cookie. And with that sort of simple amount of information, we can tailor the user experience to your status as a subscriber, to your status as a member, to your status as how much money you've given us before because we're synchronizing all of our sort of membership database. What we know about people like their one year anniversary or, you know, that's all being synced into the mail system. So we can look all that stuff with a simple, look up all that stuff with a simple API called the MailChimp, which I'm really excited about. So to be sure though, this is not perfect. I know every engineer in the room is also coming up with all the problems with how the dots aren't always going to connect. We're not going to show you the right prompt when you're visiting from Twitter if you're using the Twitter and Ant Browser. But that's okay. That's fine. I don't need to always present the perfect thing. I need to cover the big use cases. And the big use cases, a newsletter, someone who subscribes to our newsletter, reading it on their phone, clicking a link, and looking at it in their browser. And the same thing some on the desktop. Like if I can cover 80% of the scenarios, that's fine. And so I know I'm talking all fancy, like this is a fancy thing we invented. This is where I'm not geniuses. Customizing the user experience of your website based on the user's context is not a new idea on the web. Amazon does it all the time. I'm certain there are a couple of other news publishers who do stuff like this, but there really are very few. And if they are, they're not doing it well. They're doing crap like this. There are vendors who will sell you products that will figure out where your user is at in your funnel and try to tailor the experience. But all this shit is still, we're back to having these bolt-on solutions instead of having something that feels truly native to your website. And really, we're also like, we're looking at a query string and writing a cookie. This isn't all that hard. I don't want to pay someone to do this. I would like to own this, perhaps open-source this, you know. And that's, it's silly to pay someone for this sort of thing. So anyway, to make this stuff fit really nicely into our user experience and feel natural, we get to our first big reason that we ditched AMP. So we ditched AMP because we need more control over our user experience than AMP permits, right? To build an unshitty, persuasive, native feeling, highly effective membership experience. I need a little more JavaScript, right? I need a little bit more control over the page. And to be fair, we could probably make some of this stuff that's tailored for AMP. I bet, I bet there's, maybe there's a framework that's coming six months from now or another vendor, right? But the thing is, I've got, we've got two developers they're over here, they're very nice. Good talk to them. And we have a whole lot of work to do, right? We cannot afford to design, build, and most crucially test multiple versions of our website, right? It is hard enough dealing with the spectrum of browsers who visit our normal websites, right? Testing a whole second code base or a whole second user experience is far too much to ask, to be honest, even if your team's not as small as ours. Now, AMP is, you know, we talk about AMP and AMP being a thing that makes user experience better. But AMP is really only addressing, as far as I know, one facet of UX. AMP is about speed, which Barb talked about earlier, right? And speed is super important. All of Barb's charts about how speed is important and how having fast websites makes people more likely to give you money. We're all into it. But we're not a big newspaper on some shitty corporate CMS that's doing all these creepy, tracky calls and loading up your page. We are running a pretty lightweight WordPress setup that's properly cached and quick as shit, right? And AMP page might load a little faster, but our other needs far outweigh that minor performance improvement. Finally, we got over our fear of missing page views, our FOMP. I made that up. I hope it catches on. Hashtag FOMP. I mean, everyone on our team, myself included, we were all concerned that if we left AMP, we would start losing traffic, right? I mean, that was it. I go to my boss, who's a very reasonable person. I say, boss, I want to leave AMP. And he's like, oh, I don't know. That for the page views, you know, anyway. And it's not all about the carousel, but the carousel's a thing. We got over this fear when we truly, I think, understood the value of having more control of our user experience. So we realized that we needed to tailor our UX in ways that would be difficult or just a pain in the ass to do with AMP. We said, you know what? Okay. I think we can prove this with a couple of spreadsheets and a couple of back of the envelope calculations that if I can be better at making conversions, I'll take the trade-off and lose a little traffic, right? Or as Austin said in his talk this morning, we realized, another way to think of FOMP, is we realized we shouldn't be playing the attention game. That's the way Austin framed it, right? We can make more members even if we have fewer visitors, right? So I know the question you're all dying to ask, and I don't have a chart for this, but how did it go? So we left AMP a few months ago. Did our website traffic go down? Well, there's no actual way of knowing if we lost page views, right? That's frustrating, but I do know that across all three of our sites, mobile traffic from Google has increased since we ditched AMP. It might have increased more if we were still on it, but, and I'm certainly not saying that leaving AMP is a guaranteed way to increase your traffic, but I can state to you confidently that leaving AMP didn't completely fuck us, right? And I want to make this point because this is a fear we all have when we think about leaving the platforms, right? We got through it. But this fear, this fear, this FOMP, right? I fucking resent this fear. We are trying to build a new business model for local journalism. We are trying to save journalism, right? In the face of declining ad revenue, massive cuts in newspapers that are owned by stupid freaking private equity, and a political climate in which I'm being called the enemy of the state, right? I have fear from all sides, right? And so this extra bonus fear, yeah, I resent it. And I mean, the morons of the big publishers, big news organizations with their massive, super slow, super creepy, super tracking news websites, yeah, those morons need to blow up their shit and respect their goddamn users and make things fast and friendly. But that... Yeah, but they're... Well, yeah, fix your shit. I'm getting to it. That problem is yours. That problem is all y'all's just the people who are here. That problem is yours to solve and not Google's, right? By policing slow websites, Google has turned the big publishers problem into my problem, right? Because now I have to chase the carrot. Now I have the FOMP, right? It's because some clever engineers, and I'm certain they're very lovely people, well-intentioned, they realize that big websites are crappy and they set out to fix it, to me, a small publisher, to get that sweet spot in search. Now I've got to maintain two code bases. Now I've got to test two versions of my website. That's absurd. We're a very small team and we already have a very fast website. Like, just because Tron and Gannett, and you name it, made a bunch of bad choices and Google made a my problem, this is not my problem. So, when the call for speakers is posted, not taking back the open web, right? I do not believe that AMP is the open web. AMP is a short-term fix for bad choices. If you build a slow and terrible website that supports a creepy business model like tracking the shit out of your users, then yes, AMP is one path to a better user experience, right? It is an escape patch from your legacy code and bad choices of your predecessors, right? And I understand why that would be tempting. But it is also a deal with the devil. AMP and its Cousins did an article and put even more control in the hands of companies that already have way too much control and I for one cannot abide by that. So, I just ask you, do the work. Fix your websites. Make them fast, friendly and respectful. Leaving the platforms is scary, but you'll be fine and you'll have more control of your user experience and your business model. Thanks. I left too much time for questions. Questions and we will come to you with the mics. I'm just going to sneak out the back. A smoke bomb question. Cool. In the back. Hey, Brian, thanks for that talk. That was really inspiring. My question is really about your newsletter experience and how you customize that and maybe this is a conversation that we can have when we're both out taking a nicotine break. Yeah. But I'm just curious like how much time you guys spend in crafting that custom experience every day, that tailored messaging for your newsletter. It's a really interesting and cool concept, but it seems like it's... No, the actual newsletter, like the personal message. Oh, yeah. Yeah, the signed like that. Yeah. Tell me about that process, please. So, this is someone's job, right? And that's, I think that's the, you know, the failure of a lot of news organizations' newsletters is they want it to be something they get for free based on an RSS feed database, right? Instead of making it someone's job. And so there's actually a lot of work that goes into this every day. The teams, they have all of our newsrooms have a dedicated Slack channel to talk about the newsletter, the all-pitch stories. One person is assigned to be the editor for the day. That person is gathering them together, kind of assembling the whole thing. I mean, the, you know, and I think that's a lot of editors, when we talk to other newsrooms, you spend managing your home pitch, right? There's someone sitting at that thing all day, pointing and clicking and moving things around and fidgeting, right? That's someone's job, right? Which is higher value? I would argue this is higher value. No, if that's any web producers in the audience. You have a hard job. I appreciate you. But, yeah. No, I mean, it's, it's a lot of work. I mean, do it seven days a week. Does that answer the question, sort of? All right. Yeah. I mean, it's not, it's not a full day's work for someone, usually, but it is, it is a dedicated, in Denver, it rotates between a few of the reporters. It's usually in Philadelphia right now. It's our editor who's actually doing that work and she does it. She starts it at the end of the day and then basically it's mostly written by the time she goes home and then she wakes up in the morning, looks at Twitter, makes sure that like city hall didn't explode and hits the send button. Over here. So we're a, we're a customer of one big vendor that I'm happy to pay money to. They're called the News Revenue Hub. They are powering a number of news organizations, membership, membership efforts, including The Intercept, Honolulu, Civil Beats, anybody else got anywhere to shout out? There's dozens of them at this point, a couple dozen. And what their service provides is not just some technology, because technology, blue isn't that hard to cobble together, but their services is, they've helped us learn how to write drip campaign emails, sort of the emails you get automatically. So when someone signs up for the newsletter, they get a series of automated emails that introduces us to them and then eventually in three weeks, the email you get is about how we make money and then hopefully you donate when you receive that third email. So we have, there's a good deal more sort of infrastructure behind this that we've set up. A lot of the email automation is all just happening in MailChimp. It's not hard to set up. The actual sort of, the way News Revenue Hub is set it up are member databases in Salesforce, our payment processing is coming through Stripe, and all of that is synchronized to MailChimp. So we can send, like one of the more powerful things we can do, if you've got your membership information in MailChimp, is send very targeted messaging to people. So it's one way to do it. If I was starting from scratch today, I'd be looking at, was it memberful, the folks who just got bought by Patreon yesterday? You know, I think there's probably homegrown solutions that might be fine. I mean, Salesforce ain't great. But yeah, it's getting the job done. And I endorse the News Revenue Hub work and their help, so they're friendly people. They're expensive, but they're friendly. So going back to the topic of OpenWeb, how much of the tech work that you guys are doing, are you documenting, blogging about, contributing back to open source software? Where can we find that? Where can we find it, Russell? We have not been blogging enough. On my prior jobs, we blogged all the god damn time. We have not been blogging enough. We've got a couple packages. We're going to release. It was one of your open sourcing this weekend, right, Russell? Five o'clock. Oh, I'm sorry. Wait for it. But yeah, I'd say not quite enough. We do contribute to a handful of packages that we use. I saw, you know, you said that, you didn't see much of an impact on page views. But I don't think it's as much like FOP as it is FAPOG, fear of pissing off Google. So, even though your page views stayed steady or you saw a slight increase, with the more control over your UX and a more intentional and targeted kind of membership experience for your readers, is it that you saw that returning visitors, an increase in returning visitors and the number of pages that they would visit, or like, and did you see a change in traffic from Google overall? Like, from Google and an increase in return visitors page per visit, is that right? Right. I mean, the early returns on our leaving AMP was having a newsletter sign-up box that was just more effective. We saw it be more effective the day we left. I'd say an easier example of this, because we're all working for our mind's eye here, we're imagining. You all have instant articles? You all get instant articles on your sites? Instant articles has a box you can attach. What do they call it? They call it a call-to-action box that you can check a box and it shows up on the bottom of your article and it asks people for their email address to subscribe to your newsletter. So, we've had that set up and running for some time. We've got it integrated with Zapier to automatically dump stuff into MailChimp. So, I can look at, I can see pretty clearly how many people are using that a day and compare that to the number of instant article views we have. That box performs 50% as well as the box we have on our website. Right? And we were basically seeing the sort of... Yeah. I was trying to use an example you can imagine because they all look the same on everyone's websites. The stupid little box is bad messaging. So, that's why we're leaving IA. We haven't done it yet, but it's going to happen. Our... The success of AMP, we actually... I mean, one of the problems that we had was we... The pages were simple. They barely worked. I mean, I think this is a weird argument because I think if we had spent a lot of time trying to make our AMP user experience as good as our native user experience, things might have been about the same. But we wouldn't have been able to implement the things that we're implementing today. So... But we didn't have time to make our AMP user experience better because that's a lot more work. Thank you. Yeah. That was a difficult answer. All right. Time for a couple more questions. So, when did you start your UX efforts and then what kind of effect did you see in your membership and donations, potentially? Well, I mean, all of this work, this work that we've been doing over the past year or so, and we launched our membership six months ago. So, we were sort of getting ready for... I mean, there's a lot of other stuff we haven't talked about. There's a lot of work we did like making our website way faster. Doing... Dealing with images and cropping things down and caching things better. Really, kind of behind-the-scenes choices. And now, the work we're doing going forward is things like this, is fine-tuning the user experience, asking people. And I will be excited to report the results of this work a year from now because we don't have it yet. But you can imagine an analogous version of this but in the newsletter, for example. When people are getting the newsletter, we can also fine-tune the messaging there. We can ask people for precisely the right thing. And I'm really geeked to learn about how that's going to go. Have you been able to see any increase in engagement so far? Yeah. I mean, one of the... This is one of those things that... In a lot of ways, it's hard to manage. It's hard to draw charts of this stuff because we are small websites with sort of moderate traffic. So, the ideal situation is, I take this newsletter prompt and I compare it to my old newsletter prompt and A-B tested using Google Optimize or something else. And Google will tell me which did better. The problem is to get like a statistically significant A-B test out of that with the amount of traffic we have. It doesn't really work. I was really going out when I started joining this company at doing a ton of multivariate testing. But multivariate testing was not effective at... It would take literally months to be effective to learn if one choice is better than the other for something that doesn't happen that often. Like a newsletter sign-up is not something that someone does every page view. So, I can say that as we have made... This thing was... These things were much more dull before. And I can tell you what I can look at is I can draw a point on the timeline and see how many conversions per page view if that went up, how that's increased over time. And I would say that I could say pretty confidently that having a spot hand over your newsletter prompt is way more effective than having a box that says submit. If you just look at a raw number like the number of subscribers per visit over time. Got it. I think you talk about the native experience, something that I don't think about often. What do you use as your strategy? Like how do you tailor your contents for both the native experience? Do you envision yourself as the user? Like if I were reading this, this is what I would want to get out of it? So what's our sort of user experience methodology? Yeah. Well, I am not the lead designer. My teammate, Livia, was the lead designer. And I'd say her methodology... I mean, this is... Again, we're a small company. There's a mix of knowledge and prior experience and as much user testing as we can get our hands on. If we were a larger crew with a bigger budget, I'd be spending money on doing observational user testing. I would just watch a dozen users use our stuff and see what they do. But it's mostly sort of guerrilla user experience testing. You know, it's grabbed me. I think Joe Sposey called it a hallway usability testing where you just grab the person who's walking on the hallway past and you pull them to your computer and say, hey, use this for a minute for me. Watch what they do. You know, we are doing this on the cheap. Awesome. Thank you. Yeah. I might have time for one more quick one. I must have got a yes or no question. I'm a little bit curious what could actually bring you back to AMP or to actually Facebook Instant. What are the specific pain points that you found? It sounds like a lot about customization and delivering more contextual messaging. What are some of the other pain points and where would there be a point where you would actually reconsider AMP? Yeah. The pain point that sort of drove our exodus was what you just said. I don't like using the word personalization because personalization is like a fancy weird web myth thing. But the sort of contextual messaging, being able to know a little bit more of the user and tweak the user experience to what we know about them. But also, I mean, that's what kind of pushed me over the cliff. But I'm still stuck with the we're really going to test all these things? I mean, I think this is something, as a recovering engineer, I know that we all underestimate how much time it takes to actually test the damn things we build. And we find problems much later frequently. I'll be honest, we weren't staring at it to make sure our AMP website was always working. And we know what that meant. That meant our AMP website was never working. And you can blame us for being lazy, or you can blame us for being busy and having a whole lot of things to do because we're a very small team at a company that's just trying to keep its head above water. The proper choice would have been to have not done it in the first place instead of adopting a technology that we were not prepared to support or test. And I guess the question is, when will there be upside in that? Well, the upside might be one more lot bigger. And I can afford working in parallel tracks and testing in parallel tracks, maybe. I don't know. I'd love to have a zinger for you. All right. Well, hey, everybody, give it up one more time for Brian.