 Thank you and welcome to experiments to create audience connections. Oh, wow Got it My name is Sherry Scalco. I'm the executive director of injustice watch, which is a non-profit investigative news organization based here in Chicago focusing on Issue systemic issues of inequality where the intersection of social justice and the law and I am joined here by Kerry Porter Harry Backman and Sarah schlumbak schlumbak. Yeah, so if you guys could introduce yourselves and We've got back so hi everybody Thank you Thanks for coming. So I am a journalist and I work as a consultant for Shorenstein Center at the Center for medium politics third time and Third time's attract and we focus on many things but one of which that I've been working on is business models for news Can everybody hear me? Nice My name is Harry backland. I'm the director of operations and one of four co-founders at city bureau We're a civic journalism lab based here in Chicago so we're kind of partway between a media outlet and an education group or kind of think tank we're working to make Practices for producing local journalism that are responsive to communities that have a stake in the stories that are being reported I think it takes a second great. Okay. My name is Sarah schlumbak. I work for the Lenfest Institute It's a journalism nonprofit based in Philadelphia. I just moved back to Philly It's where I'm from and we are spinning up kind of a local news lab there for a period of one year So part of being here too is if anyone's interested in talking to me afterwards collaborating We want to run experiments for the next year on how to better kind of pair technology with local news with kind of a focus on how we Just find ways to for local news to be more valuable to folks How can we pair up stories of what you're really interested in? Find ways to have news arrive when it's most valuable interesting and relevant to you But before I came to Philly, I co-led the Guardian mobile innovation lab Up in their New York office for a couple of years and ran a lot of cool kind of mobile storytelling experiments there And then before then I was at USA today as a mobile product manager and before that I was in Philly Where I kind of started this whole journey Helping them launch their first mobile apps, which feels like it was like 80 years ago But it was like eight years ago So that's kind of my background I did go to journalism school as well But I've kind of found my way onto the product and technology side file this one under we love it When a plan comes together is I we understood like the last session talked about business models and about how Publishers essentially have to look for new ways to keep their business going essentially and Audience and the connection with their audience is going to be key And so we're just going to be discussing how to cultivate genuine relationships with your audience and experimenting to do so Before we begin though, we'd like to get a sense of the composition of the audience So how many folks here are from legacy news organizations where your digital product is attached to broadcast or print? Okay, and then how many are digital only? Okay, so almost like how did you are there any other folks you want to understand? How many of you come from business or how many of you come from editorial or something in between? So which one which one all him so mostly product them web-dev Hey, how about other folks who would identify as coming from community media in any sense? Couple okay, that's interesting. Okay. Got it got it. Okay, so we are also Highly interactive here very much organic I'm going to be roving with this microphone and if you have any questions Please jump in but we'll also be turning to you guys to to get your feedback to help guide this conversation In a way that's going to be more beneficial for you so I'm going to start out with the first question for you guys is There are Loads of best practices out there Why experiment I think I just have a quick thought it might not be the most insightful one, but um having worked in a couple of legacy news organizations and Worked with a lot of engineers and a lot of editors I'm always struck by Like the cost of change And kind of what you might think a small tweak on kind of the front end of a product You know it might be kind of making a small change but when you think about kind of all the pipes and the plumbing and the things that it takes to kind of Um Change infrastructure to support new product development. It becomes really complex So one of the reasons that I think that experimentation is essential in in most places and in particular Digital news is that you need to be testing things before you want to scale systems that support it right so if you want to do things like personalization or customization or One-on-one conversations with your readers. It's good to know a little bit more about what works Before you start building kind of scalable systems to support it in the long term Which is why kind of doing doing quicker tests and measuring them really effectively Which we can talk a little bit about later kind of knowing up front what success looks like It's kind of another great part of experimentation. It's unlocked a lot of You know good work for me and the teams that I've been on Is it's kind of one of the reasons that I think it's so so essential Yeah, I would just add to that that I think best practices are good Inherently but what will make one's editorial product really sing is its value prop which requires it to be unique So while they're probably a handful of gold standard rules It's also really valuable when you can identify the ways in which you zag when everyone else digs And you get there through experimenting Sure Yeah, I might zoom out on the question and then think a little bit about what city hero could bring to that conversation So if the question is why do we need to experiment or why should we experiment? I think innovation is a hot word. It's probably been a hot word for a long time probably before our time I Became in many different things and I think it's important to think about what we're talking about in the context of news I think lately when in journalism when we talk about innovation We're talking about a response to a crisis in the industry that like we have to respond to to survive And then within that too if if innovation is happening in response to a crisis Some people are talking about the response in terms of saving money whether it's their own companies because they're Running a media company or whether it's our own jobs because we love doing this and like getting paid to do work That we love and then some people also are talking about Responding to that crisis in order to save a critical public good. That's necessary for the democracy and Maybe even before that for any kind of collective life And then the kicker is that most of us are in both of those camps at the same time understanding that this is really important work and Also trying to find a business model that will work for it I think where city year was coming from I was really interested that the last discussion Kind of focused around the end of the advertising economy or the decline of the advertising economy and the rise of these other revenue models whether it's subscriptions or metered paywalls I Think all of those innovations are kind of centering around a relationship with readers that wasn't there when it was driven by Advertising because revenue was always triangulated. So you built a relationship with your audience in order to market that to People who were interested in Purchasing their attention As that drops away, I think a lot of what is driving the innovation that we're doing at City Bureau And that a lot of folks in this room are doing is thinking about ways to build deeper and more honest and authentic relationships with the audience and then Find ways to use that as the foundation for a Sustainable model so being able to activate those relationships to make the work possible and make it better so with a Good half of our audience being involved in legacy organizations the relationship with their audience has been through or because of Advertising so Where is a good place to start? Experimenting making it a one-on-one mutually beneficial Relationship a genuine relationship and when we say relationship We're talking the reader and the news outlet, right? Is that what we're talking about? I'm putting it back out to you when we talk about relationship. We're talking about the reader and the news outlet Yes, yeah, okay, so to take a step back where to start especially if you're coming from a legacy media organization But your newsletter work is that we can say you didn't like that question Well, I mean well, so I think this might be a good time Terry mentioned to talk about so one thing we've been doing at short scene that's been really fun is Doing a deep dive on newsletters So I've been spending a lot of time reading a lot of different newsletters any of you want ideas or suggestions talk to me after And and what we've come to find is that there's like three different ways that newsletters function in email But I more broadly like to think of them as like a living room and you as a reader get to Invite the news organization into the living room of your inbox and it's kind of a private place It's different from when you go to a web browser or even like a social media page It's something that's just yours alone And so the way that that kind of plays out in an experimentation is that you get to ask yourself What kind of what kind of company or what kind of guests do you want to be and what kind of story do you want to tell? And I think that becomes really fun when you think about your value proposition and where you see your ultimate ROI for the reader That makes sense. I guess one comment tangential comment I might add I was just responding to we didn't have a chance to talk about it right before but this idea like the Decline of the advertising industry or the advertising revenue I mean, I guess that the different take I have on it is that like advertising particularly online is is Alive and thriving and is a huge business just not for us anymore Right like just given the current state of the industry and and I think that kind of the current ad model with its You know what makes it so successful is its targeting And you could almost just kind of argue that metaphorically that Advertisers kind of know their audience enough to kind of give them the types of ads that might resonate with them a lot I know that one of my colleagues If you don't know you see like German or subscribe to his weekly newsletter called the solution set should he's really brilliant But he tweeted the other day something about how like an Instagram ad finally got him, right? And he was like shaking his fist, but it was also this really great Like sweatshirt that that he was really into so anyway And he thinks a lot about kind of connection with audiences And I think that one of one of the things that I think that we could do more of is in a more ethical way Collect information about our audience that gives us a sense of what they want to read what they want to see when they want to receive it and Kind of just create a better model for relationships through content and community engagement That that it has similar principles share similar principles with the kind of the successful ads that you see today But kind of adheres to all the changes and impact that we'd like to have in communities through through news and engagement So I don't know would you guys like to kind of challenge that notion at all? that we're trying to Inherit some of those principles or kind of adopt them, but through content I like that. I feel like if the compliment to that also is Engagement also a very hot word right now, but I think what that often means is Direct correspondence and relationships with readers. So being open to feedback from readers and making the experience of that feedback part of What an outlet is providing when it's providing news. So I think that That might be the sort of limit to collecting really rich data about what people want is if we already come with all of the categories Established there might be things that we're missing or we're also not making not letting readers or participants in media Set the terms for how they want to engage and and I think that sense of participation is Driving a lot of the new revenue models or at least is an important ingredient in something like subscription or membership Yeah, so Harry bring up something interesting because we talked about Newsletters and data which are digitally based, but city bureau is Face-to-face. Yeah what How did how did it come about and what is the like where did you start? especially when it came to creating these relationships with Established media organizations sure I'll try to keep this quick because it's a it's a good story, but it can be a long story Hold on a second. How many people are familiar with city bureau? Okay, not enough. Go ahead. Nice. Yeah, that's great Thank you Thank you for being here. We love sharing our work especially with people who are experts and asking a lot of the same questions City bureau started in 2015 It was started by four four of us We were all working in different areas of media come out of local publishing and operations and community media My colleague Bettina Chang was an editor most recently at Chicago magazine Darrell holiday was a local reporter crime beat neighborhood news and Andrea Hart was doing Youth media work working with local nonprofits in the city All of us were frustrated with some of the conditions of the Chicago media ecosystem Really inaccurate misrepresentative coverage of whole neighborhoods, especially around issues of race a real lack of pretty every kind of diversity at every level of Chicago media and deep distrust between many newsrooms and a lot of neighborhoods in Chicago and we're I think we're feeling very limited in our own work So it's we bootstrapped it in the beginning the zero dollar budget We got a group of reporters together and started a fellowship, which was really people just getting together to Report on stories that they felt like should be covered and kind of build some mutual support around that work So we get together on Wednesdays and eat pizza and file FOIA requests and start to partner with outlets to publish that work From there we the fellowship grew into a paid opportunity. So we run three cycles a year ten reporters Come together working teams around issues and we partner with outlets around the city to publish that work There's all kinds of reporters in our cycles, but we're really active about Recruiting emerging reporters of color and other people who are underrepresented in Chicago's media We also run a open workshop series called the public newsroom. So every Thursday evening we have some kind of free Hands-on skills-based workshop where a journalist or somebody working around a local issue comes in and presents And it's supposed to be something beneficial to the people who show up for this So they get some kind of new skill or information and it's also supposed to change the work that is being presented So if a journalist comes in we hope that they are open to the feedback that they get in that space And the last program, which I think is maybe most relevant to the conversation about innovation is documentaries We're the short version is we're training and paying people to cover public meetings in Chicago So there's a form on our website anybody can fill it out If you fill it out, you're invited to come to a training if you come to one training You're eligible to take paid assignments to go to government meetings in Chicago and either make a recording or take very structured notes Or live tweet it and then we collect all that information and make it available to the public to other journalists To try to create sort of a groundswell of coverage around Local journalism and also just engage a lot of folks and make it a lot easier to participate in the process And the last piece we're we're building some Basically a user management system and a set of scrapers that pull all the information about the meetings in the city and then make it Really easy to assign those out to Documenters who are interested so we're working on that and are hoping to have that come complete in the next couple of months All that said sometimes our work is talked about as being sort of way on one end of the spectrum in terms of engagement and You know what it means to be engaging with Readers and an audience and I think that's true in some ways. We're proud of that in some ways, but I Think there's a lot that we've learned that we want to make useful for news organizations that are working across the spectrum and I think No matter what kind of outlet you're at this question of which direction you're facing whether you're thinking more about building relationships with readers and talking to speaking to the kind of full person that you're you're there will be buying your product or Whether you're facing in the direction of what the ad economy had asked for from us, which was really heavy metrics and kind of a very rationalized picture of what Decisions like that we're gonna how they were gonna happen So I do feel like a lot of what we learn is can be applied anywhere And I like I think the newsletter is a great example because that's an example of legacy media organizations taking work that they already do and finding a way to Present it to people in a way that's much more intimate and allows for more feedback and I think these things are possible across the spectrum So does anybody have first of all does anybody have any questions? This is not the end of the presentation. I'm just checking in with all of you Good so far Okay, so what about how To innovate to experiment let's get into the really the nitty-gritty because and You guys are gonna have to correct me if I'm wrong here, but from a product standpoint You or from a tech standpoint You've got either insight into data or some sort of usage statistics You're you consume a lot of media you see how other folks are doing it Successfully and yet you've got all of these ideas and no Audience no when I say no audience no partners in other parts of your business to get this experiment these experiments going What are your suggestions? What have you seen in your experiences? From an operational standpoint Yeah So one of my favorite things to talk about is organizational design I just think that one thing I've learned is that it's wonderful to have ideas But then it's also the execution and then you can't really do the execution without collaboration and clarity of roles and so it's been really really Rewarding to work with different media organizations through the work at Shorenstein looking at their newsletters and specifically seeing What happens when newsletters are run by just one person and what happens when they're run? across different teams when they're shared across departments and What's neat about it is that I think you need to have buy-in across departments I wonder how many of you have kind of come across that where you have a new idea and You get only so far and then you realize you don't have buy-in within the Within the institution to keep going, but I think Getting everybody on the same page is really helpful So thinking of the newsletter is not just an editorial product, but something that Is also a business product Can be really helpful and that's why I have this thing I really like I agree about the term engagement because it's thrown around all the time But I like this idea of engagement is the middle path You know, there's like editorial and there's business and it's been church and state And I feel like the engagement folks often can kind of form this bridge For having conversations that are useful so that we really can actually Make sure that we're reaching people with the things that they want versus what we think that they want So I think that starting with that is like who should be in the room right now to have this conversation It's a good place to start Well one of the things that worked really well for us in the Guardian mobile innovation lab is One we were completely multidisciplinary right so I think that there's kind of this slow awakening to the fact that People with lots of different expertise need to come to the table like he said to kind of execute analyze Market fill something with content Give it kind of a spirit so In the lab, you know, I was the product manager and there was an editorial co-lead That was kind of transformational as well so kind of to share decision-making from a product perspective and an editorial perspective and Sasha and I both had experience on the other side, right? So I have my master's in journalism, but became a product manager She was the editor of the lab, but had had had various product roles at the New York Times So so that was a pretty important piece to us being able to effectively run a lab And and run experiments that were useful for users, but also like very editorially sound And then the rest of the team looked like, you know, we had two engineers an associate editor and A product designer, right? So that's kind of what I tend to think of is kind of table stakes for if you're going to do something innovative and digital but also Be a great a great piece of content. So What we did was We sat down we were there for about six weeks together and we would meet in the kitchen every day around lunch and just talk About all of the things that kind of came to mind as better ways to use technology as it relates to journalism And me being like a constant notetaker or product person ended up with like 15 pages of notes of like all the things we wanted to see in the world and A couple of them were things like, you know, a better use of notifications Instead of kind of the very irritating intrusive version of breaking news alerts that you see today And this was like three years ago now And and we had this really long list and then what I did was I just started putting them into categories Right like some of these things look similar and and this falls under this umbrella And we came out of that process with like five Areas of focus for the labs to give ourselves a little bit of structure And included things like notifications like how can we build better notification formats for news? Live coverage, you know, we hypothesized. It was a bit miserable right now. It's very text driven You can't get caught up very easily when you go into a live blog It's hard to scan stuff like that And so we had these five areas of focus and then we did that thing that like if all else fails in a news room just offer people snacks and We would have weekly office hours and just say like hey we're thinking a lot about technology and innovation and journalism and we love the Guardian and I know we're new here, but Please come eat like ice cream cake and popcorn with us And and let's see what we can come up with and yeah a really wonderful business reporter named Yana Came and said hey, you know, I cover the jobs report the first Friday of every month or the last Friday. Sorry, Yana and We send this push notification out and it's totally indistinguishable from everybody else's push notification about the jobs report Which if you've ever gotten that alert says like this is the percentage Unemployment rate and there's so many jobs are added So we just came up with this kind of cool idea where there's so much interesting stuff That's embedded in a jobs report in terms of like socioeconomically which you know sector of the population is being impacted like who doesn't does not have jobs and We built a little notification that had Little buttons on the bottom and it said here's the jobs report high-level numbers now Do you want the good news of the bad news first? And you could kind of pull yourself through like a little explainer of the implications of the jobs report without leaving your lock screen Which we thought was pretty cool And like a fun thing that we learned through experimentation mean like 14 people signed up for our first experiment And we said do you want the good news or the bad news first with like thumbs up thumbs down emoji? And we realized just for talking we talked to like four people and they're like Oh, I thought you wanted us you were rating the jobs report as in like good jobs report or like bad jobs report And that wasn't what we were trying to achieve So we changed it to like a you know smiley face and a brownie face and like those are kind of There is this tension between like granular insights and then kind of like global strategy, right and how you approach things and I Think that one of the things I struggle with when I'm talking to stakeholders about innovation work is that a single idea for an experiment Can can sound so small? But you're trying to get it a much bigger insight through kind of launching some things small And you are trying to get quantitative and qualitative feedback and get a full picture of how people are experiencing things And I mean unless I'm wrong like I think that's a very new discipline to be bringing to news in general And then layering on kind of that product approach They can be very Kind of granular and interesting, but and it builds and there's momentum, but it takes a while for people to see that so It's just one example Assessing Success so we talked about you early on mentioned what does success look like how do you how measure it? How do you look at it because? Harry Citiburro isn't natively Connected with metrics like especially like that we're used to so Go down the line and talk to me about how How we find our success. How do you find your success? How do you measure? So the first thing is this brings up another question, which I have if I may Which is you know as we do move away from advertising Those metrics around just like abundant content How are those going to shift and what are those new metrics? I think there's a lot of work happening around Moving away from like the so-called vanity metrics to these new ways of Pointing success One really clear way is subscribers just to go back to newsletters is like What's the rate of people who are subscribing and how often are they opening and Then also do by the way, they'd want to donate or become members. Those are our favorites Not that we have favorites ever obviously But I think you know those are some key ways that we can tell I think It's also just kind of a success when you create something an editorial product that didn't exist before While still you know using the resources you have and ideally valuing your staff in a way where they're not being over leveraged That's like a really big success From especially if you go back to that work design thing, so those are some of the things I would be thinking about So I was gonna say we do I think we're not at all opposed to metrics and we do keep a lot of metrics So what are you measuring? It's not I think we measure a lot of probably a lot of the same stuff that folks in this room are Measuring so we track our newsletter sign-ups. We keep careful track of social engagement We publish our stories with other outlets, but as much as possible We try to get metrics on who's reading and as much as we're able to work into our agreements I think our and we also do it a ton of evaluation But I think our orientation is like it and to your question care I think it's from what we've seen it will be a shift towards qualitative from purely quantitative or towards Maybe even towards a better relationship between qualitative information about reader decisions and quantitative Because like we'll do a survey after almost any event we'll send something out and We may not run a ton of deep analytics on the responses because we may do a public newsroom that 15 people come to and some of our best ones Are also very small because the people they're niche and the people who show up really care about them And like the percentage of people who answer a certain way to a response I mean the sample size may just not be big enough like it's the The metrics in that sense aren't meaningful But what they write and the relationship between what they write and why they check a particular number Starts to inform our decision-making and also starts to inform how we read those bigger picture metrics So I think that's also like what are we going to do with all this data? Like we're going to continue to use it I hope but I think we'll Read it in a much different way and and have different Goals and what we want to see it delivering for us and for readers because the metrics that we get also like are not Self-evident like we don't we don't actually know why things are happening, you know even if we can like really thoroughly segment and Track people through a say a subscription funnel why somebody's actually motivated to Take a particular step like they may not even know and so the more that we can Have that conversation and better understand how our stakeholders feel about us and kind of fulfill their Expectations, I think that will be important for success in this new economy So are you nodding a lot? Yeah, I think measurement was one of my favorite things about the Guardian Mobile Innovation Lab because when we first started out We didn't know how we were gonna measure And we kind of stumbled upon our strategy Kind of serendipitously, but I'm so grateful for it and and that is really that Some of the reasons that we're so stuck I think on vanity metrics is that they're so easy, right? It's like well, we published a story that had 10,000 page use and then we published another story that had 20,000 page use So the second story is twice as good, right? Like that was a success And that relies on having a benchmark for success, right? Which if we're now like eight ten fifteen years into kind of like digital journalism You've got a lot of legacy vanity metrics that people kind of keep using but for us when we were we were building completely new formats like Like a jobs report notification explainer, which has no benchmark for it, right? So we talked to some really smart user experience Researcher folks at the Guardian office in London and said, you know, how would you measure this? And they said well, you know, we set the bar really low and said listen, we're doing something brand new These formats have never existed We just want to know we apply technology to this content was this useful for the reader and was it interesting? like that was it and If it was useful and if it was interesting then we would consider it like a success in the lab Because it's all additive to kind of all of the engagement and content that the Guardian was already kind of creating through through other means And and it certainly evolved from there and we of course looked at things like how many people opted into these job notifications and We did really granular metric setting, right? So we would have a whole custom GA Kind of implementation ready to go where we could see, you know If people were tapping on these little buttons that said it was a good job support was a bad job support When did they fall off and why and what time of day and in what country, right? And we actually hired people to do this for us, right? So we initially thought that we could have one analyst on our tiny little team And we were hiring for a lot of roles that we didn't have experience in right? I'm a product manager Sasha was an editor And we quickly realized after bringing in these poor folks that we were interviewing is that we really needed like three people right because analytics and measurement to me breaks down into kind of these three categories which is somebody who can kind of talk to you about What you need to measure and track right like set up kind of all of these custom tracking implementation frameworks like Like making sure that you have their geolocation and making sure that you have the action buttons on your notification Tracked so there's kind of that that tracking framework and then you need people to be able to implement the tracking Right, which is like a really huge task in and of itself And a lot of people like to drag and drop Google Analytics tracking But if you're doing something experimental you might need to get a little bit more granular So we needed that type of person And then we needed somebody at the end of the experiment to look at all the quantitative and all the qualitative data and Analyze it for us right like look for trends that takes a lot of time So we worked with a company. They've since been acquired. It's called hero digital. They're based in Philly And they did all of that for us right and they were kind of these spiritual gurus Like we would get really excited about the product and how we're gonna build it We would talk to them for an hour and they would deliver this like four-page Bible back to us and said, okay Here's all the questions that you want to answer by the end of this experiment And here's how you qualitatively and quantitatively measure against each of these questions Before anything ever launched which was totally kind of transformative for us, right? So we knew going into every experiment that we were going to be able to answer at least the questions that we could imagine in advance and And that was a really wonderful experience and we've written up, you know I don't want to kind of go on too long about it here But that would go into every write-up after we did an experiment we would publish all of our findings on medium Which was another kind of very like, you know Open thing for us to do and all of our code is kind of on github from that lab and we aspire to do the same thing Kind of in the in the LENFAS lab going forward as well But I I love measurement and I could tell you about one metric if it's interesting Called the net interaction rate. Does anyone want to hear about the net interaction rate? Just raise your hand. Okay, cool. Awesome Yes It's so it's such a great metric and it people tell me that it's like a net promoter score But I had never heard of that before but anyway Since we were building these new features and people were like engaging with it a lot They had to like touch all these action buttons or tap on a lot of things in a new live blog format We were trying to acknowledge the fact that like some people are going to have a negative experience, right? Like sometimes they're gonna tap a button on a notification. It's gonna go away And they really I didn't want that to go away, right? And the nice thing about running notification experiments is that you can send them another notification with a link to a Survey to tell us how it went. So that was like a really great loop a few back loop So so one of the parts of this process is that we would talk about all the ways that someone could like touch the experiment Like all the ways they could interact with it and we would code the interactions as positive or negative Right, so if somebody hits the button that says like stop this thing that was negative If somebody clicks through on an alert and it took you to an article that was positive They wanted to engage with more of this content So what you do is you just kind of take the positive interactions and you subtract the negative interactions Divided by all of the interactions and you get either kind of like a positive or negative Percentage point at the end, right? And what that did was it took into account kind of those negative Experiences they were going to happen But for us our measure of success was like on the whole overall if you were to summarize the output was this positive for folks And the even cooler thing about that is that we would like recode things as positive I really advocated for neutral interactions Right, and then that kind of like changed the game a little bit But it was this really flexible and wonderful metric that was so quantitative. It's so qualitative at the same time And I you know I would highly everyone probably has their own version of this and their organization But if not it's worth a conversation With kind of your analytics team your product team or whomever about kind of when people are using your stuff What is positive and what is negative even if you don't implement the metric like that conversation? Seems really essential to be having one thing on the subject of success that I think can sometimes be lost what is right there for us, which is also ease and simplicity you know, I think that Coming from an editorial background. I I love getting my head around the story But there's also just like what's right in front of us with the beauty and the simplicity of whatever Product we're engaging with and I think that's a whole nother part of measuring success of whatever we're producing So when you say simplicity you're talking about design. Are you talking? Okay? Yeah, I mean I'm designed for sure It's really great when things are aesthetically pleasing But then in addition I do mean to I think like In the last discussion some of you may have heard one of the Q&A questions was like It would be terrific if there was a way to subscribe and unsubscribe from news publications all in one place in the same way that I can subscribe and unsubscribe from podcasts and You know, I think that's an example of something where from a you know design perspective We would benefit thinking through some of those things Just one little thing an annexed about a notification We did do experiments other than notifications by the way, but it's easy to talk about For the presidential election We worked at the Guardian to do like a live data notification So you could sign up a couple of days before the election and get kind of the real-time results in an alert on your lock screen And it would update itself right it kind of acted as like a little ticker Which is a new way to use alerts normally people send an alert when the election like kicks off, right? And then when key states get announced and will send an alert like Kentucky and Montana went for this person But this allowed it to sit on your lock screen and constantly update anyway There was a chef in Australia who kind of responded to our survey and said like this was amazing Like I was in the middle of a shift and I was just super interested in the outcome And I didn't have to kind of like open a live blog or do this or do that And it was very easy and intuitive and like respectful of his time Yeah, and yeah, we we were always really grateful to get that type of feedback Why doesn't a chef reading his phone in the middle of a shift not bring comfort? That also reminds me I think I read something today about how the economists I actually haven't looked at it But they just changed their mobile app with like this very specific intent of like Not overwhelming you Which I think is another really cool thing Yeah I feel like we're kind of like funneling down to that but so many of the questions around innovation and design and Engagement, I feel it comes and we're talking about it before this idea of friction like who's who's friction like when there's friction Is it because you're making You're taking steps to accommodate What the user wants or needs or is it because you're putting up obstacles to so that your metrics look better I just think so much of it comes down to people want to be respected and that sense of respect is a much better foundation for The news economy then the advertising model was Yes, just excellent point Does anybody have any questions, please Good start do it Growing up in the Chicago what in the Chicago news market, but you know consuming news, you know It is such a high tower thing in a lot of ways Like there's a lot of news media here is not it's and I'm hearing a lot of newsletters a lot of push notifications That's all one way communication to a user and I find and even today as comments are becoming sort of pushed Being turned off on news websites at one point do users get the ability to talk back to organizations and or Also, like, you know, we're not data. We're humans We want to be able to interact and I think a lot of the reasons why I'm leaving traditional media Consuming traditional media and going towards organic podcasts is because I have that two-way Communication with the creator, you know, what how are you creating engagements with users to combat that kind of stuff today? And it's a complicated understand. No, I love that question. Can you actually though answer quickly? What what it is about podcasts that you think is different? So it's similar and different I think the reason why I go to podcasts because they are more niche and they're talking about topics that I care about versus You know one other bone to pick and I hate to say it's like I get breaking news alerts for stuff I really just don't care about and I can't turn them off and it's like I'm not a huge sports fan But I get breaking news alerts from news organizations in Chicago. I don't care and I can't really customize that very much But when I go to podcasts, I can't select what type of content I want and want to listen to right? So I think that kind of answers Yeah, I mean I feel like everything savior is doing is in response to that problem and Each of our programs like the idea is that there should be many many points of engagement that people Can take to participate in the process of producing local media not just like in engaging within the sense of commenting on a story That's already been published But on having access to journalists while they're reporting a story Especially if that story affects their lives or is about the conditions of their lives So I mean in practice the public newsroom like we're all at the public newsroom almost every week And there are regulars at the public newsroom and we use that as a space that people come to to give feedback We have editors office hours also which are publicized so Bettina Chang there's a coffee shop across the hall from our office and Every Wednesday every Tuesday afternoon from 3 30 to 5 30. She's there with a little Sign black and really whoever wants to show up and talk about something and there's often a lot of kind of Engagement around it to people can tweet at her if they have questions and The fellowship to is designed to but there's a pretty low barrier people need to have some experience producing media But the expectation is not that you're a professional journalist It's that you want to make a run at producing media that matters to you or Or that you think matters to the city and Documenters especially like that. We have about 350 people signed up for the program right now and roughly 200 that are eligible to take assignments And we're hoping that the volume of meetings that they're taking and just the amount of engagement in that program Scales a lot over the next couple of years, especially as we grow So yeah, I'm totally with you and I think the The question that we still have to answer I mean we're very much a startup We we bootstrapped it and we got some funding and now we're building out some of the infrastructure that I think will sort of See how much we can operationalize this and make this kind of a model for how media could work Yeah, and I hope that what we do can also be useful to Legacy media outlets that are sort of in this position of having a more traditional One-way relationship and and so you have some relationships too with other organizations like Mississippi today You're about to launch up in Detroit with I'm gonna screw it up. It's either DET or DTV. It is DET What has the what what of those Commerce conversations been like how that adoption that receptiveness to what he was talking about of bringing in Audiences into the process It's it's really early. So we're doing we're piloting Documentaries in Detroit with WDET the public radio station and a organization called citizen Detroit To especially around the elections that are coming up in Detroit and the school board meetings and all the charter school network meetings So we're seeing if documentaries on a smaller scale could be an effective way to cover some of those meetings and make sure that DET's reporters have access to good information about the about education in Detroit going into that election And then in Mississippi There's we've had a lot of conversations with folks in the Mississippi Delta It's a much different landscape than Chicago in a lot of ways. There's also a lot of deep history and interesting connections and but we're intrigued because Reporting in rural areas comes with unique challenges and it feels like having something like documentaries like it's very difficult to drive around for one reporter to cover an area like that, but if you have a Say a retired person who lives up the street from where that meeting is happening who cares a lot and might be going to that meeting Anyway and asking a lot of good questions at that meeting if they could go and take some structure notes and share those It's really really early and we're also Asking a lot about the relationship between Local we think these things should exist and be adopted locally And so how how we can support those without kind of coming in and replicating a lot of the problems is on our minds too But we write about it all the time so we'll we'll let you know how it goes Cool. Yes. They have a robust a robust channel on medium That is a treacherous of information and great ideas. Was that helpful? Awesome. Anybody else have questions