 Thank you. Hi everybody. No, I've met all of you already. So I'm Cecilia, you can also call me CeCe, and I've been a ticket master for a year. I moved down from the Silicon Valley last year to be warmer, but I still wanted to be in tech because I've been working the past 15 years at Apple, Expedia, Netflix, Microsoft, all the big tech companies, but now I get to come to LA and finally have a life and have fun and be warm. And I took to product school when I first got hired as a product manager. I was sort of on a trial run and I was trying to figure out what is product management, what are all my responsibilities and duties. So product school was really great. I took it in San Francisco for two months and it was just awesome because by the end I just had a much better understanding of exactly what I was responsible for. I think as a product manager you're always trying to figure out what's your ownership and what's your part to play. What are you in charge of? And people will say everything. That's not necessarily true, but definitely covers a large swath. So I'm in the Hollywood office, Live Nation. Feel free to give my contact and phone, come visit me right there on the Hollywood Walk of Fame. And Ticketmaster, there's a lot of products. There's the B2C side, which I'm on business to customer, and then there's the B2B side, which is more of Live Nation working with the venues and the artists and those types of people. How many people have used Ticketmaster in some capacity? Wow, oh my gosh. How many people have used the mobile apps, iOS and Android apps? Oh, that's a lot too. Okay, that's what my team is currently working on. And so there's a lot going on here. We have the Ticketmaster.com site, which is, looks like it's from the 90s. It has a whole bunch of stuff going on. There's a lot of ads. There's a lot of sponsorship. It has all the bells and whistles. Then there's the iOS and Android apps, which are doing really well. They're very well reviewed. They make a lot of money. They're just kind of chugging along. We have this brand new beta site, so if you visit Ticketmaster on your mobile phone, you'll might go into the progressive web app that's being developed, about 50% of traffic is being routed there. And then we have an ancient M.dot site looked over from the last decade, but that's still something that we have as well. So working at Ticketmaster is kind of crazy because there are so many different platforms. And each one of these different entry points has its own product management team, its own development team, its own set of designers. So there's a lot of moving pieces. And then every single one of these has to integrate the services on the back end. And this is the most technical job I've ever had in my life. I've never had to integrate with so many SDKs and APIs and work with data sciences and develop algorithms. But it's so exciting. It's like, wow, I can't believe I get to do this. I feel like teaching myself every morning. It's really fun. So about maybe three months ago, we started to think about what could we do at Ticketmaster that's different than we've been doing now, especially on the beta-c side, like how can we make things better for our fans? And why would we want to build something new? So I'm a huge Star Trek fan. And Star Trek is, it's old as well. And so as Ticketmaster, it started in 1976. So it's 41 years of history. And each iteration has pretty much been the same. There really hasn't been anything revolutionary or really, really new in a long time, much like Star Trek Discovery. So we were thinking like, how could we do something really bold and interesting and innovative and experimental and different? And that's what we've been working on for the past couple of months. Has anyone seen Discovery? No truckies? Well, they also have that pain model, so I wouldn't expect a lot of people to watch it. So we started to think like, how can we create a fan-centric experience? And once again, Ticketmaster is very complex. You've got the beta-c side, you've got the beta-b side, and both sides sort of like fight over what's more important because it's more important to adhere to the client, to the Hollywood Bowl, what they want Ticketmaster to be, or is it more important to adhere to the person off the street who's paying for the ticket? I'm on the customer side, coming from Avalon, very customer-focused. I'm always thinking about that person with their phone in their hand, just trying to get their ticket to Beyoncé or Taylor Swift and what are their needs and what are their problems. And so as a team of engineers and designers and product people, we thought, how can we find out what fans want? And we thought, let's just get out of the office and go talk to them. So we stalked people near the Greek theater and we just waited for them to come by. We were wearing Ticketmaster shirts and we just started to ask them questions about how did they find out about the event and where did they buy it and what were their expectations. And we started to gather all of this feedback and really learn where all of their pain points were and all of their frustrations. People were pretty forthcoming and so we had a great time just talking with people and then we started to bring people on site as well. So we started to reach out to thousands of people in the LA area who loved concerts. We were very focused on like the artist at this point and so we started to bring people in-house and show them different mocks and designs but more just fundamentally talking about needs and pain points and challenges and also things that made them happy. And it's easy to get people to talk about music events because pretty much everybody loves music so my job's not too hard. And we found that there were a lot of problems that Ticketmaster was not addressing. Number one is there's absolutely no personalization really with Ticketmaster. There's like the minimum amount since you ever get those emails from Ticketmaster saying here's your personal events. We handpicked them just for you. Maybe they are and maybe they aren't. It's not entirely personalized. We don't have mandatory login so there's not a lot of people who log into the app. Usually you log in when you want to buy but at the same time because there's no personalization people aren't coming repeatedly back to the app to browse. We don't know that you know you like a certain artist and therefore we should recommend you these other artists. We don't have any integrations and so we started talking about personalization and we still don't have Facebook login that's what I call McLaughlin, Agent Dale Hooper is saying what here is this? Did anybody watch 20 Peaks the Return? Okay very obscure that's fine. It was really good. So we're looking at you know how can we integrate the services like Spotify and Facebook and Shazam and bringing in people's personalized recommendations. When we talked to fans we found out there was no reason for them to really engage or come back to the app. A lot of people just deleted the Ticketmaster app after they bought their ticket. They take a screenshot of their ticket they delete the app maybe they'd open it again when they needed it but they thought well I got everything that I need from this why would I still have this app because they weren't using the Ticketmaster app to discover any of their events they would hear about it pretty much everywhere else on the internet except for Ticketmaster. So we thought well we definitely need a better discovery experience and then reason for people to come back. We don't have any coupons or discounts or any incentives there's no promise of getting a ticket so we did start this really cool new thing called Verified Fan through Springsteen as a part of it and what it does is it gets you a code to get you in the door to maybe get you a ticket but you're still kind of part of a lottery so there's no guaranteed seat and then there's no guarantee of the best deal because we don't do a really good job of merchandising you don't know like this is a better deal than Stubhub or this is a better deal than SeatGeek it's kind of like Ticketmaster has the monopoly on the ticketing industry so we just expect people to come to us but those of us sitting in the Hollywood office are like well we definitely have to do better we definitely can't assume that just because we have a monopoly that everyone's always going to come to us all the time like the Stubhubs of the world are mithing at our heels. There's also no bulletproof entry so when you have your mobile phone people are concerned about cellular alert at data connections trying to get into an event or what if my phone dies or if my battery gives out and I can't pull up my mobile entry so a lot of people are paranoid and they still print out paper tickets and so it feels very archaic and then finally there's no global app you can't download the UK version of the Ticketmaster app buy a ticket for a concert in the US and come to the US and get into the concert and there's a lot of technology reasons why there's no global app so these are a lot of the complaints that we heard from fans a lot of the opportunities and challenges and we also realized that we were dealing with a whole mess of technology problems as well so we're dealing with a company built in the 70s. Iowa's and Andrey they're kind of tugging along but there's no new innovation and plus every time we put new features into the app that app just grows and grows and grows and then we did cross the dreaded hundred megabytes earlier this year and had to pull it from the app store and immediately start taking things out because once you cross a hundred megabytes and you can't download it over data you have to use Wi-Fi and we're like we don't want to do that. Then we have to wait for the app store to approve this isn't a problem with Google Play but this is a problem with the iTunes store the release process is just slow and then we can't catch up to the rest of the business channels like when the mobile web team releases something they just do it like that for us it's like really cumbersome and difficult and just takes a long time and then there's all these different tech stacks worldwide like the tech stacks are just insane on all the different platforms and all the different product managers and it's just a huge undertaking and then like I said I've never seen so many APIs and SDKs in my life a ticket master it's like every third person down the hall is making their own API and sometimes it'll come to fruition and sometimes it'll meet your requirements and sometimes it won't and you're just constantly negotiating with all these areas of the business to figure out will this work for mobile and if it won't work for mobile then what do we need to do to make it work so we had an inception do you guys use the term inception I had never heard that before ticket master okay well we're always in septic we're always having these like three day long get 40 people in a room and come up with tons of ideas workshops we're actually having one tomorrow um and of course we just put every single idea that we could think of just through the whole kitchen sink in there and came up with a mission statement so after looking at all the fan problems on the technology problems we decided that this is our mission like we believe that if we create an end-to-end experience that's fast and personal it'll result in higher conversion people will come back to the app and will have fan satisfaction so this is the principle that we've been operating on every time we try a new idea or try a new feature we think is it fast is it personal is it engaging and just kind of using that as our north star um so we're still trying to figure out how we're going to solve all of this um from the technology side we thought we need one app to rule them all and you learn the brands fans in here and we need the return of the king we need an air gone from a foreign land to come in and save us and what we found was this guy named tom gray who wrote the book on react native and he had developed a successful app a ticket master called ticker on the react native platform and he came to us the white knight on a horse and said hey there was this great new javascript technology you can make it in react native you can take ios and android components and it's easy to release that into both stores simultaneously and also you don't need to wait for out the store approval because you can do just code pushes over the air so you can do really fast development in react native much faster than you can do in ios and android and so we really started to embrace that idea of having this one code base that could be reused for both platforms and we also looked at progressive web apps as well as instant apps because i believe in the next decade that there will no longer be an app store where you download an app i think it's going to be okay if it's going to be like going in the best buy and buying a piece of software off the shelf and going home and installing it like you'll get something on your phone and it will look and function like a native app but it will be probably web based and it just won't be the same that it is today so we want to build something for the future and then we keep having to think about the fan like how can we bring them in so we really wanted to focus on product research um internal user testing is really important so we launched the internal beta program and then we hooked it up to slack so people can go download this prototype version of the app provide feedback goes into a slack channel for me primarily to gather everybody's feedback and then we started using things like user testing com and alpha you guys used alpha they're really nice we don't have an official contract with them yet we're waiting until january to do that but they um do user testing where you can reach out to thousands of people and then in a couple of days you can get so much user feedback so you can do mocks and wireframes and video um alpha's a good testing platform and then we've been bringing in fans every single week giving them $50 gift cards for an hour of their time and just talking to them about what's important to them and that's what i'll be doing tomorrow morning so it's a lot of engagement it's a it's a lot of work and we're looking also to hire um a ux researcher if anybody knows anybody because we do need somebody dedicated to doing this but we don't have anybody so it just falls into product fans so we thought about our strategy when we put together this beautiful elegant roadmap and we're like how will we get to replacing the ios android app and our our vp gordon presented this to jerry the president of north america ticket master and the timeline stretched out into june which is a million light years from now and like product development and of course he came back to us and said how can you go faster these baseball fans this is my favorite movie yes and so they're like jerry's like we need you to go faster and we're like ah so we had to like go back to the drawing board and figure out we can't wait until june we've got to have something a lot sooner um and so we started thinking about what can we do that's going to be so revolutionarily different in this app like what's its value proposition going to be michael rapino is the head of light nation which owns ticket master and he's been going around on twitter and elsewhere telling everybody that like the days of the paper ticket are numbered we still have paper tickets right now we still have standard mail tickets that you can't even track through the usps that gets sent to you but he's so into this idea of mobile entry and being able to use um your field of communication on your phone not even having your phone turned on but just having your phone and that's your identity and so we were thinking like is there was something we can do around identity around mobile entry and then we also wanted to realize that this is a more exclusive experience so when we went out and talked to people it's not necessarily about age it's about comfortability but there were a lot of people who associated the desktop experience with being safe like it's safe to have my credit card there and it's safe to go and look at a map view on the desktop because i'm behind the seat that i'm looking for like being on my mobile device i don't feel the safe i don't feel like i have all the information that i need um and so we decided that we were going to make this mobile first but it wasn't going to be for everybody it was truly going to be for that more super fan who's comfortable on their mobile device who buys a ticket on the mobile device and expects mobile entry believe it or not it has been so hard to champion this etiquette master the idea of not having a paper ticket of letting that go like that that fear that they just want to have the papers somehow and being able to convince people that it's going to be okay i don't care about paper tickets at all we have talked about providing souvenirs for those of you who absolutely need to have something for a scrapbook but i'm on a bill for the future like this app is maybe two or three years down the road we won't have any more paper tickets but right now we do i'm still focused on the future and slowly dragging the rest of my team along with me so the other thing that's interesting about being a product manager at ticket master and i don't know why all of you became product managers but i became a product manager because i wanted to change the rules um the company that i was working at i was supporting a lot of fan-facing features that weren't working properly and the more i dug into it i realized it's because the requirements weren't working properly and the rules weren't working properly and my argument was well we wrote the software can't we change the rules and so i thought if i become a product manager then i could change the requirements and then i can make things better then i got to ticket master and then i realized that i'm on the b to c side and there's somebody over there in the shadows negotiating with you know the forum or the greek or the hollywood ball that i don't even know and they're coming up with all the requirements and i'm just getting it downstream and i was like no this is not why i got into this like i've got to find a way to get ahead of all of this so then we came up with a strategy where we thought what if we just partner with a couple of five-star clients and so we picked meterlander who does hamilton line came with disney on broadway and we thought what if we just take our experience that we want to build and we focus more on building for this five-star client and we get them really excited and invested and then hopefully they will propagate and you know spread the news to other people and we'll be able to change the requirements like really weird obscure requirements that existed decades ago that don't even apply in today's world so that's what i'm focused on changing is because i want things to be better and i want to know like why can't we do certain things and why are there those restrictions and how do we change these restrictions can you give us example of the requirements sure so this is awful but you cannot buy wheelchair accessible seats on most um ticket master platforms like you will open a map you will see a wheelchair seat you'll tap on it and it will tell you to call a phone number or fill out a form and i'm like that's horrible and it's because we we do have this technology that allows people who want an ad ac to buy it directly but not all the venues want to take the time to integrate with it or support the technology and so we're making it a requirement we're saying like no you can buy a visually impaired seat or hearing impaired and that's just how it is and you've got to integrate with this thing and you've got to support it because it's right it's right for the fans so i'm just looking at every tiny little requirement and figuring out is this the best experience for the fan and why isn't it and what can i push back on like super passionate about it so we have Jason Giles as some of you may know here and he's our head of design on this on this project and he's helping us with our guiding principles um we're definitely doing everything mobile first even if we're doing something that's like a progressive web app or an instant app that can scale the desktop it's still focused on mobile which is really great because we don't have an editor i've always worked with a content writer we have no content writer so we're just writing our own content but when you're mobile first you're focused on using as few words as humanly possible and big icons and adapting those design principles we did think about two taps and you're done like tap once to select your seat tap once to buy your tickets and that's it it started out that way there's a few more taps at the moment but we're still trying to get to the two taps we're like how can we get to the two taps um we're also working on the principle that we want to show everything at once so how many people are going to see Hamilton people so we thought like what if we took the Hamilton theater and what if we could show everything in that theater at once what if we could show all the dates and all the prices and all the seats and just expose all the inventory all the special offers that you could get all the discounts that you could get because i think one of the things that's that's hard with like Hamilton or those long running shows is that people are constantly clicking back and forth or tapping back and forth we're trying to find the date they're trying to find the price to try to find the seat all of these different factors and so we're like how can we expose all of that on a map and in thinking about that we thought well we want to highlight everything we don't want to filter out or hide anything and that's been a challenge with the designers because a lot of them have been at Take a Master for a long time and they're used to filters like filter out your special packages filter out your wheelchair accessible seats and i thought about it differently i'm like what if somebody comes into the map for the first time and they've never used it but they see a wheelchair icon what if they tap on it and all the wheelchair icons get expanded and exposed and then what if it asks do you want the font to be bigger or maybe we change the color scheme or maybe we invert the colors like what if we start to anticipate people's needs and what if we start to make it like a choose your own adventure where you start tapping on different areas in the map and all of a sudden more things get exposed like special offers or maybe you'll tap on a seat maybe it'll show like for 40 more you can get this special package or for $20 less you can sit a little bit back so how can we use real-time intelligence to make recommendations so that we don't have to have filters but we're just kind of highlighting and exposing different areas for people on the map all of that is very challenging to tell a designer that's been in a company for a long time because they're like whoa i don't know what you're talking about it's it's hard to change the mindset we've had some new people come in which is good the old people who have customer stress disorder we're just trying to help them get to a better place emotionally so that they feel more free to come up with new and exciting ideas and that's the thing that i wanted everyone to know on the team is that hey great ideas can come from anywhere like i want everyone to be sketching i want everyone to be dreaming about this app as much as i am um and definitely trying to get inspiration from places did anyone go to google i o google playtime was really cool it's telling jeff it was like patello for nerves lcd sound system played um but it was great to go to those different kinds of lectures and forums because we started to learn about augmented reality so i thought how can you take your phone into a venue and maybe hold it up and you see special things or maybe we take something like a media rich background from a company like i o media and we embed it in that experience maybe you hold up your phone and you get some sort of views from seat as you're sitting in your living room see what it looks like i think augmented reality is going to be much bigger than virtual reality just because virtual reality is like you have something on your head and you can't see anything augmented reality you can interact with your environment um and i went to google playtime which was more about the play store and the android app and then there i am by a cupcake in a mountain view and but it was really fun to go all these to all these events and learn about machine learning and learn about the latest you know ways to develop algorithms um and also there was a lecture on developing with emotions involved like thinking about how people feel about something and whether it makes them happy or sad and i really like that sort of thing so we just sort of drawing as a team we started sketching things and the ideas just kept coming really fast and furious we created a product rift section and i don't know about you but all the white guys in my 40s i work with love the word rift they just love to riff on things it's like this very casual informal time where everybody gets together you just draw stuff on the wall no idea is bad um but you're just supposed to come up with as much interesting stuff as you can and from that we started to develop these more um like put together mocks we don't have our branding figured out we've gone to like black and gold i'm not sure how that decision was made we might go back to the ticketmaster branding but we thought about this we thought about this map experience anybody wants to volunteer to come to take a master to like try this out let me know so i can use you guys all of guinea pigs but we showed people this the first time and we got a lot of interesting feedback these are supposed to be suggestions of what's available some people have thought is that all that there's available like some people haven't noticed the blue dots um some people are looking for a price slider to narrow down the price um these are supposed to highlight different ticket offers but people don't always see that we're playing around with the date slider here so here we have months by month and here we have day by day but we're still trying to figure out what to put on here we're trying to figure out how far in we should zoom um one of the problems is we've zoomed in so far people have lost sight of where the stage is so we're trying to figure out that where we are taking a lot of design principles from like google maps and apple maps we're like how can we make it feel more like a map how can we make the interaction feel smoother so we have developed this prototype we distributed it internally and then we've been bringing people into the office every week and making them sign ndas except for you guys and helping us um figure out is this a good experience and this one is a really challenging mess to figure out this is you okay you've selected these four seats and now we're like how do we tell people that maybe there's a special offer maybe there's a vip package or what if they're going to go see disney on ice and they want to switch one of those seats to a child seat so we're trying to think of different things to put here people have told us that an obstructed seat they absolutely need to know that seats obstructed but i'm like can we put like a barrier here or some kind of icon signifying hey your seat may be blocked or you might have a limited view we put a caution icon all of this is an influx but it's really fun to work on um the other thing that is happening is that all this information being surfaced here is coming from the back end of the metadata there is a litany of metadata like if you let a client decide what they want to put into a venue they will write no alcohol allowed water bottle must be 20 ounces cannot bring your backpack you write this much stuff and so once again we have to go back to the business side and say like hey can we help you guys set up something like a nice checkbox because like you go to airbnb and there's just like this really nice list that tells you everything that is involved in the in the house we need something like that to help people if they want to find out more information so here i'm trying to figure out the information hierarchy like what do people absolutely need to see to make a decision um so we're playing around with that we're playing around like what happens when you tap on it what happens when you tap on upgrade available it's just going to get more and more complex as time goes on so this is something that we're still solving for we're still solving checkout i've become very obsessed with checkout um insurance is one of those business requirements i couldn't get around because it makes us 80 million dollars a year and i'm like fine okay we have to have it but how can we have it nicely so that it doesn't take up all the space because if i were to expose this insurance it's like this much information we do insurance through alions and i'm definitely one of those people where when somebody tells me like oh we have to do it this way because we've always done it i'm like no i don't want to do it that way i'm a nice charming person baby i can make a difference so with alions i got on the phone with them and i was like hey alions it doesn't have video conferencing i was like on the phone with 16 people in new york and i was teasing them because they don't even have zoom or gold hangouts and i said is there a reason why insurance is like so long like can we have an option to like maybe shorten it because we're on a mobile device and they gave us back some really short tasks and i was like that's great and then i said wouldn't it be better if we could like do some personalized messaging to these customers like let's say that you're in LA and you're traveling to new york or the lion king like wouldn't it be great if you could say like hey it looks like you're traveling a long distance just know that if you buy ticket insurance and your flight gets delayed you can get your money refunded like something that's more personalized and they said yeah we would totally like to do that but we've never received any information from ticket master and so now i'm trying to figure out what is the minimum data that we could give alions so that they could send us back a personalized message that was a huge win for me i put that to my year review i was like you're welcome that's my top contribution because we just went back to the mission we're like we want this app to be personalized we want people to have a reason to get taken insurance it's there and people are like yes or no but wouldn't it be better if they knew that you were traveling or they knew that the event was six months out it might get canceled just finding some better way to message to the person instead of just this really impersonalized insurance messaging so we are now talking about how we will launch this thing that we have not finished building do you guys use tap litics do you guys do avie testing so this is the first time in my life i've really been able to do avie testing and oh my god is it exciting tap litics is a very nice company in canada full of lots of nice canadian people and they make it really easy to do visual editing so you plug in your iphone or your plug in your android you can change the color of things the icon the text you can do visual stuff as a product manager without having to get developers involved in it at all and you can choose what people see like i want 50 percent of people to see this and 50 percent to see that or you can do like four different tests or five different tasks so you can break it out and then you can get all of this real time feedback which is great because then you can go to your executives and say this is how it's performing and this is what's doing well and what's not doing well so we want to use tap litics to control this experience that we roll out because we don't want everyone to see it at once and it helps us control the traffic as well um android lets you slowly release versions apple just came up with the ability it goes from one percent to five percent to ten percent which i find kind of hilarious i'm like one to five percent strange roll out um but tap litics is a is a great resource if anybody is looking to do AV testing um this is the most challenging thing so start like software is very emotional like there are just so many emotions running very high i don't know if that's true for where you guys work but i mean i love everybody i work with but there are a lot of strong opinions and being a product manager i i feel like this i'm like just trapped in the middle you know you've got like the president giving you feedback and the fans giving you feedback and your other product managers giving you feedback and all the time i'm being told like cc you're the decider and i'm like am i really like i understand there's the president of this company who's probably going to decide some things over my head but the most important advice i was told was to have a really strong point of view so just develop that really strong point of view just put on your big girl pants and go into the meeting and just say this is what i believe you have to have the courage of your convictions especially when things go south um the emotions of other people are difficult to handle and i'm just choosing like not to handle those i'm like you're going to be the way you're going to be and i'm just going to continue to be an undaunted optimist that's my um that's my role in the organization so it is really difficult to decide like who's the decider in all of this have you guys seen the panagon wars it's fantastic it's from the late 90s with carrie always and kelsey grammar and what happens is there's this guy who's sitting in this room with a bunch of military brass and they want to develop this tank and he has to keep going back to carrie always keep changing the requirements and there's so much scope creep that by the time this tank gets released nobody wants it it doesn't serve any military purpose it doesn't hardly carry any soldiers it's the worst design tank on the planet and that of course is my fear in every product manager's fear i think is that we're going to start with two taps and you're done all of a sudden that's 16 taps and you're not yet it's really hard to avoid that scope creep it's really hard to develop a minimum viable product and who decides what that minimum viable product is um one of the things i do like to always say is well let's experiment let's release what we have let's try it out let's get feedback i think one of the worst things you can do is not pull the trigger and get your product out sooner than later because you need some kind of feedback and that's really hard to do too um especially when as a product manager you're told you're in charge and i'm like okay well how much like when can i decide to put this out here so that's why i did the internal beta i was like just gonna do it i'm just gonna put it out there i have the power and authority and control and i can start to get feedback before this thing gets to be an unwieldy tank um but it's really important to have that vision as long as you have that vision that vision is pulling everybody together so i just keep going back with the team and say like let's just keep focusing we want to create something that's fast and something that's personalized fast people can find a ticket they're looking for they can check out easily with no friction let's get social login it's making easy to identify who these people are so that we can personalize the experience let's get all the integrations that we need um let's just create something that people love and i always quote steve jobs because i was there at apple for such a long time when he was there and obviously simpler is always better so i'm a big believer in that and also the funny thing about steve jobs is that if he were still alive i wonder what he would think of all this agile and all these mbps we have so many product research groups and i think there's so much value in it but steve jobs was like we're just going to build something that no one knows until we release it and then they're going to wonder how they ever live without it but i don't suffer any illusions that i'm like steve so i'm like no i need product focus groups i need to talk to fans i need to gather the data um but i need to adhere to the vision and try to make it as simple as possible for people and not this side of that so that is what we are currently doing at ticket master any questions yeah right here the total use of these for ab testing um my company uses optimizly and it can be a bit of a hassle to get technically initially integrated into the optimizly like what we're testing yeah if you guys have any like trouble linking up the what was the name? tabletics to your actual live product so like how does the build happen versus like do you build everything in tabletics and then like it was so fast so we have um we have a bunch of developers we're working a bunch of people in ukraine they did it in like two hours and so what happens now is that the new prototype app is in tabletics so you go into tabletics it's there and then you plug your phone in and you just decide what you want to do it's so easy and fast to set up experiments and i i know about the other one that you're talking about too we also looked at that so yeah we were looking at a platform that you've worked with react native and ios and android tabletics was good the only bad thing is that all of these sdks that you introduced into your app will add weight and they will add delays so you do have to be aware of that so the what works really well is the fact that it relies on react native the application being built with react native yeah we think it's going to be a winner but it's it's funny at the same time right because like react native came out of facebook and went to facebook one day besides that we don't want react native and the whole development team is like but it's open source so i guess we can still do it it's kind of funny when you choose like what platform you're going to go with but it's still faster than than the traditional ios android development that we've been doing requires less developers as well how long ago did you guys transition to react native about three months ago so we got and this was difficult because some people decided not to stay on the team because we're like well i'm a diehard ios we develop we're like why do i want to do react native and so we just the people who wanted to be there we trained them and then we hired um some other contractors who knew react native and we took some other people from other teams kind of hobbled together a core group of people who could work in react and how long it takes to transition in two months it's not too bad like we all went through it i went through it for three days by day three i was like i don't know i got lost a long time ago but you guys all seem like you know what you're doing so good for you uh curiously you mentioned the app was kind of targeted towards like super fans this group that like will buy tickets through their mobile phone and the how do you figure out like or did you guys figure out like what the size of those people are at our curler platform and then tied in with that question do you guys have personas or sevens that you kind of identified take a master because they've been around for such a long time they have 12 different personas when you go through their training you're like wow that is a lot it's a lot of personas and so mobile was so different though and the thing is that what's awesome about the mobile stuff that we're building a lot of it is web based right now and so we're going to take it and we're going to put it back to the desktop versions the desktop people will get to take advantage of it as well but for the mobile people like we really were segmenting out we really were just targeting those people who are comfortable buying on a mobile device um who have their phone on the go with them and it does skew younger it skews like 18 to 34 so we just targeted specifically those people but we really also believe in the future of mobile and the power of mobile and we still have um the desktop for those that are just more comfortable using that um oh you first thanks um thank you so much for your presentation uh one of the questions at the beginning you were saying that um one of the challenges for our fine engagement was um they sent it for them to come back and use the uh use the app um what's sort of like has been some of your um your insights into that as to how figure that out well it's it's so fascinating I would challenge any product manager to never see of anything especially when you work for a big organization you think well somebody's already thought of that why would I ever bring that up when we talk to fans they would say things like hey it would be awesome to know like after this presale or on sale when you sold all these hairy style tickets if there are more tickets when they get released like it would be great for me to get a push notification on my phone and we already thought that that was a no-brainer and that somebody was already doing it like the NBA will do that too they'll release a couple tickets right before the game after their initial sale and so we're now working with marketing to do like those last minute push notifications like there are some push notification campaigns but the people that we talked to were really clamoring for it they're like if we knew that there were more tickets we would come back to Ticketmaster if we knew that there were resale tickets we wouldn't go to step hub it's just like there's all this lost educational opportunity where people just don't know that they can still get a ticket so we have to work with marketing on that that was really interesting I was like oh yeah I guess you would never come back if we never told you that there were tickets to that band that you wanted because the other problem that we have right now is that you can part an artist and you can follow them but it's not integrated everywhere so if you part them on the desktop it doesn't matter on mobile believe it or not we're trying to figure out all that it's awful but there's a lot of opportunities so it's great at the same time yeah but it's um it's interesting just to go off just to go off that um sorry the low hang so basically I think Ticketmaster is basically pushing you guys to do focus groups and get those low hanging fruit because I've worked with other companies and they are set under priorities and it's really hard for like to change the I don't know the CIO's mindset to be like hey there's a lot of cleanup that we can do yeah things that we could like like small wins that we could accomplish so was that like always since you got there no no there has been pretty much zero product research focus like they were doing internal testing and I'm like no I don't want to talk to people who already work at Ticketmaster who already understand technology like I want to find the person on the street who wants to go to a concert so Prisha as you know like she was the one who spearheaded it she was like let's go find people and I was talking to my manager and I was like you know what if we go to Craigslist or walk into coffee shops because that's what you do in the Silicon Valley you just walk down University Avenue in Palo Alto and talk to everybody in coffee shops and they were all about it like we found a more formal way to do it but it definitely was more guerrilla style where we just had to go to like one person and get their American Express card and try to go buy gift cards for people and it was sort of under the radar and it wasn't really sanctioned but we went out and we did it and it was successful and we bought the results back and then everybody was like oh this is great we should create a position to do this and we should create a team to do this so sometimes you just have to right I like the whole inception go out yeah it's like the whole inception like throw everything in the sink let's go ahead and go over everybody's ideas oh my gosh we came up with so many interesting ideas that we never would have thought of like having a chat bot inside the app to help you and check out or choose your tickets and stuff like that what were you gonna ask I had two questions um far as what's the percentage of best of sales versus mobile and then the second question I had is had y'all looked into like the airline industry and what they're known with far as you could pick out tickets like you know days in a fantasy every well in kayak and people like that far as um because you were saying talking about pricing and kind of saw the design you did and they don't look into how the airline really really captivates people to buy tickets as far as like I think sometimes there's like every Tuesday sent out the email as far as what tickets are talking price you know they're like the special club where they can get $90 tickets right so they be on looking to them because I look at the airline industry similar to the ticket industry right it's people buying seats yeah so it's like like I'm looking to and especially like Google buying that um is it the company that actually creates the software that does all the ticket sales do I think that right so create a link to what they have yeah I mean you're totally right I used her for Expedia slash botlier and there's like options for last minute deal alerts and last minute tickets and we don't have anything like that so that's like a huge opportunity that we can have there is like alert me when the price drops or alert me when these tickets go on sale so that's that is something that we want to add and then um desktop right now is is outpacing mobile but mobile is like slowly catching up um but people are just more familiar with the desktop site it's also the comfortability around maps so that's like a problem that we are trying to solve on the mobile device how do we render 90 000 seats instantly and make it an easy experience on the map I still think we're gonna have to convince some people but but we do want to do it we do still we still see it as an opportunity to do it for that person on the go and just and then the value proposition for mobile is to take that technology and put it back on the desktop site so at least we have both our bases covered and whatever new technology we're doing we're propagating it throughout the company I think if you're in that space of mobile and you've got a desktop um business like in in your company you should definitely partner with them and say like hey how can we both win together like definitely don't try to do something so different that they can't use it at all because some friction between the two but yeah it's but we do we still think mobile is the way of the future and people don't even know when they're on an app like we asked them like do you buy tickets on their phone and they could be on a website or they could be on an m dot site it's just as far as they know they're on their phone so they're they think they're using an app and there's a lot of different mobile channels for them to come into I think I know um the reason there's several reasons but one of the reasons I do desktop is when I'm buying tickets that are going on selling like 30 seconds yeah as opposed to mobile like like low inventory yeah we're trying to solve that problem we're thinking of creating a thing called the waiting room exactly it's like yeah I'll have five tabs open yeah and that's what we talked to people who did that they had like from safari firefoss and they had ios any other they had everything and they were like waiting and we're like how can we solve this problem on mobile like how could we guarantee these people a seat how could we truly put them in line because right now it says you're in line for tickets but I don't know where they are they're not in line for anything so if we could really realistically say like you were number 987 at least they would like know when we would give them some notification if they're like oh number 10 yeah exactly I came up with them we had that earlier this year and it was really successful it was this notify me feature in the app where you could be notified the like two hours went on sale or five minutes to get ready but yeah we have to solve the high demand situation um because that is when if you're on a mobile device and you don't have lots of tabs how else do you get in line and how else do you find out what you can buy at that exact second the other thing is the identity part so right now a verified fan you have to get a code manually put the code in and like that's horrible you should just log in it should know who you are and it should put you in line so we're definitely moving more towards that model like give us your login information and we'll put you in line and that's the incentive because right now it's like 10 percent of people log in but we don't know who these people are anonymous people looking for tickets and we want to know who you are yeah yes like so if you had your mobile phone and you're doing the same thing you know they would be like this is your same like they would actually do a number that should show you your number 30 000 out of 50 000 but the thing is I mean looking at those numbers you kind of you're like you're they're only available yeah people buy at least two tickets you are you kind of like maybe you want to get a ticket but they did the waiting concept where you go in an hour early waiting line the waiting room and then also this is like not top secret but you guys can tell anyone um overall coming up it's like right now on ticket master when you go to buy a ticket you have a timer it counts down yeah you are carting your seat and you are putting it in a shopping cart with a new experience we're coming up with um there's no timer there is no parting you tap it you buy it you're done and so we're coming up with a whole new way to build the architecture so that we avoid collisions but then we also avoid fraud so one of the reasons it's hard to get tickets is because like bots and bad actors will cart tickets and they'll hold on to them but they won't complete the transaction and so we've got away from capture and now we're trying to get away from people having to enter their security code for their debit card so they're like if we have people log in and we don't have a timer and you're not actually parting the ticket but you're just buying it and getting the order and then getting the ticket can we reduce fraud and can we make it happen faster for people um and not kick them out obvious place would be good too so does that require a login first and then a purchase yeah which we're gonna have to abt test i would like the login my manager is skeptical yes what did you have a question well if you're asking actually about uh to that point are you guys doing like blockchain for the um ab testing no no for like fraud prevention so there's a there's a bunch of algorithms on the back end with fraud and it will do its best to figure out if you're a real person and it's what it will try to calculate is intent so obviously if you've logged in you have past purchase history that it knows you're a real person but there's all these gates that you have to jump through all these hoops and if it doesn't think that you're a real person or who thinks that you're a broker and you're a bot then it will make it harder for you to purchase and then there's a there's a limit of six tickets per order um in some situations so like every time you go into one of these different channels you're constantly engaging with these fraud algorithms it's really interesting um but i'm hoping that with i with logging in and maybe two factor authentication or a little bit more security like how kpal verifies her phone number and that this is your mobile device that we can help people just log in and get in and then they don't have to be faced with all of these different fraud hoops to jump through but it is a constant process to to fight the the bots mainly in russia what you talked about artificial learning is that what you were alluding to was the fraud fraud is one part of the artificial learning um what we're looking for from personalization is that you go into the map for the first time and it's a cold start so we might show you a couple of markers but the second time you come in like we'll know how many people you're purchasing for what price grade do you normally have where you're where you normally sit and we want to get to the place that we're so sophisticated that we could send you a text message and say like hey there's two tickets to taylor squift do you want to buy them you press one for yes we buy your tickets like we want to get to that space where we know you so well that we can just buy your tickets for you and maybe integrate with alexa like alexa buy me two tickets to taylor what i've done then i just did it