 Hey what's up? My name is Shahina. I'm a product lead at LinkedIn and my talk is about building search and discovery products to find opportunity. Quick background about me first. I grew up in the former Soviet Union in Tashkent which is the capital city of Uzbekistan and if you haven't been to that part of the world I definitely recommend checking it out. It's an interesting place. So I got myself out of there when I was 17, moved to the US to study in college, studied computer science and of course the natural step after that was to become an engineer so I did that for a couple of years and then discovered the wonderful world of product and jumped in, never looked back. So this is how we'll spend our next 20 to 30-ish minutes. I will for those of you who are not familiar with the search problem I'll give a very quick brief intro and then we'll give you an overview of the LinkedIn products, search and discovery products. Then we'll jump in to kind of under the hood to see what powers these search and discovery experiences and then at the end I'll try to recruit you to work on search and discovery problems in your PM career. Sounds good? Cool. All right so search is a computational problem that requires finding a solution in some solution space and that solution space can be infinite and of course the act of searching means you're looking for something you have some sort of a question whereas I think of the discovery problem as finding solutions even though when you're even when you're not looking for them all right so you might not even know that they exist. Can you guys see in the back actually? Okay great. Kind of. I find it pretty cool that even in nature there's so many great examples of search and discovery and especially natural selection and direct evolution is actually a really elegant search strategy right over the course of generations it's trying to find a DNA sequence that leads to organisms with maximum with the highest chances of survival and on that node I have like kind of there's a question what is the universe searching for is it if it's searching for something or anything is it that longevity survival right like life is it meaning a purpose or love maybe or truth right it could be so many different things or maybe a better question is what are you searching for and if a lot of our a lot of us in the product management role we're kind of obsessed about finding product market fit so that could be one of the answers. Whatever it is that you're searching for what kind of what's in the core of it is really the concept of intent which is a very important concept to understand in the world of search and discovery and it can be quite complex it has multiple dimensions components to it and depending on your application in use case you might want to think through certain like key dimensions I've listed a few of them here things like the strength of intent ranging from not at all to I have a very high intent and also there's specificity do I know what I'm searching for what my intent is right or am I looking for something very specific or kind of vague or I don't know at all there's the effort basically how much effort am I willing to put into my process of searching there is the control how much control do I need things like features like sorting and filtering capabilities and that that sort of thing the level of consideration am I am I just like sitting back relaxing and open to serendipitously discovering or am I like leaning in right and sifting through every result carefully to find what I'm looking for so in this particular type of visualization very high intent query based search could be kind of somewhere in the outer kind of area whereas more of a push product like more of discovery products such as news feed would be more around the center and then some sort of a browse product could be somewhere in between but like is that for your use case and application you might need to look at other types of dimensions or a subset of these or super set so another concept that's very important in the world of search is precision and recall one way to really internalize it I found the really easy way is through the statement of oath I'm searching for truth the whole truth nothing but the truth right so so we're searching for truth the whole truth not a part of it right like not half of it so that's basically perfect recall and I'm searching for nothing but the truth right like not some truth mixed in with the lies or fake truth and whatnot and that's the concept of precision right so let's switch gears to linkedin linkedin is of course a professional network and our mission is to connect world's professionals to make them more productive and successful our overarching vision is to create opportunity for every member in the global workforce and to do that we've been building an economic graph which is a digital representation of the world's economy and that includes a digital profile for every member in the global workforce for every company in the workforce in the global economy every job provided by those companies every skill required to obtain those jobs all of the school's academic institutions and we overlay on top of that the professional knowledge shared by our members our companies and schools and universities search product is place a really key role to make this graph really actionable and useful for our members because by searching the nodes and address of the graph we we can find our members dreams dream jobs right like they're or connect them to the right people or to the right type of learning content and so on so in terms of search what and like given our company mission vision what we really chase and what we obsess about is this opportunity right so it's just really at the heart of everything that we do because we want to find the right people the right jobs the right right like the right content for our members in terms of the users on LinkedIn who use search product we have consumers folks such as yourselves who are trying to find other people by names for instance or by other criteria and consumers who are looking for jobs as well or for content and then on the other hand we have outbound professionals who use our products such as recruiter and we have another product called sales navigator they these these folks use LinkedIn search as part of their daily job to find clients prospects and recruits and so on the intent as I mentioned is very complex it has multiple dimensions but in the in our context in the context of LinkedIn we kind of categorize it into two higher level categories navigational and exploratory navigational being I know what I want I'm looking for it there's one right answer I'm trying to get to it as fast as I can whereas exploratory is I might have some idea of like what I'm looking for I just want to explore and look around and then as I'm doing that I might actually clarify my intent and like I might use some of the filtering capabilities for instance right like just through the results and so on we have multiple products from in the search land to help with those use for the to help those users and those intents we have a search home screen that you see on the left here and that's the screen that you see once you tap on or click on the search box this is the initial screen even before you type in your query so that's that screen here we give you entry paths entry points to explore the graph like if you want to explore people or jobs that sort of thing also give you your show you your previous queries for recall or if you're one of those people who does the same search over and over again throughout the day sort of thing and then on the right you see a screen called type ahead so that's a screen that you the results that you see once you start typing in your query one character at a time right it's rendering the results I have the instant search once you submit your query we that the experience search results page you see on the left I have my query here as product manager San Francisco since we have multiple entities in the economic graph and this query is actually quite ambiguous I might be looking for people might be looking for jobs right like writing manager jobs in San Francisco I might be looking for content and so on so we retrieve all the results from different verticals which are these entities from our economic graph and present them in the tabbed experience also we have a top tab which is kind of the aggregation of those from the most relevant verticals we fetch some top results and then of course we have filters and faceting capability one thing you want to call out is this content search experience post-search experience that is that is relatively new we launched it last year and that has multiple different content types that we have like articles and videos and feed updates or like shares right like that you share on the feed status updates and so on so we do kind of we aggregate those and show them here and then on the right you see story lines which is which we've launched recently as well what these are these are news stories news stories about professional news happening in the in your industry and this is actually a combination of editorial and algorithmic experience and that's been working pretty good really well for us all right so now let's take a look under the hood and see what's actually powering these experiences I kind of think of this finding opportunity process as a three-step process one we of course want to understand your underlying intent either we want to predict it or like really want to understand it so that we can find the right answers for you and present them and so that's step two and then three we don't want to stop there we actually go want to go beyond and want to kind of inspire you right to explore further and discover further so I'll go through these one by one for the first one is part of kind of predicting and understanding intent even before you start typing in your query who you are actually gives us a good idea about what about your intent underlying intent and that actually helps us further down the road personalize the search results for you as well to do that we have a machine learning model that is trained on search on your profile data and your previous searching behavior and what we're trying to figure out is are you a job seeker are you a content consumer right or are you a recruiter looking for people and like I said that helps us personalize the experience down the road and then once you start typing in your query we want to understand your intent and to do that we have multiple different kind of features components spell check we want to automatically correct your query as you're typing in we have other complete suggestions as well to help you formulate your query and we have we do vertical suggestions also let's go through these one by one for automatic spell correction we do this both on this on the type ahead as you're typing in your query we want to grace gracefully recover and kind of predict you know like do spell checking and we do that on the search results page as well on the first example here at the top this is the example of automatic spell correction in the audicum police suggestion I'm trying to type in creative but I omitted a and it recovered gracefully the second one actually is the name I'm trying to spell the name I'm trying to find Marissa mayor but I omitted one of the assets there but even if I did that it found the right Marissa mayor for me and then that's the third one I did a pretty horrible job of spelling deep learning but it found it so it is another machine learning model and for type ahead use case it does support the partial queries because it's more of a spell check as you type sort of a service in terms of auto complete well this actually is my favorite feature in search engines because it's a major time saver all right so here what we have is a machine learning model that's trained on query logs and it basically it's based on like co-currents of entities in these queries that we see we've seen in the past it's personalized as well so in this case I've typed in product manager SA we're trying to figure out what's the probability of San Francisco given product manager what's the probability of SAS for instance given product manager and so on like how common these are right like in the past queries that we've seen let's see and then it is personalized for instance if I would type in MA for instance I might get an auto complete suggestion for machine but someone else might get an auto complete suggestion for manager and so on all right and then once we present auto complete suggestions if the query is ambiguous then we suggest verticals as well so in this case I'm trying I'm going for machine learning machine learning is a pretty ambiguous query we don't know if you're looking for people with machine learning in the machine learning field or are you looking for jobs that require machine learning skills so we do suggest both people and jobs verticals on top of those auto complete suggestions another ML model and it's trained on previous searches and actions so then as we're trying to understand the intent so first of all like all of that was actually query assistance meaning we want to help you help you enter the right query right so it's like a help us help in kind of a situation because if you put in junk you will get junk out so we were trying to really help you formulate the right query right like with spouse spelt checking with auto complete suggestions vertical suggestions and so on then we actually want to kind of go further we want to predict the intent behind your query so whether it's navigational or exploratory in this example my query is software engineer google new york the intent is most probably exploratory i'm like looking i'm like looking through all the software engineers in google new york it could be actually navigational as well maybe i've just met someone who's a software engineer from google and he's out new york i just don't remember their name and his name and just like trying to find it this way as well but most likely it's exploratory and we try to predict the vertical here as well you might maybe you're looking for people or maybe you're looking for jobs then what we do is query tagging which is we take your query chop it up into logical units and try to recognize each unit against the with the entity that we have in our economic graph right so software engineer is one unit and that is a title it could be a skill as well google is another unit it's a company and new york of course is a location right so we what we do is we recognize these and we annotate the query each unit basically with these with the entities and submit that to the search engine what that does is it really helps with precision so how it works we have an ml model here that's trained on the user profiles to kind of build up the language model and then also trained on query logs and kind of manually curated dictionaries that we've accumulated over the years these are the dictionaries of titles standardized titles that we know of and all the company names all right and then then our next step is actually to do query expansion so here we want to increase recall by doing things like synonyms synonym expansions an example we do that for names for in the people search case and for jobs we do that with titles as well so in the case of a name for instance i have typed in jeffrey w i'm actually trying to find jeff wainer so it but it found jeff wainer right even though he doesn't spell his name with with jeffrey because we understand that jeff and jeffrey are synonymous and we have these basically in our dictionaries and and then see i am and then in the jobs like in the title case similar situation software engineer would be synonymous with software developer and so on how we do that is we have machinery models that are trained in query reformulations for instance in the past if i have typed in jeff wainer looked at the results didn't find the person i was looking for and then changed my query to jeff wainer looked at the results and clicked found one and clicked on it so that's the type of data we train on and then a slightly kind of a different variation of names synonyms is named clustering which is the kind of alternative different spellings of the same names for instance katelyn there's multiple very different ways to spell that right like with a c or with a k l i n l y n and so on so we have solutions for that as well all right so that was all about the query understanding the query the intent behind the query then we do ranking for ranking we have separate models for different verticals that we have we have different people ranking model we have jobs you content model and so on we also have a separate model for type ahead because in the type ahead as you're typing in your query we need to be really fast right and we might we especially on mobile devices we need to really gracefully kind of recover from spelling errors or like fat fingering errors and that sort of thing so we like the features that that we use in the machine learning models for type ahead are actually quite small because we don't have the luxury of time whereas on this actual search results pages we have the luxury of time so we can kind of consider the vast number of features that we have available it is a machine learned model it's it's personalized so it is the function of the query and the user remember initially like i said like who you are actually helps us with figuring out what kind of results we should show you that we think you might find relevant and then some of the for people surges case for ranking some of the features common features that we have our network distance how closely are you connected to the person you are looking for the connection strength how many people for instance that you have in common or do you have your school in common or company in common that sort of thing global popularity even without the kind of consideration of your network like your global popularity plays a role as well in that case that i've shown you with marissa mayor since she's like globally popular even though i'm not connected to her she's out of my network that showed up as a first result in that in that example and then of course spamming us we want to demote people who span their titles with like every skill they have or every company they work that so don't do that in the content search case some of the common features are freshness like how fresh the content is especially for exploratory content search experience if i'm searching for machine learning we want to give you more of kind of fresh results and then whether the author of the content the article or you know feed share and so on is in your network or not a lot of the times that actually matters and engagement signals like number of likes number of comments number of shares that sort of thing and then in this specific storyline case as i mentioned it's a hybrid editorial and algorithmic solution so we look at the editor tax so editors basically seed the storylines with few content pieces that they have found that they think are very relevant to build the storylines first all right and then blending is a solution we have in couple of in few different experiences one of the examples is the top serp like i mentioned earlier we have multiple verticals it results from multiple verticals we want to retrieve the top ones from the most relevant verticals and like blend them and show them to you in a lot of cases this is kind of actually like less blending more like swirling kind of a thing but the same thing on the title had experience as well in this case i am typing in design we show your company results that have the design in them we show your schools groups and so on let's see so then yeah so that was ranking and blending the third step um in our overall process was just overall kind of help you discover right like explore further some examples here are just we're showing you common connections even though you haven't asked like in the people search results even though you haven't asked for it so like as the next step you might want to kind of continue exploring like that or in the content search experience uh my my query is bitcoin uh the first result here is the is an article by fast company and we know the topic of this article and we show you other related topics so you can jump in so like cryptography computing and so on in the tags there so you can just kind of keep exploring cool um all right so hopefully that gives you a kind of a good idea of how we build search and discovery products on linkedin next i would uh if you haven't worked on search problems before i think you should and here are some reasons overall i think search and discovery is actually one of the very few products where you where you get to kind of think very big picture and like high level as you're kind of thinking through like holistically thinking through all the all the documents or things or like products that you need to make like searchable and discoverable right like in your company um and but at the same time you kind of think uh all the way down to my new details like super tiny details as well like in the example of that type ahead experience where like as i'm typing one character at a time we obsess about or i just typed in j how come my result is not looking good then i typed in another character right like is the result looking good like how should we uh how the ux should work and so on um so it's just all of it's just the whole spectrum really like the big picture to like to the tiny details and i find that you actually really exercise your systems thinking muscle a lot as well because you're kind of holistically taking a look at every organization in your company and like see how they relate kind of trying to find um uh commonalities between seemingly unrelated things in terms of the ux it's actually um pretty it's pretty um hard to design with static mocks oftentimes you have to do a lot of prototyping to get really to get the feeling of it and to really kind of train your intuition and so on so i find it pretty fascinating from that perspective as well of course you're dealing with a shit ton of data all right there's like a lot of metrics that you're looking at on a daily basis trying to find insights and kind of causations correlations and so on and you're doing a lot of mental gymnastics to get to get to those insights from pure product management perspective from product building perspective it really has all of the components it has um the ux part it has the relevance like algorithmic machine learning part um it has the platform itself the uh infrastructure the search index and it has like analytics pieces and so on so you get exposure to all of these different areas search actually was the biggest the first big data application on the web right and it started you like kind of employing artificial intelligence and machine learning early on so you learn a ton about that area and as i mentioned very uh in the beginning as well there's so many there are many examples of search in nature as well so you really learn about biological intelligence too cool um in terms of how we measure success there's as i mentioned there are so many different metrics um that we keep track of implicit explicit there's like true north like what is actually what do we find uh what do we uh define as success it's definitely not that binary thing um there's whole levels of it um like whether it's like successful sessions search sessions or like success rate of the session um whether like downstream metric impact right like did people end up starting conversations with each other did they connect to each other um right like that or sharing the content and so on and so forth um there's a lot of quantitative aspects qualitative as well we do a whole lot of uer and um and uh kind of user feedback sessions and that sort of thing from qualitative aspects as well um there's a lot of engagement metrics right like conversion metrics um so it's just basically across the whole spectrum um so yeah it's it's a lot of fun cool i think that's all i got for now any questions yes uh s.c.o strategy for linkedin uh what specifically how can i increase my exposure that does my profile like how come in and increase the exposure for your hiring i still don't need to take it away so s.c.o like basically is very separate right from the internal search and discovery products that we do um so we do have uh overall strategies for depending on if you are talking about your own profile or um your kind if you're a company page owner right you have your own company um and like s.c.o strategies for that for instance and so on so but it's very very kind of separate from the search products that we have though very hard trend to in our profile so we can increase our exposure or can we make like maybe our link profile can be referred by other websites so we can also increase our exposure i see i see interesting that's a great question i actually don't know if we do publish or kind of advise on s.c.o strategies but we can try our offline i i'm not sure great question yeah yeah so that it's just basically i can't get back to it but uh you know like as um kind of it was different dimensions as you go to the outer kind of edges um it's more gets more and more specific right so the bigger kind of circle is really your high intense search like more of the average whereas like as you go in it's like more of discovery gets getting into kind of more typical discovery products so the the smallest circle in the center was more of like a feed product which is like all push and that sort of thing um and the middle one was i was thinking it's more like a browse for instance like you start with some idea but you're still kind of open to exploring um that sort of thing and the big one was like a search yeah so that's the most specific is there something more specific within search uh something so there's different variations of search too right so like whether in the search um you you have your query if we're kind of you have our definition of search as it's a query based search um you can specify additional parameters like filters for instance right like and sort order and all of that kind of stuff so you can get like more and more specific as well and then if you've specified a vertical like i'm you're only focusing on people for instance and that sort of thing yeah so the question or the other yeah oh yes thank you you have a question um okay cool let's take the first question can you share an example of a b test of a new feature and what kind of success metrics related to it um very question so as i mentioned uh new one of the new features was um the content search experience before sometime i think we launched this sometime last year before that we didn't have um the ability to search the vast kind of amount of content that we had on linkedin whether it's articles or um the status updates and shares and so on so when we did that we of course gradually ramped it um some of the metrics like we ramped it and ab tested it some of the metrics that we weren't looking at before um that came from content search where for instance uh for um especially for exploratory content search experience was time spent um so as you if you if you search for machine learning and like you're looking through the the results um how much time are you spending time spent can be a very tricky thing right so like if you're actually wanting to like if you're spending a whole lot of time that might mean that you're kind of lost you can't find what you're looking for or it could be a really good thing if you're actually really engaged with the content and exploring other metrics like number of likes and comments number of shares and really scrolls for instance as well like can be some measure um uh as well and then all the typical metrics um that we already tracked for search does that answer the question any cool let's move to the next one do any of your colleagues have non-engineering engineering backgrounds if not what are the most critical systems or languages you work with daily do any I'm assuming my colleagues in terms of product managers um so they let me think my my when I so I think all of the product managers have in the search team currently have um are coming from engineering backgrounds but um that said in previous companies I've worked at where I've worked on search um I've seen very successful product managers who did not come from engineering backgrounds as well so coming from like purely business backgrounds or even philosophy for instance and um that did not stop them um and really they they became successful as well so it really I think it it's not really about what you studied in school I think it's just your overall curiosity in what you kind of continuously learning and that sort of thing um and then the what was the the next part of that question I think like what language is yeah um so we so we in terms of like languages we use Java um a lot and um in terms of our search platform itself we have our own platform called Galen and it's um it's built on top of the open source lucene search stack so um from the kind of platform infrastructure level we use that any questions from the audience yeah quick question so question was significant portion of my talk was highlighting machine learning and algorithmic aspects and the question is how does a p.m add value there well there are different um different components to search product right like I've highlighted more of like machine learning kind of part there's definitely heavy ux component there as well and overall kind of platform uh indexed part um there's um you know it's it's a very analytical product for sure so you exercise that muscle and um at linkedin all of us actually work through all of these different components so everyone touches relevance works with the relevance engineers and with ux and designers and so on um because it just worked out that their backgrounds and interests lie there um whereas in different companies I've worked our I've worked out in the past for instance at ebay some search product managers focused solely on ux part and some solely on relevance part and that sort of thing the value added by product managers is just really across at linkedin especially um so we think about the problems that we need to solve um and um then go from there and see if it maybe the solution only requires the ux part like there's only design that needs to get tweaked or maybe just touches all of these other components as well so depending on that we work with different you know different types of teams uh question is how decisions are made combining different features together and different functions together such as engineering product management and design features themselves it's really especially for newer projects we start with like brainstorming with the whole team with engineers with product managers designers data scientists and so on and figure out what are the different features features that we need to use in our machine learning model because good ideas come from everywhere right it's not just a machine learning engineer's job to figure out the features in terms of the um who like different functions at the at the so like your question is around like what's the contribution from different functions or how decisions are made so the decisions made from the process so it is a collaboration for sure depends depends on the feature that we're launching as well for instance if it's a minor tweak let's say to a type ahead model um I just work as a product manager work with a relevance engineer and that doesn't touch any of the design for instance or like ux um you know area so it just can be a contained kind of conversation like that and we go ahead and decide then um rampant you know start an A B test and see how it's performing so it's very a lot of data driven decision making especially at linkedin especially with search products and just the nature of that um but then higher level ux decisions is actually a collaboration with product manager product managers designers engineers like relevance engineers and just overall ui engineers and data scientists um right like everyone in in the room um some things are more qualitative than quantitative some things are kind of we can test easily some things are just more of design direction that we want to take and we do more qualitative studies there so um it's really different depending on the solution how much of how much of the decisions are based on user feedback um a lot well a lot of it actually so we do um when we are first ramping a new feature uh we do quite a quite a lot of dog footing so we uh we um ramp it to initially to the team only and then to the company and everyone basically submits feedback right saying hey like you you like what changed here i can't find what i was what i was able to find anymore sort of things it's really easy mechanisms a mechanism for us to gather feedback um and then we start if everything is looking good we start ramping to public um gradually um and we start if something is really broken we definitely hear from customer support and we closely monitor that and definitely we pay closer close attention to those issues and fix those a lot of the other things um with search like you might not want to as a user of the search product might not want to report every small thing kind of nuance or like something that um really kind of annoys you right for those usually show up in our metrics so we uh yeah we look at metrics a lot let me go back to uh Slido are there taxonomies you use homegrown or third party taxonomies in terms of so we have taxonomies of standardized titles like job titles for instance and uh skills have taxonomies all of that is uh is homegrown we that we have accumulated over the years we employ several taxonomies who are constantly looking at the kind of quality of data and standardizing understanding what is in the DNA of a skill for instance um that sort of thing so um so all of that is just internal um how are you measuring success of your product all right so there's so many different uh kind of variables different metrics that we track the true north of it is from quantitative side is um our success rate of our searches search sessions how successful we have been finding what you were looking for it's a quite complex metric and we're constantly evolving it other kind of similar one to it is the successful number of overall successful search sessions and of course that gives us the kind of the how many like the unique users part as well how many people are using our product and if that's growing or not um and then like depending on the specific feature for instance for type ahead the success of it is click through race on type ahead for instance right um another thing we track especially for navigational intent is time it took for you to click on the first result so once you submit your query or as you're typing your query how long it took you to get to the first result um so that's another one um and then um let's see on qualitative perspective like I said we do a lot of user user testing um in typically in search systems um they do a lot of human judgment a lot of crowd source human judgments basically you employ like services like crowd flower and ask the human judges to kind of grade your search product for you for different queries that you give them in our case it's actually pretty hard um because um it's very subject relevance is very subjective for us because we're a network so your network your relationships actually play a key role and also who you are as well whether you are a recruiter or a job seeker or a content um you know consumer for instance um really uh dictates what you find relevant and especially for more like ambiguous queries so it's hard for us to gather human judgment uh human labeled data so instead what we use is the click logs um so from that perspective we don't like in typical search systems you do use human judgment uh we don't really although we do use human judgment for like WTF results like we show our judges um the you know queries and then the responses the results that we have and they can say oh like this is a completely irrelevant result instead of saying what's the most relevant because that's subjective we do it's constantly so the question is what's a hard definition of successful search session and it is evolving as we like currently as well very kind of high level definition of it is overall our search sessions are bounded by um the you're clicking on the search box so once you tap on your search box that starts your search section until uh search session until you do that again um and then within that what's successful is the kind of easiest measure of success is a click uh so like if you clicked on a search result that success although it's tricky because you can click on a result and actually get back to it because it you thought it was relevant but it really wasn't right that sort of thing um uh so we do kind of and try to understand what's your dwell time um in the uh in the destination and that sort of thing yes question uh what are the differences between mobile and desktop version what are the aspects uh if you look at to decide if you should be included in the uh what are the differences um between the desktop and mobile versions of search products on mobile we are um it's one thing is the actual usage of the product is different people are intent people tend to have more of navigational intent on mobile because you're just you met someone you're quickly trying to find their profile sort of thing um and recruiters for instance who do a lot of you know exploratory searches they don't do that on mobile right like they want their you know during the day they use desktop so from that perspective even the intent types of intent we see types of users we see are different uh so we optimize for those um in uh and then ux of course like we're working with a small screen on mobile so you know just over a lot like we're optimizing for that whereas on desktop we have a bigger real estate so we can show you additional kind of snippets um additional kind of results um kind of uh or um additional information on search results so those are some of the criteria uh let me go back to slido how do you manage personalization of mm plus users millions i'm assuming um so how do we use how do we do personalization um so for us um personalization is a key because your identity and your relationships matter a whole lot right your identity in terms of who you are um who you're and like where you work at all that um and your previous uh searching behavior relationships in terms of the network itself who you're connected to and all of that uh so how do we do that at scale um really we have machine learning models um that are trained uh right like because it's really hard to do it with rules based solutions so we train models um on personalized features as i mentioned in different time of type of product components that we have we're looking at people's query formulations we're looking at their past search search in behavior we're looking at their kind of trying to model the who they are um with their searcher intent um and so on and so forth question from the audience yes so uh two part question one is uh since you're so far ahead with your uh what kind of users are you looking on right now and the second part of that is uh do you go through uh a cost versus there many kind of analysis for new businesses or because you you keep the user experience as the most important thing and then whatever cost it takes you guys kind of do it great question so question is um what what um so the second one was a cost so overall kind of how since we have a very established product how do we uh i don't know okay okay basically use cases okay so what are these cases and then how do we decide on them so it is um it's a combination of things it depends on what the focus is from business perspective um at any given time as well um and of course what are the gaps in our current offering uh what we want to do and looking at the industry where the search technology the search in general product is evolving more especially moving more into the discovery space for instance um and the technology itself right like what are the different capabilities that we can use um especially for instance getting into like deep learning space and that sort of thing so it's really the combination and um we do our quarterly planning and we figure out okay what is the problem that we want to solve um for that quarter and some of the use cases the team is working on currently are um the things like kind of guidance uh how can we guide you through your exploration um right especially if you don't have a clear kind of question in mind you want to start somewhere and then as you go through how can we guide you like really to explore the graph and get meaningful information out of it um some are let's see what else are we working on the storyline um uh kind of area there's a lot of um new kind of use cases new features that we have as well um especially around the topics basically topical feeds uh based on your interests so machine learning is uh you know might be your interest for instance or product management how can we um kind of really show you all the conversations happening around it all the content the rich like knowledge base that we have um for that topic and how can we kind of um like serve you the fresh content right like meaningful content and that sort of thing so that you would come back to it over and over again