 Thanks. Yeah, so hi everyone, I'm Asian. Today I'll be talking about how we use search kit at Note Flare to build some of the search features. So to give a quick introduction, I think you can reach out to me on those platforms. And I'm also a profile of Note Flare occasionally I write technical articles. So I think to give an introduction, so anyone here thought of Note Flare? Yeah, good. So we are still at a tech career support app where every month we help like more than 75,000 tech artists go their career. So what they do is that they come to our platform to look for jobs, verified salaries inside blogs and a lot of other stuff. So when we first start, when we a year ago, we wanted to build this feature on the home page right, which let us search across multiple models. So like your company, your salaries and jobs, everything with one search bar. So that was one of the problems that we are trying to solve. And because we want it to be responsive, we kind of have a requirement that it must go within a few milliseconds. And at the same time, we also want it to be like cost efficient, right? We don't want it to be like spending $100 plus dollars on AWS to be able to build this. And also other like some bonuses that we aim to have. So for example, let's say we want to do recommendation, but I think there's also some data in portion like if React.js should be the same as React.js, and like some company like Java, space script should be the same as Java script. Because a lot of this information are from companies. So we have to account for all these age cases. And a huge bonus is that if it works with our existing current implementation, right? So we are using Ruby on AWS and your typical active record and Postgres for the database. So these are the requirements that we have when we try to look out for like what we are trying to build here. So I think before we actually build this, we have this setup where like the front end can call API to the server and then the server can fetch the data from Postgres. But I think we realize that it is not fast enough and there's some limitations to how easy it is to configure the search and results. So technically you can use this Postgres, but realize that it is too painful and it might not be the best tool. So we kind of thought about maybe we try to look out for other database that supports this and one of them is elastic search. And when doing the research some came across this gem called search pit, where it's sort of an elastic search wrapper for your Ruby on Rails. So before I dive into that, I can have some disclaimer, right? So this talk, I tried to keep it simple and brief. So I'll be talking about how to implement some of the simple search features and how you can configure and the different options that comes previous with the gem. But what I will not be talking about is why we didn't choose the opportunity and how to set up elastic search and how to optimize and scale enough. So I think those are not so tall for this talk. And because we built this a year ago, a year back, right? So I think the question that we have is that, have we made the right decisions? And so far the decision is yes, based on the situation right now, but it might be known in the future. But as of now for the past one year, it has worked pretty well for us. So I think that's good. Okay, so back to it, right? So how does elastic search actually work in this case is that it's as though you have another database. So how we use it is that we index some data that we want to do a very efficient search from the Postgres database to the elastic search database. So the reason why we don't just index all the data is that we don't actually need all the information. Like, there are a lot of models, but maybe we only need two of them. So if we index everything, it will be unnecessary calls and stuff. So we only index what we need. So the very basic update is that when you start, it's just including the search key and elastic search gem in your gem file, and then like this bundle installing it. And let's say you want to use this search across the model like job listing in our case, you just have to add the keyword search key to the model. So what happens next is that we can run the command like job listing dot way index, right? And then you will index everything guess by default so I can touch on data. And when you have to retrieve the information, it's very similar to like your active record where you just do a find buy or a where and then you look through the results. So it's very simple. Actually, you can set up everything with less than 10 lines of code for the very basic stuff. So how does it work behind the scenes? So for those that are familiar with like active record, sort of like a wrapper for your like relational database, like you write like something dot where what it does is directly becomes like S3R query. So in this case, it's very similar. So if we get an example, I type something like job listing dot search Java, what it actually goes behind the scenes that it converts it into a query for elastic search, then it will cause the endpoint. So a lot of the things are extracted away for simplicity. So some of the query options that that are very useful. And I'll say covers quite a bit of the use cases, the simple one, like which field you want to do search for. So for example, maybe a job listing as a title, a description, and a lot of things, but maybe we only want to search through the description. So we can just indicate which field we search through. And you also can indicate like some question, right, only search type, if it's an intern or junior, like very straightforward things you can refer to a documentation, but it's like since that is very similar to your active record. So in some cases, we realize that we might want to boost the scoring for certain fields. So for example, if I want to search for API engineers, I will want a job listing with API engineer in the title to rank higher than one API engineer in the description because it's more likely a more accurate result. So it's similar to when you are googling something like the title of the page is a lot more important than the H2, H3 and the content. So in this case, not sure whether y'all can see. So we remain doing this. So yeah, so we can use the full spine field for that. So in the case of title, we want to have like 10 times the scoring if the terms Java appears here. Yeah, and maybe let's say we want to like boost certain companies like maybe they are the more popular companies. We can also indicate, okay, this company ID, if the company ID is like one, make it 10 times rank 10 times higher or something along that and we can even boost by recency. So for example, job posting, you might want to prioritize and show the users like those that are like more like posted this week compared to like two months ago. So there are some of the fields where you can boost your results. And lastly, and the next one is pagination. I think this part, this is very similar to some of the pagination brand that are very common like terminally and like world pagination. And it works with those two. So yeah, it works very well with those. So you don't have to configure anything as well. All right, so the default match right is that when you search for something, you will match the entire work. So for example, if you type Java, Java script comes up, but there might be instances where you determine that you want, as long as the stuffing of the word match in match, so you can use the word start to indicate that. So in those case, Java script also sums up. So there are a list of options that you can use to determine what are the next three options. And this is something that I quite like particularly as you can also do something for the seniority search where you can just indicate that, okay, I want to find job listing. Once you have a record, I want to find similar listings based on the different fields. So this is an example that we kind of add right, I do a screenshot of what we've got where a job posting slide, we now have the similar jobs. And those are mesh. In our case, we use things like head stack and seniority law is to connect the similar jobs. Yeah, so it works like out of the box, especially if you don't want to customize a lot of things. Yeah, and you don't have to add in another gem or build your own recommendation from scratch. Okay, so the next part is that when you query something from like elastic search this way, what it does is that actually it gets the ID of all the records from elastic search, then you will fetch those records from your posts left in our case to get the other information. But what this means is that it's a two request, right? So that if you want, you can call the load false keywords. So you will take all the information from the elastic search server. And the process is that you save online one request and like it works independently from your post address database. But the cost is also that you must make sure that all the information that you need in this space are stored in the elastic, so it might incur more cost. And because elastic search is sort of a copy of your main database, right? What you might be catching is might be potentially a steal like backcourt. So those are the cons by doing so. So there are also cases, I think you will see, I was sharing that by default, you will index every single clear in a model. So let's say your job posting have like salary, by default, you will index everything. But sometimes maybe you don't need all of it. So you can indicate which are the ones that you want to index. So this could save you some space and cost. In addition, if let's say maybe a job listing have some like associations like headstats and everything. So you might want to search through a headstats. So in this case, you can indicate that look in elastic search, please help me to index those fields also. So this will make searching easier when you do that. Okay, so back to what we are trying to do, right? Because we are trying to search across the different like models like jobs, salaries, companies, blog posts. This need like four or five models. We didn't want to call like four of API just for one search code. So very nicely, such case I have this thing called a mild to search where you can just indicate like what are the different things that you want to bundle into one query and then you will patch them. And then the results will be one, one Facebook. I think I'm not sure what's the result, but it's one, one object that's written. And then you can search through that. Yeah. So in that case, especially used for us, like you're saving like four or five, like request or search. So there are other options that we didn't quite use a lot, but I think will be quite interesting in this case. And one of these highlights. So you know, sometimes when we do for something, some of the terms are like four, based on what you are searching. So highlights essentially does the same. And it will return you the query, the results with the HPML included in it. So you can just display it easier. So the difference is highlighted, but you can also configure it to be like four or like other form formatic. And it also comes with like Stanley, which is a concept for like, search engine where you try to clean up many, the fears based on like the item. I'm not quite sure that's what you thought. Yeah, I think it's quite straightforward. Yeah. And lastly, there's a editing show that is quite nice. So because when you search something, you will tell you that, oh, under these attributes, you have how many of these items. So let's say you're getting an e-commerce store or search engine where you want to display how many search results per field. It actually comes to that. So that's about the features that, like some of the core features of search kit. So I will talk about some issues that we face when we're building it, that we sort of get burned. So let's say a model has associations from another model, when the second model is updated, it does not automatically re-invest the first model. So what this means, let's say a job listing has multiple text stats. And if you update a text that it does not automatically re-invest that in the job display. So this is something to take note, you have to write a method to sort of do an all-back method to get the job listing to re-invest. So yeah, it's this also not very clear in the documentation, but we have to figure out the hard way. Second is that we are determining how to re-invest, how often to re-invest, because essentially your elastic search is a copy of your Postgres. It's a similar concept to your cache. You don't want to keep updating it because it's expensive, but you also don't want to update like full details such that the details go still. So it really depends on your use cases. So for our case where we have like salaries or jobs, and the salaries is based on like user submissions and job posting, we realize that using a refresh rate of like five minutes will do the job. So we switch to that mistake. And lastly is that when we first started like we were index re-indexing the whole like database, right, and seems to be working fine. But as we go across the months, we start to realize that some of the re-indexing are failing because they are sending such a big like request to the elastic search server and our elastic search server that instance can't take that like request. So a lot of the re-indexed like jobs were failing. So one of the way that the outcome is that, okay, instead of like, let's say you got a hundred jobs, instead of like indexing the all hundred at once, we speed it into like maybe a bundle of, and so one to 10, 11 to 20, and then re-indexed by batches. So that will reduce the load on your server. So yeah. So yeah, I think from the end of my presentation, thanks a lot. Yeah, I hope I hope it's been useful. Yeah. So I think now is the Q&A time.