 I think they're starting us off. OK. Hi, everyone. Thank you for coming. I am Karine Wallach. I lead the developer community at StarTree. I'm Ron. I'm a software engineer in StarTree. And we're going to be talking to you about real-time analytics, going beyond stream processing with Apache Pinot. So I'm just going to kick it off. I'll kind of just go over some of the foundational stuff. So just types of analytics use cases. So this is just kind of the foundation of what we're talking about in terms of real-time analytics. So some of you guys may be familiar with some of these use cases, but I'll dig into these anyway, just in case. So the first one, in terms of real-time analytics, is dashboards and BI tools. This is usually used for internal purposes. BI analysts are usually the ones analyzing and assessing this kind of data. Another type of analytics use case that you may or may not be familiar with is user-facing analytics. So user-facing analytics is basically when you are providing your end users or customers with their own real-time analytics. So instead of just offering it internally to your BI analysts, you can offer it to your end users. And this is like things like who viewed my profile and stuff like that. I'll dig into some of the use cases a little bit more in terms of user-facing analytics. And then the third one is machine learning. These are usually analytics that are fed into some kind of system that a machine does the processing. So this could be like anomaly detection or some kind of fraud detection that you have a system setup that analyzes this data and then may or may not involve some kind of human interaction in there, too. So overall, just kind of high level. These are like the three types of real-time analytics use cases. For this specific talk, we're going to dig a little deeper into user-facing analytics. So why should you care about user-facing analytics? So just the development of analytics and real-time analytics in general has kind of evolved businesses. So it started off kind of with operators utilizing the analytics. And then BI analysts is what we kind of showed before. And then organizations started enabling their users and their customers by providing them with real-time analytics. And then this business metamorphosis element here. This is when companies actually monetized the ability to give their end users real-time analytics. So you can pay for like premium features or there's like literally entire products that are just targeted to giving your end users or your customers analytics and capabilities when they have those analytics. So taking that one step further, I wanted to highlight something that's actionable insights. So this is something that's highly used inside of user-facing analytics applications. So what actionable insights are when you are building applications and you're providing your end users with these real-time analytics and real-time insights, giving them the opportunity to then take action upon getting those insights. So it can be something like the LinkedIn who viewed my profile thing, they see that once I'm gonna go, somebody viewed their profile, they might make a connection request or send a message or view their profile. So some kind of engagement there, which obviously is great for like user retention. And then it could also be things like even the news feed and things like that on LinkedIn, when you have the ability to like things as soon as you see them happening, this is when you have the ability, I'll dig a couple more use cases of actionable insights and how companies are utilizing this capability inside of their applications. Okay, so this is an example of the real-time analytics. So we're gonna be digging in a little bit into Apache Pino today, which was actually originally built at LinkedIn. So I think this is a really good example because this is one of the companies it's actually used in. So real-time analytics, things like who viewed my profile or who viewed your profile, you're basically giving your end users the ability to see who viewed their profile. And then they have the ability to slice and dice this data by location, companies, things like that. So there's a lot of dimensionality and very complex, there's a lot of complexity in here. One of the challenges being, I think there's actually more users than that. I think it's like 800 million or something like that. But you have all this fresh data constantly coming in and then you have a lot of dimensionality on this data and then you have to make it queryable to all these end users at the exact same time, which adds a lot of complexity. Another use case at LinkedIn is with a real-time analytics, which is, I mean the news feed. So what happens with the news feed is every time somebody clicks on the homepage, they're basically essentially querying this real-time database and they wanna know, I wanna know what's been liked and posted right now. And in some cases it goes through recommendations and things like that. So this is actually a pretty complex querying system when you have all these people clicking on their news feed. And then again, actionable insights, like what can the user do upon seeing these insights in this real-time analytics. Another use case is the talent insights and this is kind of what I was talking about the business metamorphosis, where organizations are literally transforming their business and creating products by offering their end users real-time analytics. Uber Eats is another really good example of a company that does user-facing analytics. So they have 500,000 plus restaurants that then have the ability to access their own analytics insights platforms with inaccurate order, missed orders, top-selling items, things like that. So one of the benefits in terms of the actionable insights that these restaurants have the ability to do is that if they see that all of a sudden there's a trend and review is going down, they need to know right now. And this is part of the optimization in terms of your user experience. I really like this example because it's not as clearly defined in terms of user-facing analytics, but this is kind of like the complexity of the queries. So Stripes use case around this technology is they have, I mean, it's a financial payment technology company and they have dozens of engineering teams and what they essentially have to do is they have these a bunch of engineering teams on one side that are constantly working with all this fresh data and they need to be able to take all this data, consolidate it into one view and then make it accessible to their accounting team, their finance team, fraud detection, data science, all kinds of things like that in real time, like literally sub-second querying. So this is actually quite complex. Also considering their financial data company, so there's a lot of sensitivity with what data is flowed through and all this kind of stuff, right? Okay, are you gonna take this? Yep, yeah, I can do that. I'm gonna hand this off to Ron. Yeah, so... Oh, is that a question? Is that a question? Go ahead, yes. He's gonna, he's gonna... Yeah, I'm going to go into that. It's like what the talk is gonna be all about. So we're gonna dig into it and at the end, if you have questions, we will be here to answer. That's good, yeah. So thanks, Karim. So let me reintroduce a little bit about myself. I'm a software engineer from Star Tree. So looking at some of the examples that Karim just mentioned, I just wanted to generalize some of the evolution that we saw in real-time analytics and how they evolved throughout the past couple of years. So what we are seeing is a trend is that when we are talking about real-time analytics, we are not talking about the exact same thing like five, six years ago. And these are the five very common trends that we saw that the online analytic processing system, or OLAP, if you will, that actually is evolving through with these trends. The first one is the internal-facing analytics are more and more going towards a new user persona, which is the end user. The second is that we are looking at more users actually ingesting semi-structure or even unstructured data into the databases instead of just asking the users to pre-process the data and then structuring them and then ingest them into the OLAP system. The third trend is that user are asking for more stronger and stronger data and query consistencies, which approximate data correctness is almost good enough in most of the old days use cases. And the fourth is the type of psycho semantics that our customers are actually currently using. It goes way beyond just slicing and dicing queries like filtering aggregations. Now we are talking about more of a complex event and analysis type of world. And last but not least is the amount of data is being ingested in the OLAP system. We are seeing it growing from maybe like tens of gigabytes going into the terabyte world. And now some of the largest cluster we see is going into the petabyte world. So let's go into each one of these points individually. One is the user-facing analytics. The real thing behind the scene about this is really the user persona change. I remember back in the old days when I first built my first real-time analytics use case was actually, it kind of like the one on the top left which is creating two of the data centers across the US continent. We're trying to do some data replications in between and we wanted to do a real-time monitoring regarding whether the data replications are actually happening in real-time, how late those replications are, et cetera, et cetera. Well, just keep me from being firing. But if you look at those use cases, it is more sort of like internal system monitoring via time series analysis, dashboarding charts, et cetera, et cetera. But now if you look at the use case that Karin just mentioned, Uber Eats restaurant, dashboards are used by close to a million users or million restaurants every day. And not to mention the LinkedIn user are used by over 800 million users actively using their profiles in LinkedIn. So the three takeaway I wanted to emphasize here is the type of query latency requirements, the ingestion freshness as well as the concurrency of the actual users being behind the scene making those real-time analytics queries. So switching from the old days where a second level query latency is probably enough, you're not going to monitor the dashboards all the time to kind of like you have to make sure that the latency is in the millisecond level, otherwise your user will have a bad experience. Now going back to the freshness, if you're having some users actually curing and interacting with the system, you want that interaction to be reflect on their user feed as soon as possible. Otherwise, if you're looking at millisecond latencies while you're actually posting, it's not immediately showing up on your fee as well, then it's going to be having some bad user experience. Not to mention the concurrency, now shifting from maybe hundreds to thousands of internal engineers to the couple of hundreds of millions of users across the globe. Next is the data structure, or semi-structured data that is being ingested. In the old days, I think engineers won't hesitate to slap on kind of like a pre-processing pipeline using Apache Fling or Apache Spark, depending on which system you're actually ingesting in and basically sanitize the data, structure them, getting rid of the things that is actually malformed, and conform them with the schema that OLAP system expects and then ingest the data in. Now, if you look at the current days, people just wanted to have this support out of the box. You want to just ingest the semi-structured data into OLAP and be done with it. One of the reasons is tightly coupled with the user-facing analytics trends, so more and more data scientists, machine learning practitioners are actually using these systems, and if you're asking them to set up yet another system that is outside of your OLAP to basically run the pipeline end-to-end, it's going to be really, really cumbersome, and then user don't actually want that type of experience when they are setting out the system itself. The other not to mention is the operational overhead, right? So if standing from the engineering standpoint, it is always easier to only maintain one system instead of two, it just makes debugging easier operational overhead less, and also you can guarantee the data correctness in an easier fashion. The third point is the strong data query consistency. So in the old days, OLAP had this impression for the users that they are ingesting data in an apparently fashion. You never change any data once they are ingested. This is sort of okay because of the best effort QE guarantee in most of the use cases in the old days that can tolerate. For example, if you're going to slice and dice some of the kind of like cross-nental, like data replication of delays, one or two bad data points or replications or duplicates is not going to affect too much. But nowadays, since we are actually doing user-facing analytics, we not only have to have append-only inserts, sometimes we need to mutate the data, sometimes the data becomes obsolete and we are wanting to delete them, and at the same time, we also still want the most accurate query result as soon as possible. The fourth point is the full SQL semantic support. So as I mentioned before, most of the traditional OLAP queries like aggregation, filter, group-by, order-by, or even most complex is like using some kind of complex UDF to transform the roles that you selected out. Nowadays, we are actually seeing users trying to do enrichment joins, nested queries, window functions, or even complex event analysis, like think about you have a kind of like sessionized stream of data they are wanting to correlate with, and these type of query really demands full SQL semantic support. And last but not least is the data size explosion. Back in the hundreds of gigabytes, maybe the low terabytes world, it is really easy to just co-locate your data with the compute. It's always easier to do that, the latency is slower, and you can just horizontally scale out your entire cluster. This is not the case if you're actually going into the petabyte world because it is just not cost efficient to have so many different nodes sitting around idle with the compute resources, just because you have to satisfy the local storage need. So what actually is a great solution here is to move some of the quote-unquote code data or less used data into the cloud storage, which the storage cost is cheap, so that you can actually still maintain a certain amount of level of the key relatedness, but in the same time, it's much, much more cost efficient. So long story short, what we actually summarized in the requirement change for the modern OLAP system is better ingestion, better query, and better scalability. So now let's go into Apache Pino a little bit and let me explain why Apache Pino is de facto platform to support all of these user-facing analytics. For those who is not familiar with Pino, Pino is a distributed columnar database that is built specifically for real-time analytics, including BI, data products, and abnormality detections. It can ingest data from various different kind of sources, including streaming like Kafka or a batch in Hadoop or other data lakes or data warehouses. It is used by over 100 companies nowadays. I think the largest clusters from Stripe, Uber, or LinkedIn are actually ingesting over one million message per second, while at the same time serving out 200,000 queries per second in width. Yeah, thanks pretty much. And then at the same time, it's still maintaining a P99 sub-second latency. So let's dive deep in under the hood and understand what exactly is Apache Pino under the hood. At the heart of it is the columnar data store. A traditional database concept table is being slice and dice into smaller Pino data segments, which is the atomic unit for us to manage replica, et cetera, et cetera, all of these type of flight data management operations. Data's are stored in columnar format, but more importantly, they can generate their own indexes and metadata's for each one of the segment individually so that it can perform cure optimizations in a blazing fast fashion. And then after that, the Pino segments are distributed into various different Pino servers across the cluster. On top of that, we have the control plane that is powered by Apache Helix to manage metadata's like which segment is residing on which server, how many replicas they are. And then last but not least is the Pino broker, which you can think of kind of like a stateless compute layer which accepts user queries, distributed them into the proper servers to do aggregations and computations, gather the intermediate result and then finally compute the final result and return it back to the user. So here are several things that Apache Pino's killer features makes it super fit for the user-facing analytics. First is it supports real-time ingestions, updates and deletes out of the box. It also directly supports semi-structured data ingestions. All of these data's are actually then structured into segment format so that it can support mid-second cure latencies, freshness-wise, it can support a second-level ingestion overhead so that you don't have to wait for the ingestion to complete. And then at the same time, it can serve out millions of users concurrently. In terms of curies, semantics features, all of the OLAP traditional curies plus some of the more complex ones such as joins already supported. And at the same time, we also out of the box support a tiered cloud storage plug-in so that we can move some of the co-datas automatically into the cloud. So let's dig into each one of these points individually, starting by looking at the curies performance. As I just mentioned, the Pinot broker is basically the distributor that basically slice and dice the curies accepted by the user and then distribute them into each one of the individual servers. So here, Pinot actually uses the segment-level metadata to prune out some of the servers that is guaranteed to not have the segments that satisfy the needs from the user query. What I mean by that is let's say, for example, by default, Pinot segments are distributed or partitioned based on timestamps. So all of the data that is being ingested into Pinot will be distributed by the different timestamps. And then this is actually a pretty traditional fashion for time series data. What happens is that users will pretty much not query the entire data spectrum of all of the ingested data, right? They are just going to go into a very, very specific size of the data in between the entire time spectrum. So what happened here is Broker can use the metadata to first prune out what are the servers that is guaranteed to not hold in the segment that is relevant and prune out those security dispatches in the beginning so that we don't have to wait for the server to come back with the results. And next, the server is doing pretty much the exact same thing, except in this case, it's doing it in a segment level rather than a server level. Afterwards, the surviving segments will be going through the future and aggregation of the positions in the segment level. Now, let's look at the bread and butter of the speed of Apache Pinot, which is the segment. So segments are designed specifically for low latency, high throughput aggregational queries. What I mean by that, you know what, let's look at one of the artificial data examples. So let's say we have one table with kind of like a country column, a browser column and some of the other columns. Like if you will, this mostly resembles kind of like an eyeball or user interaction with the website. And these data are structured in columnar basis. And let's just pretend that we wanted to have some query that does count aggregation on top of this table that filters out all the country originating from the US. So what happened for Pinot is, Pinot have this embedded inverted index that can actually do a reverse lookup from the country code to the document ID or the row ID, if you will, on where exactly are the rows that is going to consist of the country code US. With this, instead of actually having to scan through the entire data set, we just need to use this inverted lookup to look up exactly where the data is. And next is the start tree index. This is an index specifically designed to pre-aggregate some aggregational functions like count some mean max, et cetera, et cetera, that instead of actually having the system to process the data on the fly and pre-aggregate the data, what this system can do is, it can go into this start tree index tree and look for the specific country that has the US key in it and directly read out the results. So this would speed up a lot of the additional functions drastically. So here is kind of like a current version of Pinot's all supported indexes. We have the inverted, sorted, and range index pretty much designed to have the normal predicates like equal, inequality, et cetera, et cetera. Then we have the JSON text index specifically designed for semi-structured data. So we can ingest these data without actually having to structure them. And finally, we have the geo-index specifically designed to speed up like geo-spatial optimizations or like, for example, you wanted to know whether a specific lat-long point is within a specific geo-fence. And these are the things that will speed up those queries. Next, we have the aggregational optimizations. Some of these are pretty typical. For example, the hyperlot log is going to use to speed up the approximate distinct applications. And finally, we have the start tree index that sits on top of both category that provides pre-aggregational speed up. In fact, start tree index is so powerful we actually name our company after that. So let me take one step back and answer probably some of the questions you have is why do we have so many indexes? The reason why we have so many indexes is because we are specifically tearing these indexes to different type of user experience or use cases. For example, in Uber's use case, geo-fence indexes will be much more important than some of the others. So that's why we think Pinoe is the one-size-fits or platform to do all type of return analytics. The other point I want to point out is how easy it is to add more indexes towards the system. In fact, the indexing algorithm framework is so flexible. I think the range index and JSON index are actually added maybe within a couple of weeks just from kind of like an idea to an merge code into the main branch. So next, let's look at the semi-structured data ingestion. So as I mentioned before, it's kind of like a continuation of the indexing mechanism. For example, if we want to ingest some type of JSON data, they have sort of having a semi-structure, but you never know whether some of the JSON columns are going to be missing in the entire side of the JSON block, right? So what actually happened instead of actually asking the user to structure the data in the begin with, we're actually going to support a type of placate condition called JSON match. So what happens here is typically the filter expression that you will see in some of the SQL query languages is like if you want to have a top level, name filter equals to Adam, the address array filled with the street number equals to 112, or you have mix and match some of these using N or clauses. So these type of JSON match predicate is drastically speeding up the semi-structured data lookups. In this case, we are looking at 1,000 times or even not more type of like speedups in terms of dealing with semi-structured data. Next is the absurd ingestion. So as I mentioned before, absurd is a pretty common strategy for modern databases to ingest, especially if you're dealing with real-time streaming data. What happens is your real-time streaming data will never be guaranteed to be correct. Like sometimes you have incorrect data, corrupted data, so you want to correct them later. Or just by looking at some of the actual use cases that the data just natively needs to be updated. For example, in a sessionized data stream, you want it to always keep your roles up to date with the latest status rather than having the entire stream to be processed. So let's use this as an example, which is one of the Uber Eats order status update. What happens here is, actually, let's take one step back and try to explain one of the prerequisites. So one of the reasons why Pinoe is so powerful and so fast is because all of the segments, once they are generated, they are immutable. Because of these features, we don't have to keep a global centralized coordinator to keep track on which data is in where, what version is being used. But at the same time, it makes absurd pretty hard because you can't actually mutate the data once it's being generated. So a clever solution that we supply here is, we maintain an external primary index key map that maps, in this case, the audio ID as a primary key to where the latest segment and the dot ID it is to represent the latest result for that specific role. Now, let's say we have one more primary key added to the mutable segment, which is the one that is open to a set real-time ingestion. Let's say the primary key is BB. We previously saw that the BB is pointing at segment one dot ID one. Now it's no longer the case because we actually want the BB primary key to point to a segment two, which is mutable and still dot ID one. Now, next, if you have another column coming in, row coming in that basically have a primary key EE, because it's actually already pointing at the mutable segment, all you need to do is just modify that specific primary key index pair to point to the new document ID. Later, after a while, once the segment has enough data, they need to be sealed, then all you need to do is copy all of your primary key indexes from the mutable segment to the mutable seal segment, and then you can continue with your consumption later on. So as you can imagine, any time during this entire progress, if you have some query coming in and asking for, hey, what is the latest status for some of the primary keys, you just need to look at the primary key indexes and look out exactly where the segment is supposed to be for the latest status. So next is how we think Pinoe is evolving towards Fusico semantics. So one of the earlier examples that we have is lookup join. What this means is a very common use cases in real time analytics, which you have some kind of transactional data that is evolving through time, but you have a relatively static dimensional data that you want to enrich it from. In these type of use cases, usually you already know what type of dimension table you want it to be joined with. So instead of actually having the join, a Fusico join data shuffling operation in the fly, we actually predistributed these dimension tables. So this is where dimension predistributed table in Pinoe means. The data is already prereplicated and predistributed into every single Pinoe server. And if you have some type of enrichment join or lookup join request coming in, all you need to do is distribute those Pinoe queries into each one of the server that has the fact table. For example, in this case, the order satisfied. And all each server needs to do is doing a localized lookup because you already have the data predistributed. Once the enriched the result is sending back to the broker, it will be served out to the end user. Now you're probably already imagining what if the data is not pre-aggregated or predistributed or you don't know what dimension table you wanted to aggregate in the begin with. So one of the drawbacks for this approach obviously is that the other one is, let's say you have a really, really huge data being sent out from each one of these servers, then your final stage broker reduced becomes the bottleneck because you have to process so many data in just one machine. So we are actually having a new features in preview right now which is called a multi-stage query execution engine. This engine insert an intermediate computational stage in between the server and the broker so that we can actually do all of these intermediate stage reduces in a distributed fashion. What's more important is this structure can actually be replicated into multiple different stages so that we can actually support more full SQL semantics in the future. Last but not least, let's talk about the tier storage on cloud natively supported by Apache Pinot. So in the old days, as I mentioned, all of the Pinot segments will be collocated with all of the Pinot servers. You will see some of the server, some of the segments being curing more often than the other. This is just how the real-time analytics characteristics is being, right? The closer you are to the real-time site, the more those data will be used. And the long tail is always going to be exit. You're always having a really small fraction of the data is going to, the queries are going to go into some of these like older segments. So one solution for this is just to horizontally scale out the entire cluster. That would cost a lot of money because you are having a lot of the computational resources idle. Now, what Pinot cleverly did here is to decouple the storage and the compute by moving some of the code segments into the cloud storage. This way whenever the query is requiring older segments, the Pinot servers will bring these segments from the cloud storage into local and cache there. And then because of the caching effects for the least recent use cache, we are still able to serve relatively similar latency requirements by the same time we drastically reduce the cost for storing these code storage data. So to summarize the takeaway is, as Karim mentioned, the real-time analytic use cases have evolving to power more user-facing actionable analytics. And Apache Pinot is our solution for these user-facing real-time analytics because it has an excellent query performance. It can handle real-time data mutation on the fly without any problem. The ingestion can come from stream batch or any type of CQQ sources. It supports semi-structured data out of the box. It also supports hyper-architecture out of the box. And that's why it is one platform to build against all different kinds of user-facing analytics. Thank you. Is there one more slide? Oh, yeah, we have one more slide. So Star Street is actually hosting the very first real-time analytics in August in San Francisco. So if you're interested. It's also free for virtual attendees. So it's actually not virtual. It's on-demand that's available later because we're gonna clean it up and everything. So if you want, you can register for free and get on-demand access. Or you can come, one or the other. And that's it. You can go back one. Yeah, any questions? Thank you. Thank you. Is this database application playing the same space as MIMS SQL? If you know what MIMS SQL's a similar type of memory. I'm sorry. Does Apache Pinot live in the same space as the MIMS SQL application? Which is a memory-based... Actually, that is a good question. So let's compare this with MIMS SQL. For example, something that I'm more familiar with. So the idea is not actually trying to locate or co-locate all of the data in memory so that the space is really fast, but more sort of like trying to keep the most recent use data in memory. So the short answer is no, because we don't keep every single data in memory. But at the same time, we do have a lot of techniques, caching mechanisms, at least recent use, et cetera, et cetera, to basically making sure that the P99 queries, the most important queries that touches basis the most return data segments are all in memory. Like, empirically, yeah. Any other questions? We're only here for like five more seconds. Yeah, if you have any other questions that come to you later, feel free to join the Slack. You can get to the Slack by dev.startree.ai. And you can ask questions there. Ron is there. Yep. I'm there, but I probably can't help. But there's a lot of people who, from like, Startree and Pino. Oh, actually, you know what? Let's take one step back and answer the gentleman's question previously regarding how this fits into the ecosystem. I actually have to take that slides out because it's a little bit too long. Thank you, left. We left? Yeah. Okay. That's fine. He just walked home like right now. Yeah. I don't think I have the slides here. Yeah, let's find them. Okay. Okay. Any further questions or? Oh, yes, go ahead. She's coming with the microphone. Oh, yeah, okay. Making her run across the room. I kind of like did like a project like this, a former employer, but it kind of reminded me of Kafka and then Apache Spark. So you kind of like tie everything together with Pino? Yes, I think one of the trends that we saw here is that when we look beyond streaming analytics or batch analytics is where you want it to stitch these things together. This is what Apache Pino comes in because it is sort of like like London plus plus architecture where you can actually have co-located your real-time and batch data or fly data at the same time. And all of your queries going into these systems doesn't really have to worry about exactly where the real-time offline boundary is. So that's sort of one of the answers to that. The second is the ingestions and QE are happening simultaneously. So for example, if your data is ingested like one millisecond later, about a goal, then you can actually have the exact same data being served out at the next millisecond. Thank you. Any other questions? So I've actually used Apache Trino, which integrates with SuperSet for like visualizations and stuff. So I was sort of curious to know what kind of like visualization tools would be compatible with Pino. Would SuperSet be like an option? Because I would love to explore how that could be done. Yeah, I think SuperSet, I'm not 100% familiar with SuperSet, but I think as long as you actually conform with the JDBC standard, you can actually use any type of database behind the scene, where I think we do support SuperSet, give a take depending on what type of data you're actually trying to deal with there. Some of the other types are not by default supported by JDBC. But yes, we also have a Trino connector that basically you can actually act Pino as a Trino connector source. So it's kind of like Pino is acting kind of like a virtual data source that supplies data into Trino, or Presto, if you will. And then what happens behind the scene is Trino and Presto and all of these intermediate computational layers will actually push down all of the predicates, doing the SQL optimizations with the plugin that Pino supports, so that they can actually transport much, much less data instead of actually doing a table scan. You can just ship the data that Trino needs and then do the final aggregations on top of that. Oh, okay, so yeah, we do have like an instance of Trino. So it would be great if like I have some examples or maybe I can. Yeah, I think we do have some examples in front of you. I don't have that in this slide, but we can definitely connect. Yeah, sure, that would be great. Thank you so much. Okay, I think we can linger around for more questions if anyone have any more, but yeah. Thank you. Thank you guys.