 So I'm managing the analytics API and I look after the data pipeline, got to work on this over the years that we go. So the idea is an ELT-based pipeline and breaking away from the tradition. So the tradition isn't just about the tech part of here where you break away from an ETL-based pipeline to an ELT, it's also about how in your company you get people to adopt to it, like from a business perspective as well. It doesn't matter if you put in a pipeline and nobody wants to use it, like if you look like there's some explanation about ETL and ELT, like for an ETL model it's like you get your data, you take it from some place, you put it to some place else because you need to transform it, and then you take it and you put it up in some data storage and then from there probably some more transformation before you load it for your business to use. Isn't it? It's like when your programmer isn't working you ask him, he goes, like, my code is compiling. For data scientists when they're not working they're like my data is transforming. So it's more like the weight in between and the business users are mostly used to waiting because they like all the data is going to come in, it's going to transform, and when we get to see the whole data we're going to make it get insights out of it, extract knowledge out of it, all of that. But when you move towards an ELT-based pipeline so you get your data and you load it up. It's like leveraging a lot more. So now because of the cloud and the storages you don't have to worry about it. And you extract the data, you load it, and then you move it further towards your scientists or your data team or your business users and then they can do a lot more with the data coming in, get something, they can transform and they can do a lot. So one important thing between an ETL and ELT pipeline is when you do an ETL you don't really worry about what data is coming in, you extract it because then you copy it from one server to another, you do some transformation in between. So it's okay. But in an ELT your data needs to be consistent because what you extract is getting loaded. There is nothing happening in between. So if your data isn't structured, is it not consistent, whatever is going to get loaded is either useless or breaks. So we'll talk about how we make sure that our data is consistent between phases, how we avoid doing transformations, how we make data delays, vanish away. So this is more like a user journey that we have had over a long period of time. We'll walk you through what we, there was at we go and how we build up from that because this is supposed to be a step-by-step journey. You can go probably over the weekend, in a week, in a month, probably build a huge pipeline. But who's going to use it? Will your developers start pushing data to it? Will your data team going to start consuming data that's coming into it? Will the business be willing to really look at the streaming data? Or will they just wait the whole day before they actually look at the dashboards? So it's based a lot on how they adapt, how they change, and how your tech team itself adapts to that pipeline. So this is like before we had a pipeline, what we used to do was we had a server A, take data every six hours, put it on server B. And that happened four times a day before. Next day, the server B, we do the whole transformation, put it into the database. And then an application shows all our business users what's coming in. So it all worked out well because the business needed the numbers. The next day they're happy to see what happened yesterday, how many bookings happened, how many clicks happened, what happened, what not. So the first thing, for a few of the issues, one, if one of the transfer jobs fails during the day, we only know the next day what really happens. And we're keeping data on the server, the logs keep growing. There's a limited amount of space that you can keep. And you actually have to clean up after, you used to clean up after 30 days. That means whatever we have transformed and put into database, that's all we have. We lose everything that comes in that got extracted after 30 days. It's gone. We don't have any backups. So the first thing was, how can we save that data for longer periods? So really if we need our data team needed that data to do some analysis, well, let's say they wanted to find patterns over a year. We don't have that data. We already took that data, summarized it for business, and we only have that. We have lost a lot of information. The first thing was, because we wanted to store that as backups, we started leveraging the cloud. So we take that data instead of moving from server A to server B. We took the data from server A straight up to the cloud storage in AWS. So now we are keeping data in server. We don't have to worry about space. We don't have to worry about anything like flushing out, clearing the logs, anything, because we have historic data keep building up over the period. This was still happening every six hours. So we just moved it there. But it was like, we don't have to worry about losing data. At least if something goes wrong, and if the files got rotated or removed, we can still get data off. So we moved from server A to B by moving server to cloud. And then from the cloud, it would bring down. This is quite old. So it's not like something we did a couple of months ago, but this is pretty old. But at that time, things were very basic. So after moving data into the cloud, like we were comfortable using the cloud storage, we had a structure of how to save files in what format, what directory, so that they are accessible. The first thing was how to get it to the server faster. Right now, we are doing it every six hours, four jobs a day. So the next thing was, let's get the data up every hour. So how did we do that? So we started using, it's like a tool called Fluindy. I'm not sure how many have heard about Fluindy. Not many? Any show of hands? Nope. Log Stash, there you go, got a few hands. So it's Log Stash. So this is Fluindy. It's pretty much, they do the same job. They can follow the logs. You can follow files. You can capture data. You can push Fluindy. One was like, it's really light. It's written in Ruby, like Log Stash is based Java. So it's a bit heavy. This one got a huge library of plugins as well. So you can actually leverage that to do stuff. So we started off with adding Fluindy to the same server. But now we were actually getting the logs and actually passing those directly into the cloud every hour. We don't have to worry about the files rotation, running a job every six hours to copy a file off. We were actually started copying data directly into the cloud every hour. So we would buffer for an hour and then flush that data. So initially the data was, if you look at it, it's just one GB per day. That's a small and 1.2 million events. That's okay. So we started off because when you start something new and something the data is critical is obviously you won't just put things in production. And this was the first time we were actually capturing data. Before that, it was very limited kind of data that we do. So it started off, so before Fluindy, we would just upload database once the data was in. As I said, for the businesses, the data team needed it. They would just copy the tables from SQL into BigQuery. Our data team uses BigQuery. So everything we have to do, we need to make sure that all the data we have in our cloud, AWS, we move it to Google Cloud. That is into the BigQuery platform. So before that, the entire table used to get copied over. So we have to wait the whole day, the data to be transferred, the next day it's transformed, written to the DBs, and then there are actually the another job runs that copies the data. And we saw as the data kept growing, the task took longer and longer, and at times the BQ tasks, the upload task would run, but there wasn't a data because the transformation was still happening. So the transformation, because you have to manage the capacity, the data can grow unknowingly, you don't know when you're gonna get more traffic. So you can't really allocate capacity in advance. Either you over allocate or you under allocate. So that's one of the good things of using cloud is you don't have to worry about that. So with Fruendi in the picture, now we extract the data, just load it directly into the cloud S3. So now the thing is, how do we move from S3 into the BigQuery without doing another transformation? So one thing what we did is, we wrote our own service. Back then there were no, there were the, right now the Google Cloud Storage hasn't really nice service for transfer, so you can transfer data from S3 to GCS. I'll come to that as well now, how we're leveraging that. But at that time we wrote the service where we had to pull data from S3 and then transfer it into Google Cloud Storage. And then we run a task that uploads data into BQ. So you are now running data and uploading data every hour into the BQ. Like we get data, just as we fly data from our servers into S3, we can download it the moment and then we push it up into Google Cloud Storage and then that triggers a call into uploading into BQ. Why do we pull data and upload into BQ? We'll get to that as well. So the last point is quite important, where you say we use Protobuf to generate BQ schemas. This Protobuf allows us to have consistent format. As I explained that from extraction to load, you need to have your data consistent. So because we use Protobuf to generate schemas for BQ, we know the data that is to be generated has to be the same what is to be uploaded. So our schema is generated by Protobuf. So we use Protobuf because Protobuf got different libraries to support objects. So we use the same, we know we have generated a schema and the object needs to map to the same schema. So we use Protobuf's library to support, generate objects for Ruby when we extract the data for Java and the same objects actually map exactly into BQ as well. So in that sense, Protobuf allows us from extraction to load phase without doing the transformation and have that consistency that allows our team to know what kind of data they're gonna get and it's going to be consistent. So all that I tried explaining, I hope I explained it to some extent, this is how the pipeline ended up looking from that six hours job that used to run every four times a day, we build something which actually has all the services running. So after that, we actually build up a pipeline that went from one million events a day up to 100 million events, data points a day. So we were uploading 100 million data points a day every hour, 24 hours, 24 jobs per day that would go 24 jobs, that would pull data and upload into BQ. So from moving from businesses waiting for whole day just to get the data, now they were actually seeing that data every second hour. But there are some lessons that we learned while doing this pipeline and why we improved on this to move towards a stream-based pipeline. Some of the things are like the network connection between services, you do not own the network. It is always dubious, it will let you down when you don't expect, because you never know when you send a packet, will it get to the other point, will it get dropped in the middle, if you keep retrying, if some service goes down. So it's always dubious, you cannot never trust the network itself. Also, as the file size grows and if some issue happens the loss is bigger. Like even if you are flushing in an hour, you might get data peaks. If you have a peak at a certain hour, your files will be more and you have to flush more data. So the bigger the size, the bigger the loss. Also we saw something in Fluendi, like we all run servers, we all have applications, you get loads, your memory gets exhausted or maybe because you are, as I said, bigger the file comes in, your process starts using lot of RAM, you have something like a monit, it restarts your process. If the process restart isn't clean, you probably can have a lock of some zombie processes that run and then two processes try to read the same file and nothing gets read. We actually saw that, a lot of that happening because of memory issues, how the files grow large. You might think like the time is the only constraint, but space is also one of the constraints when you're looking towards streaming data out in batches or as it comes in. So yes, another process, business process re-engineering. You really have old data that's coming in, the way it is consumed and now suddenly changes the way it is. The business is looking at something different. They give them an opportunity as well to run some of their own transformation. They don't really have to wait for the entire process to finish before they could do. So if the data's coming in and they really want to just see or I run a query to see, okay, this are, where did we get the most searches from? Or for this are, where did we get the most visits from? Now they can actually do that. They don't have to wait the whole next day to do that because the data's already coming in. They can actually curate based on what their needs are. All right, so moving towards a second version of a pipeline from batch, we decided to move to stream because we wanted to move faster. As we have a saying, we go, when they go slow, we go fast. So we tried to go much faster. So we tried off with, the thing is nothing out of the box works. You have to build your own solution. You'll have to pick out stuff. You'll have to glue them together before you actually reach something. So what we started off with, because our service was, data team used to use all our service in Google Cloud, so we thought we'll use something in Google Cloud to put data all the way from AWS all the way to Google. So Google came out with Google Pub Sub at that time. So we thought, all right, let's try Google Pub Sub. And it didn't work well for us. A few of the reasons were, we didn't have any visibility over the data. Like the data is going through and we have to trust Google receives it and Google Pub Sub has a policy that at least once the message will be received if you push it, all right? So we don't really know what's going through if you want it to look like, how many events going through, what's happening. We didn't have any visibility back then. Similarly, we saw a lot of inconsistent behavior. Let's say we have, we create a topic in there. Suddenly that topic is off and we really didn't know how that happened. There was no explanation to it. Maybe because the product, though it wasn't in an L for a beta, it was used for available for public use but still had some limitations. Yeah, also it didn't have any option to load data into GCS. Something we do that for loading tasks into BQ. Because why do we do that? Going to discuss it at the end as well. So we didn't have any option to really load data into GCS. There was only a stream option available from Pub Sub so you can only stream data into BQ. So, it didn't really work well for us so we decided to try Kafka. How many have heard about Kafka or used it? All right, good enough, few people. So things, we did use Kafka for advantages that Kafka really gave us. Like it was complete visibility of data. We could actually see what Kafka was really getting because we had the log files for it. If you wanted to do any kind of logging, we had to tail the logs, account events, something. We have all of that available to do. Also, the thing was with Pub Sub when we transferred data over to Pub Sub, we can't compress that data. It has to go all the way as a regular object to Google Pub Sub. And when you're working with clouds, one thing that is a hidden cost is a data transfer cost. And you don't want to, plus, let's say if you were transferring 100 GB of data, a compressed version is 10 GB. So you don't want to transfer 100 GB, you better off transferring 10 GB of data. Not only do you save space, you save money, and it's more reliable to send 10 GB rather than 100 GB and flood the network. So for this, Kafka can receive compressed data. We don't have to worry about sending hundreds of GBs, if we can send a couple of GBs. Similarly, there are multiple consumers. For Google Pub Sub, at that time, you can only write a data proc job to pull data out of it, another Google service. So it's like a vendor lockdown. But for Kafka, there are a lot of different consumer services available that you can connect to do a lot, pull out data for your use case. Something we use is CCOR. It's a Pinterest-created service. It actually got plugins to upload data into S3, GCS, Azure, and few other services as well. And you can also use this to load data from Kafka into Hadoop cluster and stuff, if you use that. Also, we have acknowledgements on Kafka. We send the data point. Kafka can send an acknowledgement that I have received it. If you don't receive it, you can replay that event. Yeah, and also, you can deploy Kafka cluster anywhere. You don't have to worry about whether it's the AWS or Google Cloud or Azure. You can just deploy a cluster and you can start using it. So, yeah, putting that in perspective, once we get data from all our services, the way we connect to Kafka is in that every service goes and connects to Kafka. What we do is we collect data from all the servers into our relay servers, what we call. They're like series of servers. They also run fluently. We call them Tron. So the data all the way enters Tron and then goes out. So there's only one connection point from AWS all the way to GCS. And that is Tron. So we transfer data from one server, side of the cloud to the other side of the cloud. So a few of the advantages is like, for all the services that connect, they all come and can store data in one place. So we can buffer before we push all the data out. No, we don't, all the services don't start flooding our Kafka cluster. Yeah, all sound good, but that's like, when you do, it sounds all nice, but they're always something that hurdles that you don't see when you're putting something in. So a few of the things that we saw, like we had to do few PRs to solve a lot of issues. One was the connection between our Tron servers, the Fluendy servers and the Kafka was like every time the cluster, because the network is AWS, as I said, the connection would drop and then the Fluendy servers would not reconnect to the Kafka clusters. We actually had to do a patch for the Ruby Kafka gem to make sure that the cluster refreshes and reconnects and it actually got merged into the gem. So that's one of the contributions we did for this. Also, even for Seekor is a really good service. It came out of the box with all the plugins to upload data and everything, but it didn't have what we required. The file type, the file format, the directory structure that it uploads to. So we had to do few more PRs related to Seekor, contribute to Seekor to make things more flexible for us to work with. So it's always like whenever you have to work with solutions, there's nothing gonna work out of the box. You'll actually have to build things up. So another thing that we saw is like because we have a lot of flights and hotel searches, a lot of data comes in, a lot of fare, a lot of rates. We started seeing a lot of back pressure because there was so much data. Previously we were doing batches, so bigger size file, you would still push those out, but when you're streaming, you're streaming millions of events and then you have to think about buffering it and how and then flushing all those events over the network. So initially, some of the things that we saw was we had a buffer chunk size of 200 megs. We would actually buffer data up to up to 200 megs. Influently, it's time-based and size-based policy. So it's like whichever hits first, if within the time, your file size grows beyond the limit you've defined, it will create a new file. And if your time limit hits, it will flush that file. So we had something around 200 MB. Similarly, we were checking for the heartbeat interval, so all our services would actually check if the tron servers are still live before pushing. If they aren't live, they shouldn't flush the data, otherwise it will get lost over the network. So the interval was a bit too frequent and yeah, file settings were reading from the head. So if there are any reasons the server would restart or the process restart, as I said, when your server gets a lot of requests, your memory usage goes high. One other thing is your process is restart. So one of the settings were to read from head. So if a process restarts, it starts reading the file from the head. Another file on the server could be gigs. So now we were actually, let's say on the line 10,000, by the time the server restarts, it starts reading from line one to and going to 10,000. It'll probably take a lot more in memory before it actually even goes through. So yeah, and so we have multiple threads. Do the transfer because it's a single thread. If you have a lot of data, it really gets slow. Also yeah, keeping a flush interval smaller, you start flushing more data faster, your file size becomes smaller as well. So in that time, we saw like, because all the services talked to trans servers, we were seeing up to 240 GB of data buffered at peak, which had to go flushed out and would take around eight hours to flush that data out back to normalcy. So few of the things that we did was, don't read from head because you already have read what you have transferred. So when the process restarts, you just have to continue. You don't have to really go back and read the whole file. Similarly, buffer in smaller chunks. Smaller the chunks, the faster you can flush. And the lower loss you would actually have. Yeah, also have acknowledgments because you really want to know if the data has flushed, did you receive it? Did the cluster receive it or not? And with some of these smaller changes, we saw data down from 240 GB to 5 GB. And 5 GB of data probably gets flushed in 10, 15 minutes, if it is like at peak. Going from 240 to 5, just by some smaller changes in the design. So currently this is something what our pipeline looks like. I have better picture at the end, but this is what we are designing, brainstorming process. This is what it ended up like. So currently we have actually gone up to one billion events per day, streamed them at peak, as we have achieved. Yes, coming to the question, why data is not streamed directly into BQ? A lot of people say like, oh, you did all of that, you could have just started pushing data into BQ. Yes, that is one of the things that could be done, but pushing data into BQ is very expensive. And just think about pushing one billion events into BQ. That's gonna cost quite a hefty amount, but the good thing is you can upload data into BQ for free. That's like something people do, like usually people do it every hour or at the end of the day for big analysis. But as we discussed the service we did, we actually start pushing data every two minutes. So something we did used to pull data from S3 and put it into GCS and then trigger a call to upload data. We actually use the same service and actually run tasks every two minutes because we're streaming data all the way from our services, AWS into Kafka to GCS in near real time. We can actually run the task in every two minutes. The reason we run every two minutes and not every minute is based on the way cloud storage is made on the eventual consistency model. Like your file gets received, it gets timestamped for now, but when it is available, it's not instant. It's actually a small delay, which is not noticeable if you are using it in a regular sense, but if you're looking for real time, then that's not something that's gonna be available. So having that buffer of two minutes actually allows us to have all the data available when you want to load it. So this is something based on our most important dataset, which is the analytics dataset, which actually is the source of, in a way, income as well. So the thing is, when we start streaming data and we have quite around two dozen types of events and we have, on average, right now, we get like 6,000 different user agents, which means 6,000 different types of people actually use a service. So there's very hard, and then we have web, mobile web, iOS, Android, tablet apps, so everything's in place, there are different use cases, then we have different types of pages, different types of service domains, so there are quite a lot of different things happening, so it's nearly impossible to write test cases for every case covering what kind of data is coming in. So one of the things that we do is, on our staging server, because it's a complete replica of the production and all our internal testing and users use that services, mobile apps, everything, every data, that points that comes in, we actually run all the tests on that dataset. So every event gets tested in live, and we actually have connected all the way to Slack, so there's a channel, and if somebody wants to follow, they can actually search for that ID and start streaming data into another channel. So you actually see, there are two kinds of events, like three kinds of events, but this visits events, and you can see this event doesn't have the client session ID, so it gets across. So you actually can see, and we actually saw most of this happening with the, working with the front-end team, a lot of, because the rapid changes go in, a lot of new features come in, so they might have to send in more data, and the releases go in, the data, things that are going in, the page types, or the URL or something, the format isn't consistent, they're sending. So it was very hard, if you wait till actually the data gets uploaded all the way into production to really know if something's wrong. So this is when the data starts coming in, it helps our data team, while they are developing new features to really see the information that they are passing through, which is very, the most important dataset is valid or not. So yeah, so our question is, should we go stream or should we go batch? One solution won't fit everyone, and for us, we found out that stream and batch, both in combination, work well for us. So all the critical data, that is time sensitive, we actually stream. And everything that is non-critical, we actually can, our data team doesn't really need every minute, they won't probably need every hour, every six hours, anything, we use our batch pipeline to move that data every hour. And recently, we moved from using our own transfer utility from S3 into GCS. We have started using GCS's own transfer utility because it's matured over time, so we have started leveraging that, but we still use our own service to queue tasks into BQ every two minutes. So moving forward. Yeah, there's some fun facts about how fresh is the data. So the thing is, this is from one of the famous chicken chains. They batter their chickens every 10 minutes and they say that is fresh. So we forward data every two minutes, so we are much fresher than the chicken you eat. So that's one interesting thing. And yeah, because of the, we upload data every five to 10 minutes, we can do it at one minute, but only we do it at two minutes because of the eventual consistency model, because you need to make sure the data is available for your service to really upload, otherwise you'll end up missing it. We do run sanity checks every hour to make sure if something got written and wasn't available while uploading the task, it gets uploaded within that hour. But if we do one minute, we probably end up running sanity check and uploading data every hour anyways. Yeah, so this is like our current pipeline and how services are connected and what services forward and the cloud, different clouds connect and what, so yeah, this is the whole picture. And that is probably the end of my presentation, so any questions? All right, so yeah, the question is why do we use S3 and GCS? So because we started off initially in the AWS cloud, as I said, we used to upload data into S3 and pull it down and we actually have data into BQ as well. So we only move data to GCS because we want to move it into BQ. There's no other way around it. Another thing, good thing it does is it gives us backup in S3 before moving to GCS. Let's say we were to move data to GCS and it failed because it's in separate cloud. We actually have to reprocess it, but AWS and S3, they're under the same service. They're more stable and consistent. We know if we transfer data from here to S3, it is going to be consistently transferred and it gives us backup in a separate cloud. And if we need to use any of the AWS services for machine learning and any data services, we can use and we don't have to move data back from GCS. We actually can use it within S3. And for anything our data team wants to do, they don't have to come to S3. They actually have the data in GCS and they can use that or they can use it in the BQ. So because we don't lock down our devs or data scientists to use a single platform, what suits them, they can actually use. So they use Google Cloud and as our responsibility, we need to make sure whatever is captured is available to them as well. Rather than asking them to come and use tools they might not be comfortable with. So in that sense, S3 allows us backups and the advantage of using services on our end as well in AWS if we need to. Yes. If you have to redesign it today, I know you have a lot of this fun. All right, so the question is, if we were to design the system within one cloud, what would we do? So is it based on moving away or is it like you still have data moving into the other cloud? Yes. One thing is that we don't want to get locked down into a cloud. That's why we're using services like Fluindee, Kafka. Even if we were to move something, we'll actually move everything into the other cloud. It's not something we are using purely from a service provider perspective. Yes, we can. That's why, if we were to move, we can. It doesn't lock us down. So even if we were to do that tomorrow, we can. If you were to, let's say, our teams need to move to AWS tools, we can pull the Kafka cluster back into AWS and we don't have to really worry about it. That's why one thing is, as long as you're building with open source solutions, that gives you the freedom. It's not a vendor lock-in. So for your question, yes, obviously, because we have the advantage of using open source and not locking us completely down, we're using the platform more of it than the services. That's why it allows us to move across. That's mostly for our data team, yes. So they use the Google's platform for their stuff. So we actually have to make sure that we support them. All right, so the question is, what kind of data we are transferring and billions of friends? So we actually have logs, analytics events as well. So as TNShade, the types of events that we have, we have traffic logs. We have a lot of flights logs, hotel logs. We have analytics log, click stream. So the different divisions of data. So we have a backend team. They have a lot of data for their cell. They might be capturing a lot of request log, a lot of application logs. Similarly, data team has other things like microservice team has a lot of microservices. They might be logging a lot of data coming in related to different services. They might be talking to different services, recording the responses and stuff. For analytics API, we record all the kinds of events that we have with it, such as to actually know the kind of activity that's happening on our end. So we actually have different types of data. Could be logs, it could be requests, it could be user activity, it could be click stream. So any kind of data that can be used, we actually can capture it over the pipeline. Okay, the question is if this is containerized. No, we don't have containers. These mostly are all, this all is running on EC2. So, and most of these on Google Cloud services as well VMs, so they're mostly all, so we don't have containers in it. Yeah, it's like for containers, every container is a service on its own. That's one thing. So we have everything separated out. So there are different services and they all talk to each other. No, this Kafka is based for capturing data. For other queue, we do have Redis in our microservices. We use the Elastic Cache, AWS Elastic Cache and run Redis on it. Yeah. Yeah, up to. So one other thing is like, we were transferring up to one billion events from here to here, before we actually had to upgrade C core for because it needs to read data out and to upload, we had to upgrade. And since then we actually, because it always like at that point, we realized that, okay, for this size of cluster that we're running, which are reasonable, but if we need to scale up, we actually have to, if we have more data, we'll probably have to scale up more on the sizes as well. So we have moved some of the data, which is not critical, like big logs for flights and hotels back to our batch pipeline. That's when we added the GCS transfer utility. So we can actually, this pipeline can manage up to, let's say, billion events on the same size right now, when we don't have to upgrade. Like we can always, like for Kafka clusters, you have, I can add more brokers, up being up the size, or you can scale up the C core with more RAM and do stuff. But it's okay for us, if we don't, if we see that, we probably have to scale it up. But the size at which we have provisioned the stack, we can take up to a billion events per day streaming out. And the events, it's not just about the number of events, it's also about the size of the events. So we actually have some of the events, which are quite big, few actually are not just KBs, close to MB and stuff, so actually transfer a lot, a lot of GB of data. So yeah, that's, it's not just about the events, it's about the size of data as well that you transfer. It could have really small messages and you can have billions of messages. So it's more like a billion of messages, but we're probably transferring a lot of GBs of data. So this is supporting. The pipeline is like, usually it's at max around close to two people who are working on this. Like I started off on this, setting it up. So I've been lucky to work on this problem on my own. And then I had some support from one of our DevOps guy to scale this up. And recently Daniel has joined me in supporting this. And he did the, when we started, wanted to move back to the, re-initialize the batch pipeline. We decided to use the GCS tool. So he worked on that part. So at max, we usually have one to two people who are working on this and scaling this. If you have things in place, it's quite easy. Fine, thank you.