 Good morning. Good afternoon. Good evening. Hello. We'll just wait. Hello. We'll just wait a few more minutes to allow a few more people to join. Hello everyone. We'll just wait one or two more minutes to allow a few more people to join. I think we should start. Today we have two main items on the agenda. The Project Fine Yards team are going to follow up from the presentation that they gave us at the last meeting with a quick demo. And then we'll move on to further technical discussion of Rafael's DR document, which has been coming along nicely. So with that, I'll hand over to the Alibaba team for Project Fine Yards demo. Thank you, Alex. Hello everyone. My name is Ren Yu and I'm from Alibaba demo economy. Today, my colleague Andy and Tao and I are going to demonstrate our system as a follow-up of last presentation. Andy has a shared screen. One yard is an in-memory immutable data manager that provides out-of-the-box high-level abstraction and zero-coby in-memory sharing for distributed data in big data tasks. And if you want to know more technical details, you can refer to our repo. Also, you can look at the docs of the last big meeting. I will first introduce a big data task we are going to demo. And then I will hand the microphone to my colleague Andy and he will then go through all the running one yard combinations as well. The task that we focus on is a simplified version of what we do in Alibaba. It's for fraud detection. Currently, a fraud transaction indicates a customer deceptively purchased an item hoping to inflate the rating of the item. And in the demo example, we used pandas and Mars for distributed processing to prepare the data site and then use PyTorch to train a fraud transaction classifier. And we integrate one yard with these two systems. First, we can check the data, the code or data, yeah, data first. Sure. The data we use from a few series ways, typically one yard and the application built on top designed for handle large data. But today, for demo purposes, we just always use the small data just to speed up the process. But you can imagine those kind of data can live in HDFS or something. The schema of the data is like the follows. Like item is something like items listed in e-commerce websites like Amazon or Alibaba. It begins with an ID and then they follow by like several numeric features. And the user table is similar. Each entry indicates a user and it starts from ID and followed by a few features. And the transaction like indicating purchase from a user with an item. The first two columns are the user IDs and item IDs. And the third column is a label, like you to indicate whether this transaction we have labeled as a fraud or not. We use this label for training and followed by a few like transaction features. Like for the actual classification, we actually we join those two to three tables together to have a very wide lots of attributes, lots of features, a table for transactions. Let's look at the data. Sorry, that's a code. Like a simple like one single machine version is using pandas. We use vendors to load data from CSVs and load them as a data frame, as a pandas data frames. And they do us using a join like two to expand to get a new data frame and it's called a data set. And then they combine features to do this. And then we can go, can you scroll down, please? Yeah. And then we, sorry, okay. We use the make data sets to the function to like to convert those pandas data frame to num high and to be a pytorch tensor data set. And we import pytorch and it's just a simple like one layer for classification. Like it's to taking input, taking the data set as input. But what if those data sets are large? Like let's just imagine those sets which are too large to be handled on a single machine. We need to do the distributed processing. In this case, we use Mars. It's also a project from Alibaba. It's you can think it has a parallel or distributed num high pandas and the scikit, a scikit-learn package. Okay, let's go back to the code. The code itself, it looks pretty like similar to the version in pandas, you see. But there are a few notable differences. First, like the data frame here is the Mars data frame here is different from the pandas data frame. Here, the Mars will slice the actual data frame into multiple chunks and distribute the chunks into multiple nodes and machines. And you can see one yard provides the IO ability for Mars. Because like for one yard, it not only supports the local data IO, but also supports like IO from remote HDFS and various like data sources. And the actual joints like see the and you can select the yeah for the actual joints. It's quite similar to the pandas except that for Mars, it's actually like the job actually done in the chunk level. Like it will split those the joints of actual data frame into the joints of those chunks and reshuffle the data if necessary. And for this part, we only like taking care of the processing. In this script, we're only taking care of the pre-processing part. We just output a one yard object as a global data frame. We will like go back to the pytorch part later. I will next hand the microphone to Andy. Andy, you can you can take him over from here. Oh, thank you. And thanks for the code demo. Okay, now let's first check our environment. Here we have a Kubernetes cluster with eight nodes and there's no parts running. Now we install one yard with Helm. Then we check the CRDs. Here we have the global objects and the local objects CRDs from one yard. Then we run Mars now. Oh, okay. One yard, it's not running as a demon set on the Kubernetes cluster. Okay, now we can run Mars now. This process will take about one minute. It will join the three tables and produce a global data frame in one yard. Let's wait here. Okay, we get the global data frame with this object ID. Then we check the CRD. This is the global object CRD. Represent this global data frame. And also this global data frame is partitioned into chunks. And these chunks are distributed over two nodes. They are 192 and 193. Now we check the pods. We can see that the one yard pod on node 192 is this one. So we log in this one yard pod to get a chunk to see how exactly the chunk looks like. We first import one yard. Then we establish a client connecting to the IPC socket of one yard. Then we can get the chunk. We will choose a chunk lying on the 192 node. Okay, now we get the chunk. Actually, the chunk data is mapped from the one yard process to the Python process with shared memory in a zero copy fashion. And the chunk data is automatically resolved into a pandas data frame, since we already registered the pandas data frame redover to one yard. For the detail about memory mapping and the resolver registration mechanism, please refer to the one yard document. And now we try to get another chunk that is not located on 192. For example, this one, okay. Now we find that we can't get that chunk. This means that when we try to get the chunks from one yard, we can only get the local chunks. Now let's look at the PyTorch code. That will be the next step we are going to run. Okay, compared with the single machine version, the make data set from one yard is different here. Okay, here we have to first connect to the one IPC socket and get all the local chunks. And we can concatenate these chunks into a merged data frame. Then the rest part are just the same as the single machine version. Okay, so when we deploy the PyTorch pod on Kubernetes, what if the pods are scheduled to the nodes that doesn't have any chunk of data? Then the PyTorch can't get any data from one yard. So let's look at the state for set YAML of our PyTorch job. Hello, Tao, are you still there? Okay, I will... Sorry. Sorry. We can hear you. Just fine. I think Tao is offline, so he's not... Tao is the one actually typing those commands. So I will try to find the code. Okay. Oh, I see. This is what happens when you're extremely brave and do a demo in live life. Yeah, replica one demo. It's not a live demo if something doesn't go wrong, so don't worry. This is all good. Maybe we should wait a few minutes. Can I ask a question while we're waiting? Okay. Andy, can you describe your memory allocation, how that's going on and what kind of rules you're using and how the memory, that global memory is getting allocated and decided if there's enough to do the application and things like that? Here, currently we didn't have a very specific, very sophisticated strategy. And now, if the memory is not enough, when we create the object, we will return the the message to the client to tell them that the memory is not enough. So you had two servers there and you allocated during the allocation of the memory, if the two servers weren't enough, you would get the message to try to pull in more. Is that the use case or how you see it getting used? Yeah, let me share a bit more on this. Actually, with the help, we need to specify how much memory can OneYard use in the help, the command, and after OneYard is deployed and the memory consumed by OneYard is fixed. It's no larger than eight gigabytes in our demo case, but you can specify the launcher demon set of OneYard, like how much memory you want, what OneYard has. So we does not have a dynamic like an adjustment on the size of the the memory that OneYard can consume. And for the memory allocation, we use a very simple version of a single thread memory allocator. It's the same as Plasma. Hi, Paul. Are you back with us? Hi, Paul. Yeah, you can show us the spec, the stateful set. Yeah, Mo. Okay. So I just described the case that when the pod is not scheduled to the place that the chunks are located. In this case, we will add the OneYard migration to the init container. And this is a function provided by OneYard to migrate the data chunks when the chunks are not located at the same node as the pod. Okay, so now let's run the PyTorch job. And we give the object ID of the global data frame as the required object ID. Now we check the pod. We can see that these two PyTorch pods are scheduled to node 188 and 187. They are not the same node as the chunks are located. They are 192 and 193. So let's check the logs of the worker there. We can see that the data migrations are triggered here. And the data migration is actually time-consuming. Now we check the local objects. Okay, for this local object, it has been replicated here. So it is replicated from 193 to 188 so that the pod on node 188 can use this chunk. So here we can see that the data migration is both time and space consuming. So is there any way that we can reduce the data migration as much as possible? Here we try to use the schedule plug-in in Kubernetes to solve the problem. Okay, so we first reset the environment. And we will run mass to get a new global data frame. Adjust the pod ready. Hey, while we're waiting, just a quick question there. So effectively, you have a set of source data which then gets sharded across a number of nodes. And then you can start your workload that consumes that data on any node within the cluster and it will pull the relevant segments to the node where the job is running, right? Yeah, correct. Cool. I have a question about the design. Why did you decide to have an explicit step to pull the relevant shards as opposed to maybe just let the client in this case the Python application say I want this global shard and have the and the vineyard locate the needed, you know, the needed and do the migration on demand. Instead, you're doing an explicit, you know, any container step to do the shard migration or replication. Is there a reason? I can explain because like those kind of tensor need to be like the other case that's cancer need to be like used immediately after the process was launched. Just because the case we first we need to convert those like data frames to tensors and we need to get all the data frames, at least a chunk, the whole chunk to get the job started. We have a mechanism for pipelining like using streams like in my yard where we didn't demo today. In that way, we can do some pipelining like the streams are just chunk streams. You can like consume chunk after chunk and sequence of chunk. You can consume chunk at a time and we can organize that as a stream. So we can only migrate one chunk at a time in that case. So yeah, yeah, just the data in both cases, the data are directly mapped into the consumer's process. So that's the basic design paradigm. Yeah. I just a quick question and I'm learning from this is that would it be the data frames are on certain nodes? Is that correct? Yeah. Okay. And then as the application consumes them, the Python application, would it be a beneficiary to have the Python application hint in the pod specification what frames is looking for so that you could extend the Kubernetes scheduler to be out to tell the scheduler Kubernetes say place this pod on this node where its data frames are. So that way you don't have to move the data. That's exactly what we are going to demo. Oh, sweet. Yeah, exactly. Awesome. Thank you. I look forward to it then. Finally, please go ahead. Okay. Okay. Thanks. Okay. So this new global data frame it has chunks located at 192 and 193 as well. Okay. Now this time before we run PyTorch, we first install the scheduled plugin to Kubernetes. Okay. Yeah. That's a plugin. That's a schedule plugin. Luis just talked about. Yes. Yeah. Okay. Later we will inspect the logs from this scheduled plugin to see how we schedule the pod. Okay. Now let's go back to the state for state for set YAML here to let the scheduler knows which one object is required by the pod. We add this spec to the pod spec. Okay. So now we can run the PyTorch now. So actually when the scheduler is trying to schedule the pod, it will first get the one object ID from that spec. And then it, since we are using state for set, each pod will have a rank. Based on the rank and the object ID, the scheduler can understand which chunks are required by that pod. Then it will inspect the location info from the CRDs to know which node has that chunks. Then it will give the node the highest score. For example here, the node 193 and node 192 has the highest score. Okay. So let's go back to check the pods. Here we can see that these two PyTorch pods has been scheduled to 193 and 192. Let's look at the logs of the work there again. And there is no data migration happens. And the time is very small. Okay. Let's check the local objects. And there is no replicated local objects because there is no data migration happens. So here the data migration and the scheduled plugin is just one of the functionalities provided by one ad in Kubernetes. And we want to integrate the ability of Kubernetes to achieve advanced functions like checkpointing, photo recovery and so on. We hope we can build a new cognitive way of building and running big data applications on Kubernetes. Thanks. That's all for our demo. Thank you very much. This was really informative. It really helped me visualize a bit more how Vaniard works and some of the architecture because I think it's a complicated concept. Yeah, a little bit. Sorry. I missed the last presentation. Just a general question. How do you compare Vaniard with something like Spark on Kubernetes? Spark. Apache Spark, yes. You mean Apache Spark? Oh, yeah. It's quite different. Actually, the Vaniard interface is very low level. It's actually the shared memory. You can use all kinds of the language of your choice and the runtime, the execution model, your choice to build big data applications on top of Vaniard. You can use a data flow like Spark or you can use MPI jobs or you can open MP parallel computation or you can even use some single machine algorithms. You can choose everything. But for Spark, actually, it has a fixed programming model and a fixed communication model, a fixed running model. It's kind of restrictive. If we want to develop, for example, efficient graph computations or some efficient machine learning pipelines, sometimes Spark is not the most efficient way to implement that. So that's how Vaniard is different from Spark in our case. Just to summarize that, therefore, am I getting this right? Would it be fair to say that that Vaniard is almost like an in-memory kind of map reduce capability which can plug into multiple apps? Can it be accessing those same shared memory chunks in parallel when you have a multi-step pipeline for your analytics, for example? Yeah, that's correct. But Spark is exactly immersed to be an in-memory data store equivalent of MapReduce or Hadoop. But I think the bigger defense is really the flexibility of the framework, as you said, right? Yeah, the flexibility of what we aim for. For example, there are some HPC tasks using MPI. The application itself is an MPI job. It's very hard to migrate that job into like a Spark pipeline, like a category of DVM completely. It's very hard. So Vaniard basically doesn't provide any computation support. You choose your way of communication. You can use RDMA. You can use some magical tunnel. You can use the common TCP-IP socket. It's your call. It doesn't enforce any execution model, for example, if you want to run on GPU as fine. It's your call. So basically, Vaniard is just a memory storage engine like can provide a cross-process like memory sharing. It's not a computation framework. It's more like a in-memory data store. Yeah, that's true. Thank you. You're welcome. Did anybody else have any other questions for the team? No, thank you for the demo. It helped a lot. Clarify as Alex said over the presentation last week, where it would fit in terms of Kubernetes. So thank you. Okay, thank you. Thank you, Eric. We would like to submit our project as sandbox, as we mentioned in the last meeting. And we are wondering if there is any way we can have feedback from the Sieg storage community in some form. I'm not sure whether that's necessary, but yeah, we really want some feedback here. So it's, I think Amy's on the, or I'm not sure if Amy's still here. Actually, she might have dropped off. So strictly speaking, a Sieg recommendation isn't needed for the for the, for a sandbox submission. But what we can do is we can provide, you know, a little snippet of a comment to kind of say that you have presented and we think it's interesting. And it's worth going into sandbox and also provide a recording of the Sieg meeting, which you can use in your, in your sandbox submission, because the sandbox submission is just, is just effectively a form that you fill in and then the TOC review it at their next meeting. Okay, okay. Thank you. Thank you very much. Okay, thank you. Very cool. So thank you again for that. We really, really appreciate that. That was brilliant. Thank you. The, the next thing we had on the agenda was to, to follow up the discussion around the disaster recovery document that, that Raphael has been focusing on and contributing just so that everybody is on the same page. I will put the link into the, into the chat window. So we, we could, we could definitely do with more feedback and, and, you know, comments as we, as we continue to refine and, and, and bring more ideas into the documents. Raphael and I had a brief conversation yesterday and we, we discussed, we discussed a few topics which, which I just wanted to, to run through to see if, if there's sort of feedback on the scroll about things that we might want to include in the document. So, so the first thing was we, we had we had a discussion about documenting more clearly some of the, you know, some of the advantages of, of cloud native disaster recovery as a, as a, as a general concept in terms of, for example, having standardized versions of all of the software in, in, in the Kubernetes clusters by virtue of the fact that, you know, everything's containerized and standardized versions of deployments and configuration to reduce of, you know, YAML and, and being able to have declarative and, and composable application deployment and storage. And, you know, that's, that kind of leads to a number of advantages that we saw. So for example, you know, it dramatically simplifies testing failover and DR processes. It, it, it dramatically simplifies keeping multiple clusters in sync because, you know, they're all built on the same config. So, Raphael, did, have I captured that tool process? Was there anything else that we in relation to that? Yeah, no, you captured it. Well, I think one of the other advantages we were thinking of is the ability to autonomously decide that there is a disaster, right? That in contrast to more traditional disaster recovery where it's a human decision often. Right. Let's try to get that. Yeah, that's, that's a good point. I don't know if anybody else on the call who, who, you know, maybe that works on, you know, HA or DR with, with perhaps their customers or, or anything like that, whether, whether, you know, there were any other things that, that maybe we can, we can focus on in terms of the advantages of DR and cloud native. I can talk about it a little bit. So it's quite challenging to say, you know, what concerns and outage, especially in cloud environments, especially because in clouds, you know, there are different services, like load balancers, compute and outage means different things in each context, especially like a zone of outage, you know, or things like that. And there is really no good way of defining or detecting outages. So most like what we usually end up with is some arbitrary thresholds as far as if X percentage of nodes are down, we would consider like a zone of outage. And obviously semantics of outage also mean different things for, if you look, if you're looking at it from the, from the perspective of infrastructure versus application, look from applications point of view, look applications, look at CD or distributed stores, they do replication. And they obviously have, you know, different view of what outage means versus if you're trying to just look at it from the infrastructure's point of view in terms of, you know, whether it nodes in a given zone in a cluster or off or not. So I think based on how we're looking at things which layers we're exploring, we may arrive at different results. And, you know, how we can distinguish from like temporary outages, like network blips versus more sustained outages. I mean, I have more experience in GCP with all these problems, but I imagine like they're different in each cloud or, or, you know, I think what we're trying to do with this document is to provide guidance that abstracts from the underlying root cause or possible root cause of an outage and just give you a guidance on how you organize your stateful workloads to survive an outage, whatever that, whatever the root cause is. We want to be able to do it without having human intervention, like I was saying, without losing state or, you know, having zero RPO and having the lowest RTO as possible. That's what we're trying. That's where I'm trying to go with this document. I think it's still useful though to capture, you know, the things that are hard to do as well, right? Because on the one hand, you know, having a cloud native disaster recovery pattern defined is important and there are lots of pros, but obviously from an operational point of view, things like defining what constitutes an outage and what are the thresholds for failover and for, you know, the differences between the graded service versus an outright failure of his own are probably worth articulating, I imagine, because those are things that you will have to think about and you will have to define. Okay. Then maybe Ardan, I could use some help. You know, we could start. I was just thinking that's a few. I just got access to your document and I just, you know, scanned it very quickly. So I like the way you're, you know, describing the fundamentals of, you know, CAF theorem, things like that. Look, in very abstract ways, we're basically describing the problem, you know, obviously if you want to tolerate an outage, the data has to be replicated. If you do an outage or regional outage. So I mean, it all depends on like how abstract you want to keep the document or how generic you want to keep it. I haven't had a chance to read the whole thing in detail. So I don't know exactly what your goal is, larger goals, if it's just providing some basically some level setting for people to be familiar with the concepts or we're actually trying to come up with a set of guidelines so people can build solutions on top of them. It's the latter that you said, let's do these. If you're willing to contribute, why don't you take your time to read it and maybe drop comments and maybe we can also collaborate, you know, set up some meetings and collaborate together. Yeah, I'll be happy to comment on the document. Yeah. Yeah. The other thing I was going to ask for the team is, I realized this is a long document. Some of you have actually read it and provided feedback and I'm very thankful of that, but I think we need more before there is a consensus to go ahead and do something with it. So I was wondering, would it help if I created a deck with, you know, a summary on the main concepts of the document and then maybe I presented the deck in one of the next meetings? Well, a deck to summarize would always be extremely valuable and we can also, you know, use that content perhaps to, you know, create a blog in the future or a webinar or something like that because, you know, we have done webinars on the, through the CNCF as well in the past. So that sort of thing would definitely be useful, I'm sure. Okay, so I'll take that as a to-do as well as continue working on the document with the feedback that I was giving. So, Rofi, it's Aaron. I think it's good to give a little bit of history. Is this the same doc that has kind of been in progress for a while when I was still there? In that I think it's really hard for people who are adopting Kubernetes and not completely cloud deployment to understand the complexities of running it on prem and on cloud and taking our traditional storage concepts and trying to apply them to cloud native. And I think that's maybe the goal of this is to take what people are normally used to as far as their RTO and RPO in terms of storage and what that looks like if we run that in cloud and how we can achieve that capability into our one's point, like we have to have replication. I think just some perspective is also about what are the costs associated with that because that seems to be a concern when I talk to a lot of different people about deploying things this way, you know, that are at huge scale. What are the benefits and possible drawbacks? So that would be helpful if we also outline those for people. And Artalan, what do you mean by like, you know, being specific? Are you talking about creating use cases for each one of the clouds and how we know that they work or trying not to be specific? What was your comment there? Yeah, so I think it kind of depends on how we look at it. So for example, like, you know, I work at NetApp and in storage, people usually talk about like five, nine, six, nine, you know, of availability, like RTO of, you know, a few seconds or, you know, it's a very, the way we talk about our high availability is somewhat different than the cloud world, you know, where, you know, for example, if you look at the SLAs of AWS or GCP, you know, they usually talk about like 99.5 or 9.5 availability, you know, and if you're building a cloud native solution, unlike on-prem storage solutions where you can control the whole stack networking, it's a different kind of beast in the cloud world, right? So I think some of it is a cultural look. Basically, we have to talk about these terms a little differently. And that's also, I think, hard for storage consumers. But I think that that's something we should kind of like let others know that, you know, you can't make the same sort of guarantees that you can't make for the on-prem systems the way, you know, you can't do for VMAX or, you know, but also, you know, like, this is the first, I just got access to the Word doc today, just 30 minutes ago. So I don't, I don't know exactly what the document is about. But, you know, let's work together on that because I think you will find it was an interesting insight for me as I was doing this research in what I call the cloud native DR, which is defining this document, right? And you don't have to agree. But what I call cloud native DR disaster recovery, the features, the capabilities that enable disaster recovery don't come from storage. That could be a little surprising because historically storage has been essentially the storage and the storage team have been providing these capabilities for the entire enterprise, right? The inside here, the discovery for me was that these capabilities now come from networking. Storage has to be there, still has an important role. We still need backups for logical failures, but not for DR. We still, we probably don't need volume replication in this new world. Because the responsibility of keep the state in sync is to the application. Anyways, like volume or data replication becomes responsibility of the cloud provider, right? Because to some extent, it's not a start the problem, it's an application problem. So it would be responsibility of the database, it would be a responsibility of the queue or the cache, whatever middleware you're using. So that's where- Yeah, my point is it's not as critical as it is on-prem. Because of the way, for example, let's say AWS EBS does replication on their side, or GCP PDs they do, versus with on-prem, you're solely relying on the storage solution to do the replication. So that was a point of, but I agree, there is an application level. It's either on-prem or in the cloud, we don't rely on any of those capabilities. In the pattern that is explained in this document, we truly rely solely on the application. Sorry, Alexi. I just want to make an important point because I feel we're in danger of going off in a dangerous tangent. The point that we're trying to talk about here is not about the capabilities of individual platforms, it's more about the architectural patterns that would be implemented. And in fact, the architectural patterns that we're discussing in this document are not linked to any specific cloud provider or on-prem provider or anything like that. In fact, it's more about this is how you would engineer, this is about how you would engineer a system and cater for consistency and make sure that data is available in multiple places and then allow for the failovers across different Kubernetes clusters, et cetera. That's kind of the concept of what we're talking about here. We're not specifically saying at any point that you would do volume replication in a certain way or database replication in a certain way, but more around the point of saying you need load balancing capability and you need a middleware or a database layer that can do the replication, or you can have replication happening at the volume layer. But the point is, we're not talking about the specific platforms. In fact, this is, if you wish, a generic way of defining DR that can be used in any cloud-native environment. Right. Thank you, Alice. Yes. In fact, from the internal documents that Erin was referring to before, I stripped away everything that was Kubernetes related or OpenShift related. It's very, very generic now that the pattern should work anywhere, whether you use machine with the machines in the cloud, whether you use containers on-prem, wherever you are, the pattern, this pattern that we are presenting should work. Actually, I think it will work. And that not to say in this document, we are not saying that's the only way to do it. We are just saying, look, there is cloud enables this new way because everything is declarative and automated. But then you can also use all the traditional tools that you have for DR. And you see in the document that there are listed all the traditional approaches also. But we are trying to make the point, Alice, I'm trying to make the point that there is probably a new way that should be considered. I think it's ripe for discussion, to be perfectly honest. I'm part of the end user group now as well in the CNCS. And there is a lot of discussion around storage, and durability, and availability. And the consensus for most users of Kubernetes is that they can't. The storage is not, I don't think they use the word dependable, but they haven't had good success. So I think it's also one of the topics I brought up on the TOC yesterday is I'd like to have a little more focus from them as well as projects and ideas and how we take these things and socialize them a little bit better so that people can achieve the same things hopefully they can do on prem or in private cloud that they can do in cloud. So they have a consistent way of doing things. I think that's what we're trying to achieve is we can still rely on the storage regardless of how we're deploying it using these particular patterns. So is that what you're going for, Rafael? Alex, does that sound reasonable? Yeah, definitely. Okay. Would that team be interested in talking about these things? I'd love to talk about these things because I think I have definitely my old Red Hat perspective and now Apple's perspective and I think it's interesting to see the two melt together. So I'd love to hear the diversity of thought around this and what people are doing and customers are expecting. Okay. The reason I... Sorry, go on. Go ahead, Alex. No, no, no. So look, we're coming up to time. I just wanted to say, look, these are all good points but we should make sure that we contribute to the documents and if we need the ad paragraphs, add comments and we can then further discuss and merge them together and we can have separate follow-up goals if we want to discuss any particular points in a bit more detail, if we want to structure the content in a particular way, for example, I think we are at time. Rafael, was there anything else you wanted to add before we could go? No, I'm okay. Erin, I'll contact you privately. We're going to continue on that. Yeah, just hit me up on Slack. Thanks. Cool. Thank you, everyone. Thank you. Bye. Thank you. Bye.