 Hey, everyone. Thank you for coming to my talk, the AI edge and the storage walk into a Mongolian mind. Just a quick question, who of you is working on some edge compute? Most people. I was assuming that everyone's just here for Mongolian food and the weather. So TLDR, I ended up, I got hired actually to do some DevOps pipelines and for some reason I ended up in the desert in Mongolia just north of China in one of the biggest copper deposits in the world. I ended up building a hybrid cloud, cloud native edge AI earthquake detection system actually. One important takeaway actually to remember that I think some people in this community may have forgotten, I'm sure all of you know, is that runtime observability is absolutely amazing, but it doesn't prevent you from designing performance system. So performance-based design is actually a term I copied from structural engineering. It basically says you define your performance criteria and then you design your system based on that. About me, I learned programming and high school cracking software. So I learned a lot of low-level stuff before I actually did anything useful. Then I had my wonderful ASA laptop, which was so loud that everyone in the library in school hated me. Then I got a webcam and just nothing was working. Basically, that's how I got into open source. I found it to university CTAX, Masataka-san then grew it into, basically, we revived the CTAX project. Then if you do any kind of macOS virtualization in Linux, you probably run something I did. I personally, I think what we're supposed to build is like systems for domain experts to build good stuff without actually knowing anything about the system. There are three kinds of edge AI components that I would look at is one is the low-resource SOC thing with an MPU. The middle, which is, I guess, now much nicer to work with, which is commodity hardware. Think of it as a Steam Deck in terms of your industrial compute and some industrial compute casing. You get 16 to 32 gigs probably soon more, two and a half gigs per second, NVMe usually, and then what most people actually use as edge AI and talk about is the standard compute in either rack units or industrial casing. I'm just going to jump back and forth between some of these. But if you do the standard rack units and you have your Kubernetes cluster, you have a pure LB or a metal LB in front of it, and then it's like most things you generally do doesn't really matter. If you do have an SOC situation, then you need to think about, if you for example only have one gigabit per second, what are you replicating, how much of it are you replicating? Do you have a single node cluster where you replicate some things or nothing at all because the storage is actually outside of the cluster and the load balancing is outside of those compute units? Or do you think about using a storage system where you just mirror the whole thing, but then you still have the limitation in both CPU and other performance. So I'm not going to talk about industry IoT. I'd love to work on cube edge, but I've never had the opportunity. So, sorry. A little background, I didn't know anything about mining when I started. Basically, whenever you work in a mining environment, you produce hundreds of mini, sometimes thousands of mini earthquakes per day. You need to model the stress of the environment because you don't want people to die, you don't want to have them get injured. If that happens, you have to shut down production, you don't want to do that. Lawsuit's incoming on top of that. And then at the same time, you want to make sure that you never stop your production, you make as much money as possible, obviously. So how do you model the stress? Whenever a seismic event happens, two kinds of waves are generated. Waveforms generated, one of them is faster than the other. And then by measuring when they arrive at the different sensors, you can interlocate the event and by interlocating that and taking the travel times, you can basically create something like a tomography, like a CAT scan, but for the ground and the environment. That's all measured with a geophone. It's a magnet suspended in the spring. When the waves arrive, it just starts moving. There's a coil around it, it generates velocity. So a couple of problems. We generate about one megabit per second, terabyte a day, but the entire facility's interconnectivity was just 100 megabits. So you can already see you can't send it to the cloud. You want to make sure that you only send the stuff that you actually need somewhere else if it needs to be processed. I couldn't actually go physically there in the beginning. They have quite stringent immigration policies, I guess for European that grew up in Germany, I used to worry about my biometric data. I'd used to not get my ID with biometric information as long as I could. And then you go to Mongolia, you go to the office where you take your biometrics. They take a copy of your password. There's a staple there with a bunch of copies of passwords. You could just take them, nobody would notice. And then your biometric data is on the Janky PC. Anyone could take it, you wouldn't notice. And so you realize you spent your life chasing a dream of privacy that is just not respected elsewhere in the world. So anyway, we have storage limitations, we have compute limitations. We have the basic problem where I speak to an IT department and I don't know what compute they have. So I have to kind of simulate something that I build based on my assumptions and then I have two weeks to deploy it all in an environment that I don't really know. So what would we like to have in a normal environment? We'd like to have a Modbus, Ethernet, MQTSN. It goes to an industrial MQTSD gateway and then you send it to your time series thing on the cloud, which you can't do here. And then you just process your data and you build your standard environment around it. That's what you would like. So what do the scientists actually work with? They have these monstrous XML files, they have their own libraries to deal with them because it's not really humanly usable. And then they have their own time series storage format which again has some metadata that is actually stored in the other XML format as well. But what did I actually get here? I got this data acquisition system so with a propriety sensor network that goes to a proprietary data acquisition system and they don't provide an API, they don't provide the documentation. So the first thing I needed to do was get onto that system, reverse engineer the data format and then reverse engineer the acquisition format. Then I just built a simple Go connector and I pushed the whole thing to Kafka and then dumped it into timescale DB. Because quite a comment on Kafka, a lot of people use it, you probably don't need it, your consultant lied to you, the messages are probably too big, you don't wanna use it in this case after actually extracting the data, it worked quite well. Containerization is usually not a problem even if you have big data science kind of containers with TensorFlow and all sorts of other stuff in there I don't know, two gigs big, you don't really care about it. I mean, you should but you don't really need to care about it because when it goes to your artifact registry from the CI and GKE pulls it, it doesn't really take that much time. I mean, it takes some time but it's not gonna kill you if you don't pay attention to it. In this case, I had a hard drive and I had two weeks to go there and then for updates, there's like you wanna keep the software delta as small as possible. That means obviously that there's a problem with dependency and security there but if it is an agept environment, you have a different attack factor. So what did we look at in terms of storage? So I actually the initial, all the initial deployments, there's a mention on Ansible in the beginning were done with Ansible-based things. So I use Kube Spray, I noticed they have a standout there, they've like improved a lot of things over the years. I actually used Gluster in the beginning, it was a complete nightmare, I will say. Ceph actually nowadays with Rook is a delight to use even though it's not always very performant. These environments all have a data center so they all have a hyper-converged storage rack somewhere that you will not get access to, at least not in the beginning. Longhorn, if you guys know the call me maybe joke then from the distributed computing environment, you will notice that sometimes when the data is gone, it's gone and you can't recover it. I think the Longhorn consistency model is a little bit based on the fact that you want, they want you to have a mirror somewhere else, like you basically mirror it to another site which isn't really feasible in this environment. Minio, most of the times you can forget about it, one file per object is a complete performance killer if you do anything that has anything to do with, if you do anything that has anything to do with small files, lots of small files, Minio is going to kill you. A lot of appliances, Trunas included, actually do use Minio for the S3 layer. C-Wheat FS doesn't have that problem. I just mentioned it there, I will talk about it a little bit later. Okay, so I used a bunch of different pipeline tools. I use Packyderm, I did an environment in Spark, I did an environment in Airflow, I did an environment in a bunch of other ones that I don't actually remember right now, and one of the things that most of these have in common when they deal with passing data around is actually to have immutable artifacts at the end of each step. The idea is that when you have a problem, you basically, you get to go to every step, like redo the next step and like figure out what you wanna do there. That's nice, except it may not fit your purpose and the premise that every of these pipelines have that it will just magically solve your chaining or functions is not necessarily accurate. Powered a little bit through the slides without paying attention to time. So what you wanna do, what we used to do in the past is you profile your code. So like especially in an environment where you're limited in compute, you need to profile your code. Like, EVPF is awesome. We have all these observability talks, we have a bunch of EVPF solutions, but they're not going to excuse bad design. They're not going to solve, magically solve the problems that you created before you actually deployed your stuff in that environment. So I just, I don't know how many of you used to do database stuff and a few decades ago, but at some point, every SAS platform started running into database problems because their ORMs used to serialize and deserialize huge amounts of junk. Yeah, so it turns out in this particular instance, with the background that I told you about the storage formats that they have, a lot of IO actually is what the entire compute time is spent on when you throw it into these pipeline systems. Okay, so the consistency model here, if you think about it is we have these windows of time where an event occurs, when an event of importance occurs, or of not important. So a blast, something else, you need to be able to gather all the sensor data from that particular window and correlate it to each other. Once you have correlated it and stored it somewhere, you don't necessarily need it. It's nice if you can store it. In the event that you can store it somewhere, you want to store it somewhere, but in the event that you cannot, you don't have the bandwidth or the capacity, you can discard all the other data once you've located these events. And obviously then the question, like you have to pay a little bit attention to how much detail, like how, what is the word, like how, what your tolerance on that event is. Obviously, if you have a lot of space, you can store events that are probably noise, but you might want to get something out of later. For the jobs, I decided, in this case, actually instead of, after I evaluated all of these things, to just go for a Redis queue. And I don't really care if the jobs, like if the Redis dies, which it doesn't generally, but if it dies, the job can be rescheduled. There's no real necessity there for consistency. And the same goes for the artifacts that are temporary in between. So if those disappear, we just regenerate them, because after improving the whole thing, the goal is to have a reproducible artifact from end to finish. And as long as it's reproducible, and it computes in the reasonable amount of time, we can just rerun it. We don't care if something happens in between. You need to make sure, obviously, that your time scale, hyper table, like, discards all data. You have to make sure that your Kafka discards all data. And yeah, that's it. So quick recap. This is actually a fairly simple architecture if you think about it. So I use the Kafka for stream ingestion, which we don't need if we actually had access directly to the sensor data, because we could stream it directly into an MQTT ingesture. And then the Kafka Streams connector like dumps it into the time scale. And then the pipeline framework, which is a domain specific framework that I wrote, allows the scientists to just like hammer away on that data. The thing that's important is, I don't know how many of you have worked with scientists that write Jupyter notebooks? Yeah, okay. So you know that when they write Jupyter notebooks, it's not necessarily something you can use. And if you don't force them into a structure, then you get a notebook where you're gonna spend a lot of time trying to get this into some sort of production state. So what you wanna do is actually build them a framework where they have their building blocks, all the building blocks that they want. And if they then use, if they then use their Jupyter notebook to do whatever they want, you can still use that code afterwards because it has like a base structure. It's like, it's kind of the web framework of the scientists. A few things I really like, I have been trying to integrate and move into, but I have not finished yet, is I quite like both NATs and seaweed FS. Seaweed FS has an important caveat that you need to consider, which is that when the amount of small fires that you have grow, so as the amount of fires that you have in that fire system grows, the memory that it uses scales in relation to that. So you have to make sure that you kind of understand what the scope of your data is, how much data you're generating. In this particular instance, when we generate those travel times, there is a problem because it's a really old code base. I've been trying to migrate some of it to Rust. It's what scientists use and it just generates a lot of small fires. And the both of these projects, if you look at the GitHub issues, you will see that they understand distributed systems. They pay attention to it in their design. They will not audit it, so there is a possibility that it will still like, you know, but at least they're trying and at least it's part of their architecture. NAT says something that Redis doesn't have, for example, which is auto reconnect logic in any queuing solution that you have with Redis, that logic is usually in some sort of library. Here it's part of the architecture. JetStream is something I thought might be really nice for storing those artifacts, those temporary artifacts in between for the pipelines and KubeFS is actually a very interesting project from OPPO, which they actually specifically designed to address some of the issues they had with CIF. And it's part of their design actually specifically addresses storing small fires. So for the model training, the scientists knew the access to useful data. So there's like a couple of problems here. Enterprises don't want to share data, but they should. If they don't share it, then they're just wasting everyone's time. But when they do it, how do you share it well? So you don't really want to have an API where you're constantly pulling data. You don't want to share the entire blob storage with the scientists. You kind of want to have them access the streams as something transparent in their code. So TileDB is an interesting, in my opinion, concept which follows a little bit the design of InfluxDB 3 for the way they store data. The performance very much depends on how you structure the streams that you store, the time series streams. And it's basically an array store designed actually for scientists. And then you can mirror a partial, like a part of that TileDB to the scientists environment. So this, by the way, is a lamb head. A head in Mongolia. It was delicious. I guess the lesson you can take here is that in Mongolia, they respect the animals. They feed them well. They raise them well. And then they use every part of it. And I guess what I'm trying to say is treat your Kubernetes well, and then it will be a treat to you. So a quick recap. So like every pipeline system you look at will tell you this is the one that solves all your problems. And it might if it's actually addressing your problem. So you kind of need to know what problem you're trying to solve. The messaging bus is the same thing. So I took the example of Kafka. This has been iterated over and over. You kind of need to know what kind of packages you send. You need to know what you want to send, how big it is, how much of it goes through. And then that's when you choose whatever solution you take. In such an environment, especially if you want to design it for the SOC kind of size, then you actually really need to think about how many replicas you have, how much data replication you have. It affects your bandwidth available. It affects your CPU time. It affects your memory. All of these things are part of it. If you do take runtime observability and you add it on top of it, it's going to affect your memory. Sorry, yeah, it's going to affect your memory. It's going to affect your CPU. I tested a bunch of pipelines, a bunch of Kubernetes distributions. Some of them were not exactly a delight. Some storage solutions are not something you should use. And I will tell most people should not start with a cloud-native storage solution. Unless you have to, unless you know exactly what you're doing, don't use it. By a trueness, it costs about as much as a cloud-native solution architect. And you have it for a while, and then you can build your actual software-defined storage if you need to. Again, Cloud Iops, I didn't really mention it here. It's a bit related to the fact that I simulated everything on the cloud. And then turns out those Iops numbers you see on Azure website don't really apply until you do other things or you provision a lot of data, and then suddenly it gets very expensive. So these are all things to consider. But I guess the summary is I basically got hired to just containerize some code, some scientific code, and ended up saving some lives in the Mongolian mind. So I'd love to work more on this. I'd love to talk about these things if someone has any suggestions, if anyone has ideas on how to do things better and how to turn this into something more. Come talk to me. I actually have pineapple cake from Taiwan that I brought with me. So there's a coffee break in a moment. If you want some pineapple cake and talk, you can find me. Any questions? No questions? Well, the food was great. The desert is actually not as cold as you think it is. Thanks, everyone. Just one question. When you use the, say, for whatever storage you tried there, you use RWA export? RWA. You needed those files to be accessed by multiple consumers? So yes, one of the biggest issues is that I mentioned the small fires a lot. I didn't actually mention why. So those travel timetables, you need to basically load into, either load into memory or load them really quickly. If you use a normal one of these, so for example, if you use Minio, it's way too slow. You can't load it into memory at any kind of reasonable speed. So basically, if you cannot read it into memory fast enough, you cannot actually scale your functions. So you have to make sure that they're running constantly. And then the issue there is that all of these functions access the same thing, because you need to generate it. So you need to regenerate the velocity model. And so you also don't want to regenerate this velocity model all the time. And again, you have the same problem where when your storage is low and you regenerate this, it's just going to take a lot of time which you actually want to use to process the data. Thanks. Thanks, everyone.