 Thank you all for making it to this talk. So this talk is Kubernetes bees, what bees and Kubernetes have in common and how bees have solved many other problems that Kubernetes solves and that we have in distributed systems except 100 million years ago. So this talk was supposed to be together with Simon N, a colleague and friend of mine who unfortunately was not able to make it. So you'll have to make do with me. My name is Chris, working at Gipod. So few disclaimers to begin with. First, much like there is more than one way to build a distributed system and much like there is more than one way to run Kubernetes, there is more than one way to do beekeeping. And the views presented in here are a very Eurocentric view of this practice. There is a strong cultural element to beekeeping and we do not represent the entirety of all practices that exist. We'll also very specifically talk about the kind of bees that we'll find in these areas, so specifically the north of Germany and the middle of the UK. The purpose of this talk is to leave you with an increased appreciation for bees, their complexity and the analogy of problems that they face and that we face. And with that, let's get right into it. The things that we're gonna talk about is predominantly bees, specifically apis melefera, which is a type of honey bee, a cultivated type of honey bee, so we're not talking about bumblebees or anything of that sort. Bees in our analogy here are the workloads. It is the workloads and the people who work with things. So there are three main types of bees. One is the worker bee, and we're gonna talk about that first, then there are drones, and there's what's known as the queen. The worker bee are a strange combination of what in Kubernetes you would consider a pod and people who operate those pods. And they go through a series of steps along their career. So they start out as an intern and they get to clean up the things that others didn't wanna clean up. That's the first three days in the life of a bee. Then they move on into a sort of DevOps and delivery role where they help bring new bees, i.e. workloads into life. So they're literally nannies that feed other bees and help them get out of their comb. Then they become builders, which in our world would be the equivalent of an SRE. They quite literally built honeycomb. They quite literally produce wax and they stand the infrastructure that the rest of the hive runs on. After that, they move to the blue team where they prevent bad actors from entering the hive. So they're quite literally god bees that stand god at the entrance and followed by becoming a forager. And when you normally think of a bee, this is what you would think of. You would think of the one that flies around and goes to flowers and collects nectar and pollen and brings it home. That's only 20 days, only half of the entire life of a bee. So let's do a show of hands. In those 20 days, how much honey do you think a single worker bee produces? Who thinks it's less than one gram or 28 of an ounce? All right, I wanna say that's about a quarter. Who says it's five grams, about a fifth of an ounce? Wanna think that's about the same, hands? And then 10 grams, a third of an ounce. I think that's about half. It is really less than a gram. It is 0.5 grams on average, 156 of an ounce of honey. That means to fill one jar of honey, you need the life's work of a thousand bees. The reason bees live 43 days is because they quite literally die of exhaustion. So they keep collecting and collecting and eventually they just can't collect anymore. So a single jar of honey represents a thousand bees. Let's talk about drones. So that was the worker bee. This is the bee that makes the worker bees the most common bee. It makes the mass of bees in a hive or put into the Kubernetes world. It's the mass of workloads that you would run. And then there is a drone. And the drone doesn't have a good equivalent in this example or in this analogy. You could think of it as a way to bootstrap the hive. In nature, the purpose of a drone is to ensure genetic variants within the bees. They do have a service discovery aspect. So drones and Virgin Queens meet to introduce genetic variants. And they need to discover where they would meet. And the way that happens is through the use of so-called drone congregation areas. So there are specific areas that drones fly to and that Queens fly to where they meet. And there are about 100 meters in diameter and they can contain drones for more than a hundred different colonies. So thousands of drones. And the interesting thing is we humans have absolutely no idea how they do it. We don't know how they identify those areas and we don't know how they, how to do that kind of service discovery. And yet drones as well as Queens are capable of doing this. So they're capable of, most of the best theory we have is that they can detect landscape features to find those areas. Now we don't understand how they do it. So you might be wondering, how do we produce maps like that? We produce maps like that by sticking tiny antennas onto bees and then we use harmonic radar to track their flight path. And this is how we study the flight behavior of bees. There is a out-of-resource pressure eviction. So in Kubernetes when a node runs out of memory, out of disk space, it will start evicting workloads. And the same thing happens in bees. As the winter nears and food gets more scarce and there are fewer queens that would need drones to match, drones are evicted. And in English this is quite literally called drone eviction. In German this has a very poetic name. In German this is called Dronen Schlacht which literally translates to the battle of the bees or the slaughter of the drones. Let me show you how that looks like. So you see here the two bees sort of fighting and the lower the bigger one is the drone and the upper one is a worker bee. And you might be wondering, how is it possible that this tiny bee can manage to fight off this big one? And we're just saying this is a resource starvation situation, right? We're running out of resources. And so the drones just no longer get food. Drones are incapable of feeding themselves. They cannot sustain themselves without the help of worker bees. And so worker bees just stop feeding them and so they get weaker and come eviction day they cannot fight back. The queen. Intuitively we think of the queen as sort of the head of state and the name would imply that. In reality, the queen is really just one other function within a hive. And in our analogy to Kubernetes, the queen is really a horizontal pod autoscaler. So the horizontal pod autoscaler will increase or decrease the number of replicas in a replica set depending on different signals such as CPU or memory needs or requests per second. The queen does very much the same. What a queen does is she lays eggs and scales up and down the mainly scales up actually down as a natural process the population within a hive. And so here you see the average colony population as it happens over a year. Again, this is a very Eurocentric view and this curve will look very different than in other parts of the world. What you see here is that we go from a mere 5,000 bees in the winter to up to 60,000, 60,000 bees in the summer. Now let's do the show of hands again. How many at their peak, how many eggs can a queen lay per day? Who thinks it's 500? It's about five hands. Who thinks it's a thousand? That's considerably more, I wanna say a third. And who thinks it's 2,000? That is the vast majority and you are indeed correct. So a queen will be able to lay or bring to life, so to speak, 2,000 bees per single day. There is a failover mechanism that does not exist in Kubernetes. In Kubernetes, if your horizontal pod autoscaler goes missing for whatever reason, it is not automatically replaced. Whereas with bees, if the queen goes missing, that does happen. So when the queen goes missing, the hive will recognize that and will take existing brood to rear a new queen. So what they'll do is they'll take existing larvae and feed it with an excess of so-called royal jelly or chaleureale, which is a very nutrient-rich mixture. And that's the only difference between a worker bee and a queen bee is what they were fed when they were young. Should there be no brood anymore, worker bees can also start laying eggs which are then not fertilized, and so only drones will come out and the hive is doomed to fail. So these are the different types of workloads, the different types of bees. Now there is communication. It's a distributed system, so they need to talk to each other. And one common pattern that we see in cloud native applications and in applications that run in Kubernetes is using a PubSub mechanism, something like NATS or Redis or RevitMQ or something of that sort. And bees have several PubSub mechanisms. One of them is known as the waggle dance. And the purpose of the waggle dance is to demonstrate where there is food to be found. So one bee flies out, finds nectar or pollen, and then comes back to the hive and if there is enough food for other bees to gather, the one that just returned will advertise where that food is to be found. And it will do that by walking in a figure of eight and on a straight line will waggle their tail. And the angle at which they do that indicates the angle of the food where the food is to be found with respect to the sun and the length of the dance indicates how far away that food is, roughly one second per kilometer. So let's see how that looks like. So you see the bee walk in that figure of eight and shaking their tail and that tells all the other bees around where to find nectar. The very same mechanism, the very same sort of PubSub mechanism is also forming, also forms the basis of quorum which we'll see later. There are other PubSub mechanisms, specifically pheromones. And pheromones are widely used in between different participants in the system and for different purposes. So for example, the queen emits two pheromones, nine ODA and five HDA, which you can think of as a health endpoint. And that's really what it is. It's sort of the life-ness check of that queen. And it tells the rest of the hive that there is still a queen here. You don't need to produce a new one, please don't. It is also something that really calms the hive and you can tell the difference. If you approach a hive that has a queen versus one that doesn't, the one that doesn't will sound very aggressive, whereas the one that does will sound very calm. Worker bees also emit pheromones amongst each other. And so there's for example, the alarm pheromone. The alarm pheromone unsurprisingly is an alarm signal, something is wrong in the hive. Now the alarm signal is chemically very similar to bananas. So don't eat a banana and open up a beehive because that immediately triggers the alarm signal and you might be confronted with 60,000 very unfriendly creatures. There's also the Nazanov pheromone and the purpose of the Nazanov pheromone is basically a you are here signal. And it's again part of a, if you're so like stretching this a bit, service discovery mechanism where it tells the bees where they need to go. And so the way this is used is that bees will hang on the warm summer days. You'll be able to see that they'll hang outside of a hive and they'll fan air out. And part of it is to cool the hive. The other part is to tell incoming bees where they need to go. So it looks kind of like this. They just hang outside and they fan their wings. And what they do here is they really spread this Nazanov pheromone in a stream outwards. And it's kind of like a landing strip that tells the incoming bees where they need to go. This is really helpful when you have multiple colonies right next to each other. Where otherwise there might be a chance of confusion. There's also more direct one-to-one communication. You know, things that we might be using something like GRPC or REST style APIs for. Bees have something similar. And it's called, its scientific term for all I could decipher is the whoop whoop sound. And the whoop whoop sound is used as a means of collision avoidance. So a beehive is a rather crowded place. And they need to avoid bumping into each other. Let me show you how that sounds. So that was the whoop whoop sound. And the way they do this is by disconnecting the muscles from their wings and then vibrating really quickly. And that sends a vibration through the honeycomb and tells the other bees that hey, don't bump into me. Honeycomb is a really good segue. Honeycomb in this case is pretty much our nodes and control plane in one. So when Kubernetes we have multiple nodes and we have a control plane that controls it all, essentially, honeycomb acts as both in one. So first off, it's quite literally the thing that bees run on, pun intended. It's also the place where they collect and store the nectar and then turn that into honey. And so this is an important thing to know. Bees don't actually collect honey. Bees collect nectar and then they turn it into honey. And there are two key differences between those two substances. One, nectar has about 80% of moisture content whereas honey has about 18, 18% of water content. And honey has gone through bees, so to speak, multiple times. And every time they add enzymes that make the honey more durable and antibiotic in the end. And this process happens on top of honeycomb. So what you see here is at the bottom, the sort of glossy stuff is basically nectar and honey in preparation and the covered stuff is honey that has reached the required moisture content of 18% and then it gets capped. And this is one of the many ways that bees do state management on top of their control plane, on top of honeycomb. So once honey is done, they'll cap it mainly for preservation reasons but also to show that nothing no more needs to be done to it. And it helps beekeepers understand when that stuff is ready to be harvested because it's capped and that's a good indicator that now is a good time to pull it out. Honeycomb is also used to grow the bees themselves. And so what you see on the far left are the eggs that the queen laid. And then the further we shift right we go through the different stages that it be grown in. And in the end again it gets capped and you'll notice how the capping here looks different. It's much more yellow, it's not as white. And in there the bees will take a few more days until they're ready to basically bite their way out. And so whenever you pull a frame it's very likely if there's brood on it that you'll be able to see a bee trying to make its way into this unknown world. We also spoke about the queen as a horizontal pod autoscaler. And the size of honeycomb determines what kind of brood gets laid in it. So the queen doesn't actually get to make that decision. This decision is made by the bees as they build the comb and the size determines what gets put into it. And so again pulling this back to the control plane analogy. Like if you think of these individual cells as resources they tell you, they give you information about what needs to happen with that. So specifically worker comb is about five millimeters measured from flat to flat side. Drone brood is about 6.4 millimeters or quarter of an inch. And then queen cells we spoke about that auto failure, failover mechanism in the autoscaler that then grows up to 8.5 millimeters. Now the difference between a worker egg and a drone egg is that the worker brood is fertilized and the other isn't. Which also means that drones don't have a father. So male bees only have a mother, not a father. You might be asking why is honeycomb shaped the way honeycomb is shaped? Why are bees doing that? And the reason is it is the most efficient way to build this structure that with regards to how much material do you need to build the walls and how much surface do you get out of it? It is also the only polygon that tessellates across the plane. So take this example, when you look at the triangle we need six pieces of outside wall to fill four triangles of surface area. Whereas with a polygon we can use those six pieces of outside wall to produce six pieces of surface area. Again, we humans don't really know how they make this hexagonal shape or this polygon shape that we all know of honeycomb. The best theory we have right now is that they build something that's essentially cylindrical and then they shape it into the shape we all know. So we've spoken about the workloads, we've spoken about nodes and the control plane, and then there are clusters as a whole. And in this analogy, clusters are the hives. And so hives are a collection of all these things put together. This is what a modern so-called Langstroth hive looks like. And most sort of Western styles of beekeeping use some derivation or another of this kind of format. And so what looks like essentially server blades is really individual frames where the honeycomb grows in. And then the different levels are used for different purposes. So at the bottom, the deep super is where the brood grows and then there is what's known as a queen excluder. It's basically a grid through which the queen cannot go. And on top is where the bees will put the honey. And so then we humans can just take the honey and extract it. There's also multi-cluster setups which is actually the most common way of keeping bees where you would have multiple hives standing next to each other. And you might have seen the colorful paint that gets added to bee hives. And that's not just decorative. It also helps the bees actually find their home. So as they leave the hive, they'll circle around and sort of scan the area that they're in and they remember the color of the hives that they left from. There is an admission controller that exists for hives. So admission controller in Kubernetes will reject workloads that we want to run on it or not. And something similar exists in bee hives. So if bees that are not part of that hive or say wasps or bumblebees or anything else tries to enter that hive, that will be rejected. And the way that looks like is a bit like this. Again, you see one bee trying to convince another not to enter that hive. And these are the guard bees that we spoke about earlier that are part of sort of a worker bee's life. Control loops. Control loops are one of the, if not the core concepts in Kubernetes. So at KubeCon in Valencia, there was a talk that looked at different real world principles and ideas and how that applied to Kubernetes. And one that was specifically given emphasis was control loops. And so what they do is they take a desired state in the real state of the world and then they'll perform some action to produce that real state of the world. And so in a Kubernetes context, think of that as I want five replicas of some workload and then there's a replica controller that will produce formal replicas just so I get to this number. And bees run a host of control loops themselves. So specifically looking at temperature and humidity. The chart on the left shows the temperature inside this particular bee hive that was looked at. The green curve is the outside temperature and the orange curve is the temperature within the hive. And you'll notice how it's approximately stable and little disturbance from the outside temperature makes its way to the inside temperature. And they do maintain that within sort of a perfect brood temperature range. Humidity is even more impressive. The blue curve is the outside humidity and the green curve is the humidity within the hive. And bear in mind that in that time when this was measured they'll collect a lot of nectar so they'll import a lot of humidity and they manage to get all this humidity out to keep the overall humidity within the hive stable. There is a failover mechanism that exists in bees that does not exist in Kubernetes. So if the hive runs out of space, so to speak. So for example, in Kubernetes, if you can no longer schedule because you don't have any more nodes and you also cannot get more nodes because I know you've reached your quota limits, for example, or you don't have an autoscaler, you just don't schedule. Bees, in turn, will basically create the equivalent of a new hive. So think of this as your Kubernetes cluster doesn't work anymore, so it's just gonna go and create a new one. And that's literally what bees do. The process is called swarming where approximately half of the colony will decide to just go and find a new home and create a new hive. The way this works is that they'll all stream out of the hive and it literally looks like pouring bees out with a fire hose. It's quite impressive and a little bit scary to see that. They'll then find a spot where they'll come together and after that, a search for a new site happens. And this search for a new site happens when scout bees leave, try and find a new place three to five meters above ground. Ideally, bees have been there before. And then they'll come back and they'll advertise for what they found. So, and the advertisement happens through the waggle dance that we saw earlier. If another scout bee will see that, they'll go with that original bee and look at the site that was found. And this process is repeated until enough scout bees have decided that this is a good site. So there is a quorum mechanism by which the decision is made where to go next. And once that decision has been made, the scout bees will emit what's known as the piping signal. And the piping signal sounds very similar to the whoop whoop sound. And it's basically a, ladies and gentlemen, start your engines because all those bees, very tight in this, hanging very tightly together, if they all were ready to fly at 35 degrees, they'll just boil. And so they don't do this, they cool down and then when that piping signal comes, they all warm up and get ready to fly. So, sum this up. We've seen that a lot of concepts that exist within Kubernetes also exist in the real world, specifically within bees. From auto-scaling to failover, even failover modes that Kubernetes does not yet support, to service discovery, role-based access control. We have not spoken about metrics. Bees emit a variety of sounds, weight, count, temperature, et cetera, that we as sort of operators of those clusters can use to gauge how well they're doing. There are means of message passing. There is a concept of time and there's quorum. So in conclusion, inspiration for solving our problems can come from pretty much anywhere, specifically things that look so tiny like bees. And I know this is a really lofty conclusion. So I'll end on something that's a bit less lofty. These are pretty awesome creatures. Thank you for your attention. And with that, happy to take questions. There's a microphone coming, I believe. The question was, can you define quorum? In this case, I would define it as sort of a majority decision, basically. So there's not consensus. It's not that everyone agrees. It is that there's a majority that agrees. So the question was if there were any other metaphors that I liked, but that did not fit the mold of the story. We thought to, like my co-author and I, we thought to pull this a bit, like structure this along sort of this a distributed system, this is what Kubernetes does and then this is what you can run on top of Kubernetes. And we did find metaphors for these individual points, but as you noticed, some of them were stretching the analogy already and for structuring this way, we found that most would have, yeah, strained this beyond credibility. Hi, you mentioned in the beginning that you're looking at bees in two very specific areas, but why was that? Why not bees in general? Are the bees significantly different and match more with, I don't know, hypervisors or something? Yeah. So there are several reasons. One, Simon and I, we are both beekeepers. We keep bees ourselves. And the ones that we spoke about is just what we're most familiar with. It is something that we've very intrinsically or intricately looked at, whereas other bees, some behave similarly. However, there are about four or five types of honeybees that behave in the way I showed. Wild bees, for example, behave completely different. Most of them are solitary creatures that don't form hives and don't exhibit sort of similar behaviors of distributed systems that we saw here. So part of it is, this is what we know, part of it is some bees just really behave very differently. It's a great metaphor, but is there any design inspiration we can take in terms of implementing new features or strategies based on honeybee? So the failover modes are particularly interesting where, for example, the cluster essentially replicates itself in a way. I'm not even like, when I think about needing to operate something, I'm not sure I would like that. Also thinking about cloud cost. But this is one thing, like the many ways that bees can failover, the resilience that is built into the system is astonishing. And goes, like Kubernetes does a really good job of that. What bees do goes way beyond what that system can do today. So if any, I would say it's that one. So earlier in the talk, you had the heat map showing where the bees kind of were and you said that there are antennas attached to these. How do you get the antennas on the bees? Painstakingly. So marking bees with something is actually quite common process. So queens, for example, are commonly marked, sometimes with kind of little stickers. And so there is glue that does not hurt the bees and the outer shell is a Hineen. So you can stick things to that. So the way you add the antenna is you catch a bee and you glue that antenna to the back. How do you catch the bee? Great question. Bees are super calm creatures. So you can pull out a frame and they'll just walk around as if nothing happened. And so one way you can catch them is quite literally you can just pick them off and they'll do nothing. Cool, thanks. My father is a beekeeper. I think this is fascinating and I love the metaphor. I'm wondering if you can extend it further and comment on the role of the beekeeper in this versus the role of humans working on distributed systems or their lessons we can take away in things like doing maintenance in a non-impactful way, that sort of thing. Yeah, yeah, that's a really great idea. I briefly alluded to this when showing the overview of how concepts that bees have and that we have in sort of this world or in the Kubernetes world overlap. One thing that we did not speak about is exactly what you alluded to, the role of the beekeeper. I would think of us as sort of operators, really, operators of those clusters and hives we do use signals and we do use observability into those hives. So if you wanted to stretch that, you could say that, for example, the sounds they emit or the weight of the hives is kind of like a count of what's happening. So there's a whole host of signals that you can pull out. Most of it is through manual inspection. And so here maybe there is something that we as beekeepers can learn from this community where this community puts a lot of emphasis on automation and good observability. In beekeeping, it's a very manual process and the equivalent is literally do a cube control get pod on every single pod in your cluster just to check if it's still healthy and working. You wouldn't do that, but beekeepers do. Awesome, thank you very much. Thanks for coming. Thank you.