 So now we have John Mark talking about monitoring strawberries also happens to buy my favorite fruit, which I don't get in India Okay. Hello everyone Thank you for having me. Thank you for organizing this room I'm very happy to be here today to discuss how we build observability in our projects. It's very cool So you see it's not going to be a very technical talk But what can be interesting for you is that we're going to look at observability from the perspective of IOT and also from agriculture Because as you will see we basically leverage IOT technologies to grow fruits and vegetables inside cities. So first we need to talk about strawberries I've been living in Paris this past two years and And most strawberries you can buy there They actually come from Spain or Morocco So they actually do travel a lot before they reach the store and This is why sometimes on the top of the strawberry you might see some white circles on a white crown It's because the fruit was harvested way too early before its full maturation and They do that in the industrial agriculture so that the fruit can continue the maturation during transportation And this is how they can extend the life of the product But the issue with that is that during the transportation the fruit is going to attack its own reserve of nutrients And so when you buy it it might not be that tasty and the nutritive quality can be very bad and Of course the second big problem with a long transportation is the use of pesticide So another technique used to extend the life of fruits and it raises a tons of questions about The impact of that farming model on our health So what we try to do at Agricool is To get rid of the transportation part in the farming model So we produce fruits directly inside cities So that we can harvest them at full maturation and so that they are very tasty and very healthy and of course always pesticide free and the way we do this We we buy these containers and inside of them we set up an IOT infrastructure to grow plants and to grow fruits and vegetables This is how it looks inside So we always start from a blank container and Then we set up Vertical columns we can plug the plants inside these columns We have walls of let's to provide the lighting and the energy for the photosynthetic activity of the plants We also set up an irrigation system that goes through the columns and we also have a quite complex climate regulation system made of an industrial air conditioning system and a bunch of misting pumps to control the humidity in the air and so a set of Air fans to control the direct the air circulation So whatever we need to set up in terms of sensors and actuators to have perfect control over the condition of stories so that they get like the Perfect setup to grow we install it So for example, we also install hives from bubble bees So that bubble bees can take care of the pollination of flowers And this is also something we can control with IOT so that we can control the door of the hive And so I find great control over the exposure between flowers and bumble bees This is the end result. This is how it looks like This one is located in Park de Bercy in the very center of Paris Here is another one decorated by an artist and so now we have around 15 containers in Paris It means that if you buy a planet of strawberry from a recall in a supermarket You know that you're dealing with a fresh product The harvesting date Flatories will always be on the box. So, you know, you're buying stories that were harvested the same day or the day before And so, you know, you're dealing with a food that is fully matured and with no pesticide We've run some analysis to compare stories with other stories from supermarket and on average We reach 20 percent more sugar and 30 percent more vitamin C. So pretty high Nutritive quality If you want to build something like this, you need to run a ton of research on plants So we are running a couple of buildings in Paris where we have set up This these cells for experiments. These are like smaller versions of containers and in each cell We can test one variable of the environment so that through trial and error you can we can discover the perfect recipe for the strawberries And to run one single experiment it takes a bit more than three months because that's the time you need to grow strawberries So it means that you need pretty good reality during three months in your IoT system otherwise, you might mess up the experiment and your results won't be useful and To build an infrastructure like that you need a team of people Made of a very diverse set of technical skills So of course we are a lot of agronomists To run the research and so they basically discovered the agronomic recipe Then our engineers will try to build the physical infrastructure that will make that recipe a reality inside the containers and Then we have the software team made of software developers and DevOps engineers And so our goal is to work very closely with these engineers and agronomists so that we write The best IoT infrastructure to automate these containers and make the recipe a reality so right now We are on 90 sorry between 80 and 90 people and seven developers in the software team So let's look inside In every container we install a machine with a debian instance on it This machine can communicate with the outside through a 4g connection and Locally it will communicate with a PLC controller You can see this controller as a unit that centralizes all the input-output interactions with our devices So all the sensors and all the actuators So basically with this controller in one place you can read the values of all your sensors and also the positions of all actuators and You can also forward new commands to change the position of these actuators On the debian instance we can deploy our software stack mainly we have a set of Microservices written in Golang and a Prometheus instance So in Prometheus we will store time series for the values of all our sensors and also the positions of all actuators and The main goal of the microservices is to regulate the environment in the container So for example, we have a service for climate regulation That service will communicate with Prometheus to read past values from sensors So it might send an HTTP request to Prometheus to ask what's the average temperature over the last five minutes And if the service sees that the temperature is far too off from our target It will run some climate regulation algorithm and the output of the upgrade might be some new commands for the actuators So it might decide to change a setting in the air conditioning system or to activate a misting pump to bring more humidity in the air So this is how you can use a PLC controller with time series to to regulate inside if we zoom out You can see that we replicate the same infrastructure in every container and we use docker images to to make these deployments and Then in the cloud layer we deploy all the user-facing web apps to build the observability So all the time series from Prometheus we summarize them into an influx instance We put a graph on top of it to build all our dashboards and graphs We also use red ash to build dashboards of our SQL store the data in our postgres instances And we also build quite a few observability pipelines over Slack Because all our players are using it so for example in Prometheus we have Alert rules to tell us when there are incidents in the containers These alert rules will be received on Slack so that we can send the technical support so in the container to fix things And so in these alert rules you might find your typical IT rules about your network your services CPU loads memory usage But you also extend the definition of rules To new layers in the artist tax so we write rules for the physical machinery with there and also rules for the plants so For example in every water pipes in the containers we have a pressure sensors and And so the values on the sensors are in Prometheus so we can detect when there is a huge change in water pressure and so these changes they will trigger alerts and They might indicate that there is a leak in the edulic system and so this is how we can send a plumber on site to fix it So when we think about observability we often think about these techniques, you know time series metrics domain events traces alerting and then we apply them to different sets of abstractions and so in our projects what we try to do is to be very conscious about extending the application of these techniques to the physical machinery in the containers and Also to the world of agronomy and plants And the only way you can do that successfully is to work very closely with domain experts You need to involve the engineers and the agronomists in the process so that they can Basically educate you about the domain and so that you can discover together. What are the most valuable? other IT pipelines to build and When you do that you will discover new techniques that you need to to develop So for example inside your containers, we have a set of cameras that it pictures automatically of the plants and the pictures get uploaded to a WSS tree and Then we have our agronomists reviewing these pictures And so they can discover if there is something wrong with the plans by looking at the leaves and the flowers and the fruits So it's very human centered pipeline, but we'll still consider it as a full authority pipeline Because it gives us really good indication about whether the system is healthy or not and We do this type of human centered approach in different ways we send agronomist inside the containers to check the plants we run testing sessions for our strawberries we analyze water samples from the hydraulic system and We also make a ton of measurements on the plants like measuring the size of the leaves to see if the plants are developing correctly So we always try to automate the pipelines as much as possible But when it comes to to plants, it's it's too complex and we need to put humans in the loop a lot So some of the specific specific changes with agriculture First we are an IOT company. So we we have to manage this huge fleet of sensors and actuators and they are distributed in different places in Paris. So it comes with its own set of constraints Then with plants you can get a very tough set of agronomic SLAs So for example a single incident in one container may be enough to impact the full batch of strawberries Let's say your irrigation system goes down for a couple hours It might be enough to to damage the entire production and so I have a very big impact on the bottom line of the business And when it comes to strawberries and berries in general it can be very tricky Because the flowers and fruits are super fragile We're pretty well known in the indoor farming industry for growing strawberries because most companies they grow herbs and salads But with berries you you need to be to be very precise in terms of temperature humidity irrigation levels and Finally plants can are very complex It's an illusion to believe that you can reto engineer a plant. It's way too complex So in practice what this means is that if there is an incident You cannot always tell if it has an impact on the plants or not You can look at them. You may see some some sign that it's healthy or not But you will never get the full story because you will never know everything that's going on inside the plant So it's always a good practice when you write a post mortem to try to measure the impact on your business But when it comes to plants is just super difficult for us to do So what we've learned bar in navigating these constraints is that Your obscurity pipelines they should always come from anything So you should not build them just for the sake of coping with the ongoing observity trend And so to do that we have this very simple framework that for us what's valuable is to get feedback on whether we're building the right farming model are we going to have great roots enough volume and Do our unit economics make sense so that we can scale to multiple cities And so from that definition, this is what we want to see We want to see observations that help us react to incidents to understand All the stories respond to the conditions we're giving them We want to also to try to predict what's going to be the yield and quality of the product and Also any insights that could help us improve the unit economics So for example in our system, we might have a Water pump sending water in the hydraulic system If we start monitoring the state of that pump, we may discover that we don't use it that often And so in this case later on we might make a decision to Don't grade it to a cheaper model and this is how we can make a unit economics decision driven by monitoring data So let's see the framework here in action What's always super valuable for us is to assess nutrient absorptions by the plants So basically if a plant is absorbing its nutrients correctly is going to grow healthy and provide us with great strawberries One way to do this is to use the VPD metrics VPD stands for vapor pressure deficit And it's kind of an indicator of the intensity of the transpiration of the plant So if you take a plant the leaves are saturated with water and it needs to diffuse some of that water in the hair to be able to grow and This is called a transpiration During transpiration the nutrients are moved inside the plants and this is all they are absorbed So at every stage of growth of your plant you need the right level of transpiration intensity and VPD is a good indicator of that basically a ratio between the level of saturated water in the leaves and the level of saturated water in the air and What's nice with VPD metrics is that it's fully automated. You've got your time series. You can be the Automated reporting on top of it. You can build the alerting and Then we have other pipelines for the same concerns. So we have our cameras that upload the pictures for us and If our agronomists look at these pictures and if they see some droplet of waters at very specific times of the day They may get very valuable information about the the intensity of transpiration We also run water analysis of our water sample from the irrigation system And if we do that multiple times through time, we may discover some interesting Long trend about nutrient absorptions So again some automated approach some semi-automated approach and we need to put experts in the loop This is the only way we can get like the best observations for us to to know that our system is working So all of this it relates to what we start phrasing as a domain driven observability So the idea that the pipelines are building. They're not just about techniques, but they are always related to Either a business objective or some knowledge from from the field of agronomy And so in practice it means that we want our system developers to work very closely with the domain experts On one hand you want these experts to share their knowledge about the domain so that developers can learn about physical engineering about climate regulation about agronomy about nutrient absorptions So that the developers they know why they're building these pipelines and they know why they should focus and put the priorities And on the other end It's nice for your domain experts to learn about the world of reliability and observability So that then they can understand the technicality and they can put the project in the right direction So at agri-cool we have a ton of engineers and agronomies that know a lot of stuff About time series, alerting, primitives, grapana We try to talk to them about best practices like post-nortense and defining SLOs over plants And the only way to do this Is to speak the same language So you want the wording inside your system To be the same as the one used on the field by domain experts Because if you have some type of cognitive distance between the two it will be very difficult to To add new features to the system and it will slow everything down And you could also have like these Miscommunications that could result in some nasty bugs For example, if we don't have the same VPD definition, we're going nowhere And so sometimes when we can we go as far as Enabling our domain experts to build our derivative pipelines themselves. So we have Non-computer people if you want That learn about influx that learn to write influx queries so that they can build their own grapana dashboards We have people learning about SQL so that they can use red ash and build the views over SQL data We even we even designed a web editor for primitive alerts So that we have agronomists that can write their own primitive alert rules And it will get deployed in any container they want for them And so it's kind of a trade-off because you are putting more people in the loop So there are more risk for errors or not following the best practices, but on the other end You remove the bottleneck from the small software team and everyone learns about building authority pipelines and you can move much faster So in a startup, this is like kind of a trade-off that you need to navigate So that's it for the slides. I guess my final message is that Authority is not just about techniques It's very much about the application domain. And so I think most projects would benefit from bringing domain experts in the authority journey and to teach them about the best practices and Yeah, so we are Growing startup. We are hiring. So if you like the kind of stuff for your building, feel free to send us a message Thank you Thanks a lot. We have four minutes for questions. Please don't get up. Please only get up after Q&A is over Thank you for the talk. Is any of what you show here open source? No, we we don't do We're not doing open source right now But we we want to do it in the future right now. We're trying to figure out how to grow these plants so we're not investing in opening stuff because We're not ready to bring that kind of quality, but we are very much looking to to to build some type of indoor farming community Starting in Paris and run some open source like at home and this kind of stuff But it's the very beginning for us. And so we really hope like in the coming months to distract something. Yeah Hi, I have a question So you mentioned there are a lot of manual work involved when your colleagues are looking at the pictures Do have plans on implementing? I don't know like machine learning models to Automate this process because essentially they're classifying. Yeah, they say they can say it's good bad or the classify the disease or something Something happens. Thank you Yeah, absolutely. We started looking at it So right now we we we do the manual approach basically because we don't have time to build the computer vision stuff We see it as like a new research project because it's very hard to estimate the effort you need to put into this Since we already have we always have the constraint of already running research on plants We try to remove that risk, but we could definitely because we know the dates for each picture We could definitely do some supervised learning about try to estimate The level of growth of the plant so I just discovered this water droplet automatically so we don't have Nicely tagged the data set today, but it's definitely going to at scale something we want to do to Make this already pipeline better how can we compare a good cool to the normal farming in its effective and How many containers doing do we need to cover needs of strawberries for Dubai? or some small town So it depends on what you mean by effective if you mean by by yield volume of production Yes, well enough production comparatively to the normal village is it So it's it's much more productive than classical agriculture in terms of the surface you need to use So what we right now we're producing between 15 and 20 kilograms per square meter per year, so it's much much more than industrial agriculture in terms of the The space You need to use So what's the big challenge today is more about the unit economics and make it work? Like make this investment in this big IoT infrastructure worth So we get much better yield, but we still a way to go before making workable at a bigger scale So we have one more question Have you planned something else fun to strawberry? Yes So our stories are in supermarket and for a few weeks now we have Herbs so coriander a basil salad that are getting in the store so these are more easy to produce in terms of The reliability level you need to reach in the system and yeah, our goal is to in the future Try to to address all the main vegetables and foods that people have in the fridge But right now. Yeah, we focus on herbs salads and strawberries Yeah, but yeah, it's a good question. Yeah, so I know this is not something we want to do today We so we are we are not Just an IT company we are first we are food and vegetables brand that's that's that's our project So we are very much about food And so we are focusing on strawberries and vegetables Thanks a lot. Thank you