 Good morning, everyone. QMX, as she said, no one knows me by Douglas. I work at Digital Ocean. And yeah, thanks for them to actually bring me here. I have a very important disclaimer to make at this very beginning. So this talk tells a story about how things went badly, badly wrong. And sometimes when you have to explain the reasons why you came there, you have to use examples. And this disclaimer essentially begs you a favor. Please be nice. And why I'm asking this question and asking you to be nice is that, you know, this is a very cute animal. I mean, I really love the golfer. And I have to talk about Golang problems, because that's what motivated us on trying to look for us. But I really like Golang. I mean, don't take it personally. It could be any other language. That's just the example du jour. So let's get moving. Please, leave the golfer alone. This is story time. I was working at my company, and I joined a team for tackling a very interesting tech problem. We needed to process a bunch of data and do essentially some string processing. It had like high performance requirements. It was a distributed system. And we're going to build microservices. And you can imagine, like, this is like the buzzword bingo. I was going to use other words with B, but whatever. So essentially, this is like a classic problem when people say, yes, yes, this is perfect. Everyone kind of handles the same stuff. And this is Unix. We know this. No, actually, this is Golang. We know this. That's the standard answer to that question in general. And we fast-forward a few months there. And the project, actually, was a success. I mean, we did a microservice using Golang and all the technology that is there, all the tools, and everything was happy. It was running in production. It was happily ingesting over 4k very complex messages over at KafkaBus. And that was it, like, the project was running. Everything was happy. And are we done? The reality is that software is never, ever done. I mean, when you release it, then the party starts, right? And then is when you find out that not everything is roses. So we were running. We're in production. Customers were using this software indirectly. And my process crashed. And I had no idea why. Then that's kind of like where you start learning more about the language that you are working with. And I started looking at Golang with different eyes. Essentially, I have two structs there. Nothing special about them. They're kind of like the same as the C, the same as Rust. We found a huge difference. I didn't know at the time that the default value for structs and for things inside structs was new. And it manifests itself in subtle ways. Because when I kind of try to call a method on a pointer to a structure, this pointer can be new and the compiler is not going to complain. And of course, I did a big mistake. I had a new reference there. And then when I called, it exactly prints wave on a new reference. If you are not assessing anything inside the structure, this is fine. But as soon as you have data there, boom, you're going to have a crash. And a seg fault. And it's terrible to debug until you grow older and kind of you learn more about the language. And then, OK, fine. I'm not going to do this mistake again. But it will be really nice if the compiler will bug me that this was like a new reference. Like, don't do that, please. OK, this was a real problem. This caused it an outage. And it was my fault. But whatever, it was annoying. But there are some things that are more subtle. And I would say annoying, but not like life and death. But it's still annoying. So when you're trying to process a lot of network metrics and things going from the fire holes of events, like we're a cloud provider after all, there's a bunch of machines there. And suddenly, you have to deal with a lot of counters. And those counters, usually when it's positive, you are going to go for unsigned integer 64. That's fine. But what is the difference from the first to the second? One might mean megabits. Another might mean megabytes. And this sounds like a very naive mistake, right? If you get the units wrong, everything is going to be wrong. And the only way that we found out that we were getting things wrong was because some customer was looking at the charts and like, wait a minute, it's impossible for me to have all this bandwidth. It's kind of like, yeah, we were doing things wrong. But this was like on the very early days on the beta access happily. And we got the bug before it went to general access. But there's another fun one. As we have a lot of networking gear, we have a lot of CPUs. And when you start doing processing on CPU metrics, sometimes you have to do the linear interpolation for getting the points. And suddenly you have a bunch of slopes and a bunch of counters that it's a pain. Because you probably need to do some functional programming style, folding over several values. And oh, boy, that's rust functional support will be so nice, so much better than my function that was returning a function that was applying a slope. Yeah, it was not cool. So I have very one thing. There's only one thing that I really disagree with Go that's kind of like, and I'm not going to talk about it because it's generics. I think it's like even their own. OK, I love them still. But let's talk about rust, which is why we are here, right? So there are several ways of you introducing a tech to your company. And my approach was to try and make sure that we did things on as grass roots as possible. I'm kind of like not trying to make a big boom, not trying to force this choice over anyone. After all, we're a Go Lang shop. People love Go there. That's fine. So the first thing I did was to create a highlight for rust in slack. And then another one that was kind of like a little bit more evil, which is kind of like, all the problems that I know on Go Lang and on Ruby, we have a lot of Ruby. And I kind of like I have a very special set of highlights. So when people talk about those problems, I'm aware of them. And then I'm not going to go like rust evangelism strike force, please no. I'm not going to ask people to rewrite things in rust. Please no. I'm going to just be aware. Because my end goal is that we get better software. And we know those start commies can be addressed in a better way. So we start with a side project. A very small side project, kind of like a toy project. But the main difference here is that I try to make it production really. Like, what would I do if I had to ship this to production? So it's not only like it compiles, it works. We know that this is kind of very true on rust. Like it's amazing. But yeah, let's do more. So it might sound like overkill. But if you keep practicing and trying to make things better and better and better, they are going to eventually better. So the first project was actually a Grafana Notations Data Source API. I will have ever used a Grafana here. The definition is a dashboarding thing. It's a very oversimplifying it that shows amazing dashboards and gets Prometheus metrics and things from other sources and actually show you nice charts that helps you when you are paged at 2 AM and try to debug something. It's great. But those charts actually don't have some context. Like, I didn't get a deploy, but I added an index to my database. And then what is the effect of changing another part of the system that's going to impact my application? It's going to reduce CPU usage. The database queries are going to return faster. So we kind of like put a marker there. It's a dashed line. By the way, Grafana introduced this as a native feature in Grafana 5. But there was not there yet. So we created a very simple API where I just go into a Slack bot and say, just added an index to the database. And then we would see like a line. This is the time where the change had happened. And then you could correlate what was the behavior of the application. And it's a very simple REST API. Like, this could be like drawn in 20 lines of JavaScript, whatever. So we added Prometheus metrics to this endpoint in this application, made sure that we had all the data that a big application would have, like, how long are we taking to write to the database? How long are we? Yeah, we're just doing the metrics the right way. And for things that are exceptional, that should not happen on the normal course, we have structure logs. And we're right into the centralized logging infrastructure that we have at the company. And not only that, I made sure to have a CI-CD pipeline. So we could actually have the same practices we have for our production software. But this is on this toy project. Of course, after a CI-CD pipeline, you're going to package it, and we made sure that it worked well with Docker, make sure that we are caching the artifacts with a different set of hell when you're trying to deal with Rust, because depending on how you do it, every build is going to take forever. Even if you're just doing like a hello world, you're bringing like several crates and suddenly your build takes 10 minutes, it's not cool. And finally, we deployed it to Kubernetes. It's kind of like almost the standard nowadays. We still have a lot of mazes, but yeah, I think Kubernetes won't. And after you have done all this, after this thing is running, it's on production, you are like seeing how it works, then you need to start asking the right questions. And what is annoying to develop on Rust? Like how hard was to actually build the simple application in Rust? How do you debug this thing? It's kind of like, what are the failure modes? Like what happens if the database dies? Is your application going to say false and crash? Are you going to just exit? Are you going to try to recover? So there's a lot of questions around that. Like how to onboard new team members? Like is learning Rust really hard? Are people going to feel comfortable? How about IDE support? People, everyone loves their preferred editor, like we know that there's more than Veeam and Emacs, right? So essentially, everyone has their preferences. You kind of have to try to find the compromise where you can make everyone happy as possible, as happy as possible. And this was a success mostly. The project was running, it was fine, everyone was happy and that's it. And then we move it to the second project. Now we're getting more ambitious. We had an internal hackathon on the company. And again, I have the opportunity of replicating the first system, the bigger one that was written in Golang, which was like, it's stream processing again, reading data from Kafka, and then I found the Holy Grail. There's a very nice library called Liberty Kafka, which is extremely performant. You can get a lot of performance by using this thing. And I didn't have to write it. It's amazing, awesome. We use that and we start parsing a binary protocol with some proprietary stuff. Yeah, unfortunately, but whatever. This parsing then do some real time analysis over this stream of data. It's kind of like, it's pumping from, literally like all the machines that we have and getting this data. And we got impressive performance results by just trying to do the same thing that we're doing in Gol, but doing using the Rust Machinery. And it was a success, like a side project that went really well. We had basically zero problems. Awesome. And then we go to the third project that we get like amazing, even bigger scope, even more chances to fail, right? Okay, let's do a microservice in Rust. This time like with the full, brown structure. The idea was to fold over a Kafka stream again with a lot of machine data, process it and do like the normal operations that we needed, persist the results. So this involved ORMs, databases and all those things. And finally, there's another bit that was really painful, which is actually exposing it via a GRPC API with something that it's so easy to do in Golang. And then when you come to Rust and oh shit, now I have to use build steps and compile the proto and generate some things to help. Yeah, okay, it went well. It was a success, right? I mean, one, two, three. The three need to be a success, please. Okay, mistakes were made, a lot. And what happens is a combination of failures. This combination of failures starts with a very interesting problem. We suffered from the zero syndrome. And I just realized that my slides are not synchronized with the older ones. Okay. So sharks tornadoes, diesel powered rockets, pineapple and pizza. What those things have in common? A bunch of those things happened on last RustConf, where we had Sergio showing amazing things on rockets and we got like, yes, we're going to use this, this is amazing. And we have seen Sim Griffin also saying like, great things about diesel and like and how zero cost abstractions are awesome and all those things. Sure, but when you add sharks and tornadoes, you get shark noddle and you don't want that. So essentially the problem is when you jump the shark, but intended, everything gets complicated. And so when you have zero allocations, like I wanted to get the date, they straight from Liberty Kafka. And without allocating it again, I parse it in memory. I have the references going there and then I finally use this thing and make it work and everything is awesome and rainbows and unicorns. And then you have like, I try to make my type zero size it and I go diligently making this work. I push really hard my zero code abstractions. I'm going to end with zero working software because I'm wasting a lot of time. So, yes. You sum this with the fact that I was the only experienced Rust developer on this team, like new also, a little bit more than the other guy. Essentially my takeaway is clone away. This might sound awkward and even counter production like, wait a minute, we wanted to use Rust because of the ownership. You know, learning things is complicated. And if you just clone away while you're getting started, you are still going to be miles ahead on memory safety if you are doing malloc and free. It's a trade off, it's kind of like training wheels for your bicycle. You're still learning how to deal with the language. And this helps a lot for you to get speed up and get productive and fun story. Even cloning, it was faster. But yeah, whatever. Learning ownership is not easy. That's my takeaway. I've been doing Rust for several years now and I still don't get it properly, I think. It's always something new that surprises me. Usually I'm wrong. That's why I get surprised. Like, oh yeah, I wasn't paying that much attention. But it's really rewarding. I had the pleasure of having an intern for this summer doing a Rust project. And he was like, oh wait a minute, I have to think before I build stuff. I just cannot go bashing and think about the types, thinking about the things that I'm trying to do. So essentially when you spend some time trying to learn the language better, you become a better software engineer, which is amazing. It's really cool. And talking about cool stuff and tornadoes of sharks, Rust nightly is super cool. And this was one of the main consequences of we trying to use Rocket. And this is not to Rocket's demerits. I still want to use Rocket. But suddenly we found out that nightly is awesome. But it's a nightmare. Because suddenly we had a problem that Rocket was having because of the plugins on the compiler. And then we had to upgrade to a new nightly snapshot. And then suddenly we broke diesel. And then oh shit. And we went into that very inconsistent state where we couldn't go back because then other things will start failing in very unpredictable ways. And in hindsight that was a very stupid idea. But yeah, it's kind of like everyone does mistakes and that's it, it's fine. So the takeaway for this is like fewer moving parts please. It's kind of like if you can stick to a stable version you're going to be a way, way happier person. Even if you miss those cool features that nightly has. So a high rate of change, which is actually what happens in nightly, leads to an even higher cognitive load. And when you think for a bit, wait a minute, we were already kind of learning and onboarding people on Rust. And the reality is that with this context a stable can be bleeding at enough already. So yeah, Rust is cool and it has its price to pay when you're learning something new. It could be any other language. And which brings me to the other points that I wanted to address today, which is this shiny object syndrome. Let's be honest, shiny objects are cool. It's kind of like playing with new tech, new hardware, new approaches, new language like Rust, which is not that new if I'm understanding correctly, but whatever. So sometimes you get really smart abstractions. You really want to make things shine and be really efficient. And sometimes it feels like magic. Like, so I'm having this type that is being automatically dereferenced into something else. And then the compiler is able to align this thing and make a zero cost abstraction or whatever. It's amazing, but sometimes you end up with types that have like this, where, trade, trade, trade, trade, trade, several trades. It's kind of like it gets really complex. I think that the good example is that has anyone heard about Scala's third incomplete type system where you can do like Hanoi towers in compile time just by using types? It's kind of insane and super cool. Check it out. But yeah, it's kind of like you want to play with those things and have fun. But the reality check is that this is not Hogwarts. If you're keeping doing magic, you're going to have a lot of pain to deal with this thing later. The fact that you can do it like the Hanoi towers doesn't mean that you should do it on production or on a real serious project. And then I have something that's always bugging me. My background is Java programming and very big and boring enterprise systems. And I ask you, oh, yeah, I jump in the gun. What is the idiomatic code? I kind of gave it away that when you are on Java or something more traditional, you have a lot of literature. Like people have the information like people already went through many problems and the patterns emerged from the need. And then you kind of like have good solutions, not only for cold issues like object orientation or whatever or some functional paradigms that you can use. But you also have patterns for things like enterprise applications, which is going to have a talk today that talks kind of like how to structure a worst application, let's check it out. So the takeaway here is that Ferris is pretty young. And it's really cute, so by the way, I love this. I wish that this was like the official mascot, not the unofficial one. But the thing is, we don't have established patterns yet. We have a bunch of things that are happening. And yet, we don't have them yet because the community is actively working on trying to find those patterns, which is great. So it's really easy to end up writing X and Y, like language A and language B. And this is like a great example of the things that you should not do. It's kind of like try to make abstract proxy factory from Java to go into interest. Or you go for the if-are-new style of early returns and you can make like very long functions, which is pretty OK and go in general. Because you're kind of like that's the style and you get used to it and it's really like easy for you to reason about it. But this doesn't mean that it's idiomatic rust. Also, if you end up doing a lot of match, match, match, match into nested things, and then you go with Pyramids of Doom like JavaScript. I mean, this is not JavaScript. This is Rust. We need to try to play differently and actually experimenting with those things is really important. But you really need to be mindful of your own business. We all have business. We all have experiences in our lives. So this brings me to the most important point, which is the human factor. When we go over all this experimentation, see all the mistakes, and ask the right questions again. And by the way, asking the wrong ones is fine as long as you're asking. So the first question that I have is am I too excited about Rust? I mean, we all love the language. It's kind of like it's really cool. I would easily like, oh, do you want to write Rust full time? Sure, why not? It's kind of cool. But when we get too excited in hindsight, it's kind of like hard for you to pay attention because you're so excited. And sometimes people are talking to you and saying like, hey, this is not cool. This is kind of like a trouble song. RLS is crashing every hour. I mean, I cannot jump from a function to another. I have to use C tags, really. I mean, yeah. Am I listening to what people are saying to me and the signals that they are giving me? Or I'm just listening to what I want to hear. I love Rust, and I want to make a successful one. YOLO, let's go. No, no. It's kind of like we really need to be mindful. And this is an interesting one. Make sure you have explicit buy-in, not only from your direct manager, but beyond to the infinite and beyond. Why? What happens if your manager moves on and he was cool with a Rust project and suddenly the director wakes up and wait, we have a Rust project here. I didn't know that. And suddenly you have a very big ball of mud to deal with. It's kind of like it's nice to have like the experimenting in a corner, but make sure that this is being communicated somewhere else. Coming with a surprise can be really bad. And essentially can mean that even a successful project can be killed. So, and I think this is the most interesting one. Like people have opinions. Some people were like, you don't need a package manager? No, no, it's fine. Just vendor everything. You don't even need the goal link tools that vendor things. You just copy stuff there. I heard this, but yeah. People have opinions surprisingly. Yeah, and you need to be mindful that not everyone has to think like you and you don't have to agree with them. You have the keynote saying that we have a continuous tension between different ways of seeing things and we need to be mindful of them because at the end of the day, cargo-cuting is totally a thing. And that's fair. Because at the end of the day, we are humans and people and interactions should matter way, way more than the technology that we are using. So on a closing note, there's a secret sauce for making all of this be a little bit easier. The secret sauce is empathy. Empathy is a powerful feature to actually make you go out of your perfect roast roads and understand that sometimes goal link is the better solution. Even if it hurts to say that. But yeah, there are some situations where it doesn't make sense to try to force things. And honestly, this is really hard to do. I would say that by far is the hardest thing like to actually admit defeat. Okay, this is not the time yet to use roast. But yeah, if I have to give just one advice that kind of like applies to me and I failed on this and please don't do what I have done. Please use empathy. Thank you.