 what is terminal, what are some usage examples, then I'll talk a tiny bit about myself and then we'll jump into the actual meat of the talk where we'll touch on some of the tools that we've been using and then a little bit of just general tips about how to write rust systems, I guess. So a little bit about terminal. It's a complete container management suite. It's basically the value proposition. We start very large containers in seconds like anywhere on the cluster. You can do full disk and ramp snapshots also in seconds and then load back from them or load them on other machines or anything like that very quickly. You can teleport running containers from one machine to another with like zero downtime effectively. There is really fast resource scaling if you want to add more CPU, less CPU, more RAM, that kind of thing to your containers. And we have a very efficient like diff based distributed storage layer which like enables a lot of the speed of startup and that kind of thing. So some of the usage examples of things that, oh yeah, we also have a really nice web interface that allows you to do things like share a terminal basically between multiple clients. So like I could have my container running and be in the middle of a debugging session and send a link to one of my other devs on my team and they can also hop in and maybe I'm working remotely and like you can do all kinds of really cool stuff with that. So some of the usage examples of interesting things that people do with us are we, they create many copies of for instance, their production database to be able to run integration tests very quickly for like CI or even just development. You know, if you're running some complicated server and you can easily pull down basically a full copy of your database locally and then run all kinds of tests and queries against that the CI things are gonna work out. And you can also with the capability to launch many instances very quickly you can launch thousands of like isolated environments that people can use for things like, we're used by like Corsair and Code Academy for like online education and you can do all kinds of pretty cool things with that. So, tiny bit about me. I really like open source and Rust. I'm an engineer at Terminal. I've written a bunch of crates that I own crates like iron and hyper and a bunch of other smaller things such as my GitHub. All right, that's enough about me. So, a lot of the success that we've seen in using Rust in production has been for basically thin infrastructure services, things that need to be reliable especially with regards to the threading. Rust is a really, really nice concurrency model that you do really kind of ambitious things without like being afraid that you're screwing it up which is really great. Things that need to be really fast that have like very low resource requirements or that need to very tightly manage the resources that they do use and things that need to low, that you want to have a low maintenance burden. You want your, you want to write the service, you want it to be correct, you want it to be well tested and you want that to be like the end of it. Now you can move on to other projects. So the things that we've done with this are we built a networking firewall for intra cluster routing, a server that deals with basically metadata for a whole cluster of containers that tracks ancestry of containers and snapshots and what comes from what that is used by the distributed storage system and a cluster job and container scheduler which was going to be run in C++. So that's a win. And some more to come which as we'll continue to use Rust we'll see new things coming along. So some of the tools that we've developed or used is our fork of Apache Thrift which is an RPC system which is available on GitHub and also on craze.io and that enables basically Rust support. So you can write your services in a single language agnostic Thrift file and you can generate, you can write your server in Rust that you can also, there are all kinds of, you can write clients in many different languages including Rust. Shout out to Spackler for sending some patches in for this that enabled like way better errors and optionality support. It's turnall-thrift-on-craze.io. So I've talked a little bit about the internals of this before but effectively what we did is we have a very thin C++ front end that is integrated with the rest of the Thrift like cogeneration libraries and what that does is it just generates calls to Rust macros and the Rust macros actually do most of the heavy, like most of like the heavy lifting in the cogeneration but really we also use traits to sort of avoid generating a lot of code. The previous implementation used the C++ generator the whole way and it would kind of recursively generate a lot of code that was sort of repeated depending on your interface and by using macros and traits instead we can like force that, we can let the compiler do a lot of that work for us which is really nice. For the curious the macros are in code.rs. They're a pretty good example of kind of like a complicated use of macros that's not necessarily trying to add like new language features but is really just like expanding things. So some of the pain points that we had for using this are that the generate code takes quite some time to build. If you have a large interface you're generating potentially a lot of code that LVM needs to optimize and Rusty needs to do all of its stuff for. So one of the ways that we have gone around this is basically by faking incremental code generation by moving things on to its own crate. This is kind of a repeated thing that we found that is a good thing to do. It helps get around a lot of the problems or the not really the problem but the fact that Rusty kind of takes a long time to build like very large crates. You can break things out pretty easily. And we also used cargo as awesome like build system to be able to kind of build the like cogen and test the code like on every change to even a through file which is pretty nice. Generated code is macros. So sometimes when you're after you're trying to figure out like how are things actually work what is the exact interface that is actually generated. It's not super clear because even if you look at the generated code all you see is macro calls and those could expand to anything. So a nice solution to this you can just use cargo doc it'll generate nice really understandable output where you can see like the traits and how everything is actually generated not just the sort of unreadable macro spew. Another really interesting tool that we've used a lot that has great support is Postgres. So there's a great driver also written by Spackler which we've had really nothing but success in using. We find that it's like very well designed. The interactions are usually type driven which is really nice and that means that you can get rid of a lot of errors or just kind of force things to be compile time versus kind of we're rewriting one of our services that used to be in Python to use Rust instead and we found that we can get like we've eliminated a ridiculous amount of errors by just like introducing typing. We also use cargo's bin system where you can have like many kind of sub tasks or different things that you can use cargo to run for your project. We use those to like set up and tear down the database and do all kinds of operations like load the schema and stuff like that which is really convenient for kind of testing and local development that everything is in one place and you interact with the whole application through just one tool. We also use from the traits in this from SQL and to SQL which function is like a really, really super thin like very bare bones or like RM that you can use to sort of like isolate the places where you translate between code. I guess I can take the question. Yeah, so what you have to do, the question was how do we get the type information from the database? Do we like read the schema or something like that? So it's not, you know, it is not that well coupled yet. There are some kind of experiments where I'm sure it's a fact but I could tell you a lot more about the details here but there are some experiments where you can use a syntax extension to actually communicate with the database at compile time to get this information but the way that it's done right now is you basically just have to know and then you tell it what the types are and you can sort of isolate this into a small component and like using from SQL and to SQL and you can also create your own traits for like converting between a row and something like that but the interface basically you ask for something of a specific type and it will like deal with getting the raw value and turning it into a nice like type representation for you. You get a panic, which is a little unfortunate but you can also catch this in terms of results but for the most part you sort of just know the right answer so if you were gonna have a result you don't wrap it anyway and you might as well just use the panic and default. So those are some of the tools that we've used. I'm gonna get into a little bit into some tips and then like tricks and things that we've learned using Rust. So just some tips for writing reliable Rust. Use the type system to your advantage. Rust has a really excellent type system that you can encode all sorts of interesting variants and this can really, like this can go a long way towards just making your code a lot more like stable and easy to work with and like not have runtime errors. String typing causes an excessive number of bugs. You can almost always avoid it in Rust with the tools that the type system gives you. You probably should. You can factor code using generics and out macros so I know that you know we just talked about thrift that's using macros to do all this code generation but by using traits in the existing system we can actually avoid most of that code generation and for the most part generics are checked and that yields just much better code structure than trying to use macros to accomplish the same sort of things. You can keep unsafe, contained, and tested. This is sort of like a really important point if you actually want to benefit from Rust's safety guarantees. Sometimes if you're interacting with like very low level APIs you're calling it to see you need to do unsafe. It can be very tempting to just mix in these like FFI calls into your normal code especially if you're only doing a few but that usually makes the code a lot harder to maintain. It's a lot better to sort of segregate the unsafe into as small a section as possible and then have that expose the safety of the eye and like heavily test it as well. We use RAII and borrowing for basically all resource management. So not just file descriptors and memory you can really use it to manage all kinds of things access to the database like any number of examples can you use it for some forms of access control and a quick shout out to cow for being pretty underrated I think but actually really amazing for writing very flexible and very efficient code. Some points on maintainability. I'm gonna say avoid trade objects for the most part. They're not very flexible in the sense that you have to make the choice up front about exactly what traits are going to be included in a trade object and this can be really inconvenient especially with interactions with OIBITS which stands for opt-in building traits. These are things like send, sync. Those are the ones that come from the standard library at least so you have to when you make a trade object you have to decide upfront whether it's going to implement send and sync and sometimes you make the wrong choice and it can be a lot of work to go back and make this change later. Use generics, use the sites, the usage sites which you need send and sync can just add it just a lot more flexible than the trade object set up. Be very careful with panics, your writing code think about what happens if you receive a callback you have to be sure you have to be ready for that callback to panic and that yeah. You have to unify concepts using traits as much as possible. This allows you to as I said use traits and use generics to factor your code and kind of like drive interface design and design interfaces conservatively with a focus on stability. I'm gonna get into this a little bit more in a sec. Also use cargo for everything which I already touched on briefly but we'll get a little bit more into. So in terms of conservative interfaces a lot of this is take inspiration from standards. Standard is designed to be very easily extensible it's designed to be stable, it's designed to not break and to be more importantly for people to be able to add APIs and add new things without breaking old code and a lot of these lessons can also be applied to your own libraries. So some of these things are using private fields with accessors instead of public fields for the most part, it can be really nice to have a public field in the sense that you don't have to write a immutable accessor and immutable accessor and all of these things but often you actually only want to allow certain forms of access or you want to be able to change the underlying type to do something more efficient in the future and using accessors makes this all a lot easier. You can also much more easily rename things or deprecate certain things if they're methods versus fields. You can use from as ref and into to extend like pre-existing APIs if you have something that takes a concrete type and you want it to also be able to take like other variants of that type you can use these traits to like backwards compatibly extend your interface which is really, really convenient. Result is often better than option for basically anything that's not like a get operation on a data structure, something along those lines. Not only does result work with try but it also allows you to add information in the case of an error if you want to do so later. If you started with option moving to results is like almost always a breaking change whereas just changing the error type and your results are adding more information to it especially if it is already a type that only has accessors is really easy and usually won't break things. Errors should be types with typed metadata. If you use strings for your errors it is both basically impossible to catch errors and it's very easy to kind of break things and not be able to extend your error types very well. So touching on a little bit of getting the most out of cargo, there are many traits that I owe releases, this is sort of a thing that's kind of taking hold in the ecosystem at least for large packages that in their pre 1.0 versions they will try to not break on patch releases and only break on like 0.x releases. So pinning versions to like some like 0.x version will can save you a lot of breakage that comes from like the ecosystem even if it's like the standard library stable. You can use bins for utility tasks so we talked about this a little bit with Postgres and but basically using, you can use multiple bins, you can set up things to like run your tests, create databases, like tear down databases all that kind of thing. You can split code into crates plus path depths to reduce your build time and sort of fake incremental code generation or incremental compilation generally speaking which actually hopefully we won't have to do in the future as incremental compilation actually becomes a thing. And you can run all of your tests via hashtag test. We have at least one situation where we had a pre-existing test suite that uses like the Python thrift client and we wanted to just call that. So the easiest thing to set up is you just create a folder or a single file with one hashtag test and it can just call into the, you can call your Python test runner directly. And a little bit of general rest and production advice before I tie up the ecosystem is in a really early state, contribute. There's a lot of territory that's still unexplored and if you know your company is the one that is building those libraries, you can get a lot of recognition and a lot of goodwill from the community. The other side of that coin is that Rust has a really strong open source community and you should use it. Like as demonstrated by our thrift implementation, we threw that out there and we immediately started getting contributions from the community and those like helped us enhance our features a lot and save this bunch of work, which is great. You should write and talk about your experiences. If you're using Rust and production or really using Rust at all, you are an early adopter and your message is very important and a lot of people are interested in it. And the last thing, this is really useful. Use it liberally. You can just force everyone to have to document their code and you just can't land a patch if it's not documented, which is great. This has been Rust and Production, that's all. Thank you very much.