 Hi folks, welcome to another de-crusted stream in these streams We go over some, you know relatively well-known commonly used rust crates and we try to figure out both how they work How you use them common pitfalls We basically try to sort of peel back the the lid a little bit and make sure that you really understand the crate You're working with in this particular stream We're going to be tackling Tokyo the well-known asynchronous runtime in rust and we'll dig through all of the different Aspects of Tokyo that are important to know about and also they might come back to bite you if you don't know about them Or don't use them correctly We're not gonna go too deep into like how futures work like, you know, the future trait interface and Pinning and those kinds of things. That's not what this stream is about. I have another video It's a little old, but it's it mostly holds up that talks about that. We're also not gonna get too deep into like How to write an asynchronous scheduler? That's not the goal here. The goal here is for To sort of give a guide or walk through for those who use Tokyo, right? How do you use it? Well, and how do you make sure that you understand what happens? Why? Why am I go wrong how to debug it? So it's more so that then sort of reading every line of code in Tokyo, for example Before we start just two very quick things the first is I have a get up sponsors now if you haven't heard If you have a chance to sponsor me that makes me very happy This is not a I'm not in dire need it's more that if if you have a chance that might allow me to make more of this kind of content going forward and the second is there is now a Discord for me I suppose or for Jay rather that I Jay happens to allow me to be with him in in this in this in this discord And it's on discord dot John who dot EU that gives you the discord invite There's a bunch of channels you have access to Immediately off the gate including an announcements channel Which is where I post both whenever there are new streams coming up But also when there are talks that I give that have been uploaded to YouTube and the like There are other channels for actual chatting But those are only accessible if you sponsor me and so you know another minor incentive to sponsor All right, let's get into Tokyo. So Tokyo is On the face of it what it does is it takes things that implement the future trait So live if we pull up the future trait here It is just the thing that takes things that implement this future trait And then returns the thing that is in the associated output type that is all Tokyo kind of really does Right, it runs these futures and produces the output that that future represents the computation of In practice, there's a lot that goes into that kind of machinery And so when we go into Tokyo, I'm really gonna sort of split this roughly into four parts Let's see if it actually ends up being four but roughly four The first is about Tokyo the runtime like basically how does it execute futures? How here we're gonna talk about things like worker threads and the thread pool work stealing blocking shut down the sort of inner workings of the scheduler although not at the very very lowest level but enough that you Sort of understand how the pieces fit together The second thing we're gonna talk about is resources So this is anything that has to do with IO anything that has to do with threading processes the file system Timers as well. So all of the things that a future might use in order to interact with the outside world Then we're going to talk about utilities So this is more about how What sort of what additional Useful features Tokyo provides you with both in the crate itself and it's sort of slightly adjacent crates for things like coordination and synchronization between futures handling multiple tasks things like the select macro For allowing sort of whichever one of these futures finishes first so you can run multiple of them concurrently As well as things like the sort of Tokyo util crate We'll touch on a little bit on on bridging between async read and async write which we'll talk about and sink in stream Which we'll talk less about And various other things like that and then the fourth part is going to be talking about common Complications when working with Tokyo and I don't mean that as in bugs with Tokyo itself. I mean that as in Sort of things that are easy to shoot yourself on the foot With if you don't realize exactly how something works or or the caveats that comes with something the trade-offs that are involved with Something as we'll talk through some common one of ones of those That is That's sort of the rough outline here, so let's start with how Tokyo works as a runtime So in Tokyo First of all the documentation is quite good like I do recommend that in general Spend the time to read through the Tokyo docs They will they do talk about a lot of the things that I'm going to be talking about here It's just that you have to remember to read them in order to realize these things There's a lot of really good documentation in there So in Tokyo there's a bunch of sub modules and one of them is runtime and the runtime is sort of that that the Main the heart of Tokyo if you will it is the thing that you give futures to and it gives you back that the thing that the future computes and at its core The runtime and it says this as at the top here. It has an IO event loop this gets back to things like resources We'll talk about later Timers which we'll also talk about later and then a scheduler and the scheduler's job is to take futures that come in Call the pole method on that future So remember futures are these things that implement the future trait where async blocks and async functions are examples of things that Automatically sort of the compiler turns into things that implement the future trait The scheduler is going to pick a future that hasn't completed yet call its pole method The pole method just executes like a normal function and at some point the pole function returns And if we look at what it looks like The return value of pole is this pole type that is either ready Which means the future has finished it doesn't need to be run anymore and that includes then the return value of that asynchronous Task or pending which means I didn't finish. There's more work to do and when a future returns Ready, then Tokyo doesn't really do anything more with it Depending on how you gave the task to Tokyo in the first place It's either going to give you that T or that T is sort of going to be dropped into the ether usually because it's just units So no one cares about it anyway But when a future returns pending from pole What happens is that that future gets sort of put back into the runtime to be executed again Like pole for pole to be called again at some later point in time And we'll talk about how that mechanism works in a little bit more detail So the scheduler part is the first thing we're talking we're going to talk about There are two schedulers in Tokyo. There is a multi-threaded scheduler and there's the current thread scheduler for almost all use cases You want to use the multi-thread scheduler the multi-thread scheduler Creates by default at least one thread one operating system thread for every CPU core that you have And the idea here is that the different worker threads can be sort of in the process of calling pole on different futures at the same time And so that way you get to have actual parallelism Where these different futures are executing in parallel with each other So do you get to use the CPU's resources of your machine as opposed to for example Just switching back and forth between them quickly on a single core where you wouldn't really get the parallelism But you get the perception of concurrency The multi-threaded scheduler is configurable so you can choose how many threads you actually want But in general having one per CPU core is is usually the right choice There are some caveats to that when it comes to blocking threads that we'll talk to in a second When you have the multi-threaded scheduler actually I'll I'll I'll talk about the current scheduler first before I dive into either of these in more detail The current thread scheduler does not start any threads Instead what it does like it doesn't start one thread per core. It doesn't even start one thread It just uses as its name implies the current thread and it starts up a Tokyo runtime on the current thread That is able to that has this sort of IO integration and this little Q for which things haven't been run which things can be run And it just stores at all of that locally on the current thread It executes some future that you give it and ultimately blocks the current thread until that work is done and only uses sort of CPU cycles on the current thread and nowhere else in Order to drive those futures to completion We can talk about how these work internally and that might be a little bit interesting But before we do that I want to show you what this interface actually looks like So if you have a if you have a runtime and this is the same type whether it's configured as a current thread runtime or a multi-threaded runtime Wish this was the default view When you have a runtime you have this method called spawn which we're gonna ignore for right now And then you have this method called block on and block on is the the most straightforward interface to a runtime. It is also not the one you're gonna use most commonly, but it is the sort of most straightforward Interface into a runtime it takes a future that implements the future trait and it returns the output of that future That is all right So if you create a current thread runtime you pass it some future to block on then it will do all the stuff That's needed behind the scene until that future ultimately returns the ready variant of the pole enum And then it's gonna extract the value and give that back to you if you have the current thread runtime at that point The runtime doesn't do anything more right because it does not spawn any threads It doesn't do anything in the background So the moment block on returns the current thread is now under your control again And there's no way for Tokyo to run anymore logic because you're not in Tokyo anymore This function has returned and whatever is in your the sort of calling code is what's going to happen next And and this talks about a little bit what what that actually means So block on is sort of the main interface here to to Tokyo for turning futures into their their outputs In the multi-threaded scheduler And actually in the current thread scheduler You can call spawn and what spawn does is it takes a future and It puts it onto the the sort of queue of futures In fact the queue of tasks and I'll talk about the distinction between those in a second It puts the future that you give onto the queue of tasks for the current runtime Regardless of whether that's a current thread runtime or a multi-threaded runtime But it doesn't execute anything it just sort of puts it on the queue and then returns and it returns you a join handle So this is sort of similar to if you use thread spawn so if you use thread spawn from the standard library It spawns a new thread that is eventually going to run the closure that you give it and it gives you back a handle to that thread that you can do you use for example to Wait for the thread to complete and get its result and the the semantics here for a join handle are very similar So the join handle you get back from from calling spawn is something that you can use to Like abort the tasks you've given or you see it in it itself implements future And so you can await to the join handle you get back in order to wait for the task the future that you spawned To yield its value eventually and and get it back If you drop the handle nothing happens there's no there's no automatic abort when you drop a join handle This is the same thing as with Standard thread spawn the join handle you get back or the thread handle you get back does not do anything if you drop it The distinction between spawn and block on as I mentioned is that block on will block Where you call it until the future is resolved spawn will just sort of throw it out in a queue and then return Every future that has been spawned onto a runtime becomes a task That does not mean that every future that is on a runtime is a task And the distinction here is that futures can contain other futures So imagine that you have you know you write async and you like an async block and inside of the async block you You call some asynchronous function That asynchronous function itself creates a future and maybe inside of there. There's an async block It may be inside of there. It calls. I don't know It tries to read a file asynchronously or tries to open a socket Whatever it might be that is also a future And so these are futures that contain futures that contain futures that contain futures And when you await one of those futures You just sort of resolve that sub future But the larger future you're inside of is still there and has not returned ready yet It's only when the outermost future returns That that sort of the that task has completed. And so again Only the thing only the futures that you pass to spawn become these top level tasks And tokyo only knows about the top level it can't see inside your future types and see all the inner futures It only sees the top level tasks So the futures that have specifically been passed to runtime spawn or if you look in tokyo itself, you see there's a macro called spawn And the spawn macro Under the hood really does the same thing it gets a handle to the current runtime And then it spawns the current the future you give it onto that runtime It gives you back the joint handle and so it's only things that have been passed to spawn that turn into tasks In addition to you know, the the thing that you pass to block on also becomes one of these top level tasks And the tokyo scheduler only knows about these top level tasks When the when you have a thread in tokyo regardless of whether it's the current thread or a multi-threaded runtime And that thread is that worker thread is looking for the next thing to work on It will look at the set of tasks not the set of all like nested futures um And the thinking here right is that the only entry point the tokyo has into your futures is calling the poll method On the the future that it was actually passed and then that sort of cascades all the way down into polling all the inner futures But but the real interface is just the poll call on the on the topmost task. Uh, the topmost future So now we get to the distinction here between the current thread runtime and the multi-threaded runtime In the current thread runtime if you call spawn nothing happens You it gets put the the future gets turned into a task It's put on the the task queue and then it returns The future does not actually run in any meaningful sense It is only when you call block on that you're sort of giving over execution control to the current thread runtime to tokyo at that point with the current thread runtime tokyo is going to look at the set of Tasks that it has execute them Uh until they yield either poll ready in which it's going to return or yield a poll pending in which case They're going to put them on the like not ready task queue and then it just keeps doing this in a loop It keeps calling uh poll on all of the tasks that it knows about Until it gets to a point where the future that was passed to block on returns ready and at that point it returns And if there are tasks left in the queue it just leaves them there and then does not execute them any further So it's only in the context of block on the things in the task queue Actually run if you have a multi threaded scheduler, which again is the default It's also what you get if you use like the tokyo main annotation On a function Then what will actually happen is you have this pool of worker threads, right? So when tokyo starts up starts one worker thread for every cpu core you have That pool of worker threads Each one has its own local queue of tasks that it knows about and there's also a global queue that they can all read from And the idea is that every worker thread is just in a loop And what they do in that loop is they look at their local queue and see if there are any tasks that can be executed If all of them are pending and there's no reason to continue assume that you might make progress if you poll them Then it looks at the global queue and sees if there are tasks there if there are no tasks there either Then they can actually steal from each other. So it's a work stealing scheduler so, you know, if you have let's say you have two cpu cores and you know, this thread is has has No tasks to run and this one has a queue that's full of tasks that are ready to run when this one realizes Oh, I don't have any more work. It's actually going to start looking at this other worker threads Queue and start to steal some tasks of it so that you're better utilizing your cpu resources So this means the futures can start to move between threads and this evens out load between your cpu cores so that you get to Make more use of the compute resources resources you have but it also means and this is sort of where the That requirement that is sometimes annoying comes from is that futures must be send They must be allowed to move between threads They must implement the send trait in order for tokyo to be able to do this load distribution And because tokyo uses the same interface for current thread runtimes and multi-threaded runtimes All futures you pass to tokyo have to be send regardless of whether you're actually only using the current thread runtime There is a way around this. We'll talk about that in a second about local sets But this is sort of the the basic premise for the setup Before I now talk more about, uh, how Sort of how this interacts with blocking When you can block threads The co-op budget block in place Shutdown that kind of stuff does the general sense for what a runtime looks like makes sense Let's see How does this model work with blocking work? Let's we'll talk about in a second What's the use case for the current thread scheduler the use case for the current thread scheduler? So there are sort of two One is less so than the other the first one is you just Uh, like let's say you're running your test suite every test is annotated with like tokyo test Um, and you don't really want every test to spin up because you're gonna run them all in parallel You don't want every test to spin up n threads Where n is the number of cpu cores because now you can end up with n the times your number of tests Um, as you might use the current thread runtime there in practice And I think tokyo the tokyo test annotation does is by default now It actually starts a multi-threaded scheduler with only one thread And so that is very often the thing you want instead of a current thread scheduler Just so that you don't have the sort of complexity of having to manage multiple schedules and test multiple schedulers The real reason for using the current thread scheduler is if you have things like You want to limit the amount of movement that happens between threads Because you have a workload where you know that it's very costly if you're executing some sequence of instructions on one cpu And suddenly they get moved to a different cpu right that ends up sort of making your cpu cache utilization worse sometimes you have Sort of cpu pinning that you do so that all of your workload happens on a particular set of cpu cores And you don't want the other cpu cores to be used at all for some reason Maybe you've dedicated them to some other worker pool So the current thread runtime gives you more control over exactly what runs where But it does come with the cost is like it's it's much less of a general purpose executer Why would I use tokyo spawn over rusted thread spawn So so the difference there is that tokyo spawn lets you give a future The standard library thread spawn spawns an operating system level thread So you give it a closure and it executes that synchronous closure. There's no async involved Um Is tokyo main also a task? Yeah, so so this is actually a an interesting thing to look at so if I go here and I do cargo new been Kyoto, aha Kyoto Yeah Because tokyo, right key. Oh, anyway, um, let me do add Tokyo here with features full Source main and I'll just do tokyo main async fn main And then I run cargo spanned. Let's see what this gives us So this is what we actually get when we run When we add the tokyo main annotation to an async fn It really just replaces your main function with a non async main function It takes the body of the main function that you gave and turns it into an async block Uh, and then it creates a tokyo runtime Enables all the features on it. It's a multi-threaded runtime and then it passes that body to block on And so in other words when you use tokyo main all it turns into is that future just becomes both a top level task And it becomes the main thing that the the runtime is blocking on And so when the main function returns when the future that is in your async fn main Is is completed it returns ready Then the whole runtime is considered done and yields Control to the actual main function which then returns and actually and closes your program Do you know what the tokyo approach is regarding the work stealing logic? Because when we think about work stealing we think about costly context switches potentially So there's a there's actually pretty decent documentation in tokyo about exactly how it does Work stealing if you go to tokyo runtime you scroll down a bunch and you go to Multi-threaded runtime behavior at the time of writing This one talks about the the different cues that they have and what the policies are for when you look at the local cue When you look at the global cue when you're allowed to steal and they basically try to balance The cost here right because there is a cost to stealing but there's also a huge benefit to stealing And so you you you basically tweak the parameters until it hits something that generally works And there's a bunch of configuration patterns Configuration parameters you can set if that the default behavior doesn't work well for your application But you should really benchmark and make sure that if you change these settings It it's actually under like real load realistic load That these numbers make sense Uh Why looping on pending check when available instead of park and unpark like threads So it's not actually a loop on just if it returns pending than loop. Um, this is where we get into the the idea of Wakers and and notify, uh, which is sort of the asynchronous version of park and unpark Um, so so it is actually more sophisticated than that which we'll get to when we talk about resources a little bit later How that mechanism works for for choosing which futures have become ready So in general the sort of logical way to think about it or the theoretical way to think about it Maybe is that every worker has uh, or every runtime really has two cues one of them is a a cue of runnable tasks and one of them is a cue of Not runnable tasks Or or pending tasks if you will but not runnable tasks The when you choose which thing to execute next you only look at the things in the runnable task column in that in that cue And those are things where you know that if you call poll on them, they're going to do some useful work The ones that are in the non runnable cue Those you have no reason to believe the calling poll will make make progress on and so you just don't do it And then there's this mechanism that moves things from one cue to the other so from the non runnable to the runnable cue Whenever there's a signal that indicates that this future this task may now be able to make progress Again, we'll talk about that when we get to resources And when you run a future from the runnable cue and it returns pending you move it to the non runnable cue So you don't try to execute it again And if a if a worker thread runs out over the runtime runs out of thing in the runnable column Then it will actually park itself And only get unparked when there is Some future some task that has moved into the runnable state again Why not spawn more threads than cores? So there's no real reason to have more threads than cores, right? The number of cores is the amount of work you can do in parallel anyway Um, and so there's there's not really a benefit to having more of them Uh, current thread schedulers also important if you're on a system without multithreading Yeah, so so things like in wazi where there are no threads The current thread scheduler is the only scheduler you can have because there aren't threads You can't spawn them and so you can only execute in the the current execution context anyway Okay Ah, so kernel cpu scheduler So the operating system cpu scheduler also implement work stealing In the sense that like if one cpu core is executing a thread Then and another cpu core, you know wants to choose something to work it can steal the The threads that are sort of locally scale locally queued on some other cpu Sort of right so so basically the observation here is that threads on this os level can move between cpu cores And that is totally true It's not usually work stealing but but it does mean that the threads can move in this way Um, the difference here is that the operating system is not aware of futures It doesn't know about different futures different tasks Um, and so you could spawn one thread for every single task that you have the problem is every time the Kernel wants to switch between threads It has to do like a kernel level context switch which are fairly expensive whereas futures are a A user space concept not just a user space concept, but it's a rust based concept And so you don't really need to go via the kernel So if you instead have your own scheduler that moves a single operating system thread between multiple tasks You can do that way more efficiently. And this is sort of the whole the idea behind green threading Um Is the queue of tasks an os based event queue No, so so these are actually just just You can think of them as linked lists that tokyo keeps internally Of tasks that it just you can atomically add things to either queue There's no operating system level part of of those task lists The only place with the operating system comes in is in spawning these These different threads the worker threads initially And some of the mechanisms for dealing with io resources and timer resources Which we'll talk about in a second Okay, great So now it seems like there's a sort of general understanding for what the the runtime here is Let's then talk about blocking So when you given that we only have these n worker threads where n is your your number of cpus or whatever value you configure it to Where you run into trouble is if you pass a future an async block and async function to tokyo And one of the worker threads is executing that future And that future ends up calling something that blocks the current thread think of like it calls Let's see it tries to read from from the standard in from the terminal But it doesn't do it through like an asynchronous stood in handle from tokyo. It uses like stood io stood io stood io stood in And does like a normal read call. Well, that's going to block the current operating system level thread This is not a tokyo concept. You're just doing something that blocks And at that point you're not just blocking the execution of this future You're actually holding up that entire worker thread It has no way to now start working on some other task because the operating system level thread is blocked And so this is why it is really important that when you have asynchronous futures that you pass to tokyo That you don't have them do blocking calls Because if they do they're going to hold up the execution of any other task on that cpu core Again because the number of worker threads you have is just the number of cpu cores uh, and and this is not a sort of hard limit right like what does blocking mean because Reading from standard in just means you're blocking until the user presses a key So it's not an infinite block very few blocking things are In general what we're talking about is just you should avoid holding up a worker thread for a very long period of time And what very long means is sort of up to you But imagine that you have a you have a system where like a bunch of your asynchronous futures all do some kind of blocking call And those blocking calls take i don't know they take a second for whatever reason If you let's say you have two cpu cores, uh, both the both of them end up executing futures that have this one second block Then now you're going to have some periods of time where for You know 500 milliseconds at a time or something No futures are being executed because both of the worker threads are held up in the blocking call So even though there are lots of other futures that could be making progress There are no worker threads to pick up those tasks And so that's why when we talk about you don't want to block in an asynchronous context We don't really mean like you can't do an operating system call that might block We're talking about you don't want to hold up any worker thread for a prolonged period of time because that's going to mean Not picking up other asynchronous tasks for that period of time So so even if you do just very um compute heavy work So let's say you're not actually pausing you're not blocking you just have like a matrix multiply or something that takes like two or three seconds Then that still has the same property that you're holding up the the pickup of other tasks Uh, and so there isn't really a rule here for how long to wait It just depends on your tolerance for futures not being executed the general rule of thumb. I've seen is that if you have Chunks of code in your futures so that is Code that executes between two await points with no awaits in this critical block If you have blocks like this that take more than 100 milliseconds to execute You should probably move them to a separate blocking thread or use block in place Basically, you need to tell tokyo. Hey, I'm gonna do something blocking now Do that in a different thread or We'll talk about block in place in a second, but basically you need to notify tokyo I'm about to do something that's gonna take a while on this thread um, and and there are There are also calls that are Fast most of the time and sometimes really slow So you can imagine things like taking a lock And I'm not talking here about the tokyo mutex. We'll talk about the distinction in a second. I'm talking about standard Sync mutexes If you take one of those usually it's super fast Usually you take the lock you get the lock and then you do some work and you release the lock There's no real delay But it could be that you're just like some other thread in your system has the lock and it's doing something that takes like 10 seconds And so now your thread the text to take the same lock is actually going to block for 10 seconds while waiting for this other thing To release the lock and now you're holding up your thread for that period of time as well And so What we're talking about when we say blocking is like in the worst case does this take too long Or longer than you're willing to wait for your other futures to be picked up Um, there are two ways to deal with this problem. One of them Uh, where's my runtime here? um One of them is this spawn blocking thing So tokyo actually has two worker pools. It has two thread pools One of them is for the workers. So these are the things that execute asynchronous tasks The other is a blocking thread pool. These are threads that Are expected to block. They're just doing something that we know is going to block the current thread We have a huge like the the limit for how many blocking threads you can run is like super high We don't really cut you off at any point that there is theoretically limit But in practice we assume that most of those are sort of sleeping or something And that pool is initially zero or rather that's not quite true All of the worker threads are actually in that pool. Well, let's ignore that for a second The the blocking thread pool is a place where you can spawn your own threads using the spawn blocking thing And this is almost like using std thread spawn. That's basically what it turns into under the hood Except the tokyo sort of remembers that thread for you and deals with it when it comes to clean up a shutdown for example but but um, but this is really just an interface for like Spawning uh blocking synchronous threads The reason it's a little more efficient than using the standard thread version is because tokyo actually knows to reuse these threads So if you call spawn blocking and something runs and then finishes and you call spawn blocking again tokyo might realize that oh There is already a thread in the blocking pool that doesn't have any work to do We're just going to schedule this closure on that same thread And there's some policy for when threads get um, sort of actually shut down from this pool Um, but so spawn blocking is one way in which you if you know that you're going to do some kind of blocking work You can use spawn blocking to take that closure Spawn it on one of these blocking threads and you get back a join handle That is a future that you can then choose to await whenever you want to wait for that thing to have finished And you can await it immediately after spawning, right? That's fine because you're awaiting Which means that you're going to return pending which means other futures are going to get to run While that blocking task is executing out the other pool The other thing you can do if you're not willing to take, you know, whatever the whatever the Thing you want to execute blockingly is if you're not willing to send that over to another thread, for example, because it's not send What and and you'll notice here that this indeed requires that the closure you pass in is send is you can use block in place so in tokyo task block in place this is a A function you can call you pass it a closure But this closure does not need to be send and what happens when you call this function is that tokyo takes all of the state that the worker thread has and sort of saves it somewhere And then it Which you can think of as when it saves it somewhere actually sort of gives it away All of the state related to this worker thread and it becomes a non worker thread And it sort of gives tokyo the green light to start a new worker thread in its place And so that way this this thread that was a worker thread Stops being a worker thread and gives its its worker state away to tokyo back to tokyo And then it just continues executing your blocking code And at this point you're not holding up any other futures because some new thread the tokyo spawns Can pick up the worker state and then resume executing all of the futures from here So you can think of it as a spawn blocking if you have this as the worker thread It starts a new thread and moves your work over there Whereas block in place is you have a worker thread You start a new thread that is the worker thread and you continue your blocking work here So they are sort of Symmetrical if you will in that sense now Block in place comes with a couple of performance things to be aware of This is fairly subtle stuff. So I recommend reading the the docs for lock in place In general only use block in place if you truly need to do some blocking stuff that is not sent But when you do that's what it's for Okay, let's let's pause there before I continue on on runtime stuff How much sense does this make? How much overhead would using async in an embedded system with one core b? There is overhead to using asynchrony right because you now need to have this Stuff that keeps tack is a track of the Uh Of the task and it chooses how to schedule them the overhead is less than you might think um, and usually the overhead sort of Uh, you you can make up for it by the fact you get to use green threading rather than going through the kernel if you do a lot of context switching, but But in terms of performance like async is usually slightly slower um the difference again for exactly the same workload assuming that Assuming that you can use can't use threads Where where asynchrony shines is both if you have a huge number of tasks So think of things like you have a task for every connection and you're expecting You know tens of millions of connections or something then doing them in a green threading approach rather than than going through the kernel every time Does end up increasing your overall performance The second reason if you want to do things concurrently So one pattern that we see a lot when used in in asynchronous context is that you want to like wait for Either a tcp's packet to come in or the user to press something on the keyboard If you were in the threading model, you would need to have one thread the blocks on a read from the tcp socket And one thread the blocks on a read from the the standard in and then you need some way for those threads to synchronize In the async world each of those is just a future and then you can use select which we'll talk to in a second So the one task Awaits either of the two things and you don't actually need two separate threads that then need You know atomics and or in order to uh to synchronize because they're happening on one task So there's no synchronization needed um And very often programs end up wanting that kind of um of concurrency And and it's actually I would argue easier to write programs in this way when you have a lot of these uh either or events Um Does tokyo create a thread pool and automatically spawn more tasks if needed or is this something one has to manually set? um When you start tokyo by default with the it will use the multi-threaded scheduler Which starts this thread pool for you with n worker threads where n is the number of cores And with a blocking pool that has no threads and then it will start blocking threads as needed whenever you call spawn blocking Uh, are you planning to talk about cancellation? Yes, I am uh Is it over at in promoting the threads right so this is the You call block in place So you you take the worker state and you give it away to tokyo and then tokyo is going to start a new thread That picks up that worker state and the answer is yes There is some cost in starting up a new thread and then it picking up the state In practice tokyo is actually a little bit smarter than what I just said What will happen is when you call block in place the worker state is saved in tokyo This thread runs its blocking code and when it finishes it'll look at has another thread picked up this worker state yet And if not i'm just going to take it back And so that that sort of minimizes the cost for short blocking sections um So you can often avoid this this overhead of um moving the execution What does the worker thread state contain? It's mostly the queues of tasks Uh, and a couple of other things. So it's mostly the queues um How much of this stuff we're talking about can be generalized to other asynchronous run times A lot of this applies to other asynchronous run times as well um the exact nature of like how they manage blocking Whether they do work stealing whether they have pools and how large they are will vary but but many of the concepts translate Um So the programmer needs to be aware of the functions that may block Can we instrument the code to send off blocking calls to separate ioworkers? um Yeah, you you can think of this as like In a certain in in asynchronous context you should not be doing significant amounts of work without calling a wait um Like between any two await points you should not be taking You know hundreds of milliseconds or or many seconds before you call the next await because chances are you're holding up the runtime Detecting this automatically is super hard. I'm not going to claim it's impossible But I haven't seen any sound strategies for doing it automatically Which means that it becomes a sort of manual task. There are tools like tokyo console will highlight when these kinds of things happen Um, but but you still then manually need to figure out why and how to rectify um Doesn't it still need to be using an os level asynchronous primitive in the end? Yeah, we'll talk about that when you get the resources Okay If a worker thread is blocked by synchronous code, wouldn't it be fine because we have other worker threads? It's fine until it's not right So if you have a bunch of futures or want many instances of a single future all of which may block at some time What is inevitably going to happen is you're going to get at some point in time All of the worker threads are in the blocking section and so there are no other worker threads anymore So that only works if you generally don't run into the situation Um, but but you generally don't want to rely on that being the case that there's always another thread Okay, um, there are then three more things to talk about with the runtime the runtime is the biggest segment of this I think the first of them is Shutdown so when you have a runtime Down here, um, when you have a runtime The general rule is that whenever the future that is passed to Block on resolves finishes Then the runtime yields control back to its caller and like we saw in the expansion here When you use tokyo main that means that when block on returns the whole main function returns And so at that point the program exits and when the program exits the operating system shuts down all of the threads Um, that may or may not be what you want, right? This means that when the main future exits everything else just gets dropped, right? there are ways to try to mitigate this um by having the main the Sort of thing that you do in the main future actually wait for for example all the join handles So you've gotten back from spawn so that you make sure that you don't actually exit until all the things you spawned have finished There are some tools that can help with this. We can talk about that later on when we talk about utilities but but you also have the The means to shut down a future a runtime by using a shutdown background or shutdown timeout Which lets you give a grace period as well when you call shutdown background what actually happens is that The moment things that are spawned yield They will then be dropped So it won't immediately shut down because there might be futures They're still in the call to poll but when the call to poll for every spawn future returns That future or each future as it happens to them will then be dropped and not executed anymore Even if they haven't returned ready yet And so the moment you call shutdown background The the number of tasks is just going to steadily decrease until eventually there are none left and at that point Shutdown background will return saying okay. There are no more futures now. You can safely shut down This does mean that they are not guaranteed to run to completion, right? They will just be dropped They will be cancelled like we'll talk about a little bit later Um But these are in some sense a graceful way to shut down if you instead just return from So we had that here If you instead just return when the main future returns or it resolves Then at this point after you finish main after you return from main All of the threads are just killed by the operating system What that means is the drop method will not be called on the future So they don't get a chance to clean up from after themselves That might not matter to you But if it matters to you chances are you want something like shutdown background as a call at the end of your At the end of your main to make sure that you actually wait for the spawn tasks to also finish and at least get to get dropped The next thing I want to talk about is send bounds. So Tokyo does have a way to run Futures that aren't send So these are things where once you start executing them on a given thread You are not allowed to move them to another thread. In fact, you're once you've constructed them You're not allowed to move them to another thread. This obviously sort of Doesn't really work with work stealing right because work stealing assumes that you can move threads to another To another task So Tokyo has this type called a local set and a local set is a set of tasks that are all guaranteed to be executed on the same thread So the idea here is that you create a local set and then you can spawn things Using the spawn local method onto the local set and that adds the future to this local set But again, the local set does not send it's guaranteed to not move between threads So these spawn locals just creates additional top level tasks for the local set And then you can await the local set at the end And that will run all of the futures that have been spawned on that local set until they all resolve The trick with local set though is that it cannot be used anywhere You can't just like arbitrarily use them somewhere deep down in a future somewhere And the reason to that is if you can start the local set deep down in a future stack Tokyo doesn't know that there's a local set there So like when it call when it all it sees again are the tasks at the top level So up there it doesn't know that this thing can't be moved between threads because that would move the inner local set And so local set can only be used in top level tasks. It can only be used as you'll see here On it passed to block on Or Yeah, so remember tokyo main basically ends up calling block on right so here we're allowed to use the local set because we are in a block on You can also use this run until which is sort of a block on on a local set But let's ignore that for now So you can just await this inside of a block on and if you Need to be able to use it somewhere else what you actually need to do is you need to spawn your own thread That owns the local set and it locally Executes with its own current thread runtime and then you send tasks to it over a channel It's a little awkward But again, it is because the the it needs to be known at the very top level that this task is not safe to move between threads If you truly need something like this if you need nonsense futures I recommend you read the docs for local set because there are a bunch of nuances here That we don't quite have time to go into But but know that this mechanism does exist And then the last thing that I want to talk about when it comes to the runtime And then I'll I'll do questions after this last one is the difference between a tokyo mutex And a standard library mutex So and again here you see there is a giant section that talks about this in the docs But they're you know the standard library has a standard sync mutex and tokyo has its own Tokyo sync mutex and you might wonder well, why do we have both? Why do they differ? Well the tokyo mutex the sort of main difference at a High level is that the lock methods on the tokyo mutex are asynchronous, right? So if you call lock you then call data weight great that seems straightforward. And so you might then assume okay, I should just use tokyo mutexes everywhere and That's not quite true and this ties back to the the nature of how these runtimes work and the nature of blocking so When you take a standard library mutex What you're doing is you you might block the current thread But let's assume that like this mutex is only held for a short period of time in general So you take the mutex. It doesn't really block the thread because you got the mutex right away You do a little bit of work and then you release the mutex there's not really a lot of risk there because As long as your critical section is not very long and as long as your your mutex is not very contented So you're never wait like, you know a second for it to become available That works fine, right? The fact that this is using a synchronization primitive at the os level tokyo doesn't really care It just cares how much time you spent in there How much time you spent holding up that runtime thread that worker thread And it turns out that a standard library mutex is actually really efficient. Um, like just just performance wise, uh, it it is, um Uh, it is a it is efficient synchronization mechanism The tokyo mutex in order to make lock and unlock asynchronous is actually fairly inefficient And so when you can use a standard library mutex, you probably should because it has lower overhead than the tokyo one Where this gets subtle and the reason why tokyo provides its own mutex It's not just because it's nice to be able to call lock that away But it is because if it might take a long time for you to get access to the mutex to lock the mutex Then you're holding up the worker worker thread. That's not okay. So you need to use a tokyo mutex Or if you take the lock and then you call a wait on something before you unlock Because so the way this would actually look is, uh, imagine in here Right. I do something like, uh Mutex.lock.await So that gives me back a guard Uh, that's not how you spell guard. Uh, and then I do Some funk.await And then I drop the guard Right, if I do this The problem you run into is that some function here Could take an arbitrarily long amount of time Maybe it tries to read from standard in maybe it tries to wait for a tcp packet Maybe tries to send a tcp packet to a peer that's disconnected, whatever it might be This can take an arbitrary amount of time What that means is now that you hold the lock um For the period as that this blocks what will happen is that you're uh, you're holding up Well, so so if this was Uh, if this was not Uh, um, tokyo mutex if it was a standard library mutex, then during this time you're still holding the mutex Even though you've yielded and you're trying to now execute other tasks Imagine that there's another task on this runtime that now tries to call mutex lock It will now be blocked forever or at least until this await call returns Because it can't get the lock because it's held up by this await call And actually gets even worse than that which is that a mutex guard in the standard library is not send But if you're holding it across an await point That means that this entire future is not send because it might need to hold on to that guard until this await comes back Because who knows we might do you know guard dot foo equals 42 down here So during the await call it needs to hold on to the guard in here and guard is Guard from the standard library is not send and therefore this whole future is not send and therefore you can't use it with tokyo And so there's some really tricky situations that come up the moment you have await calls inside or during which you're holding a standard library lock The tokyo mutex does not have this problem The tokyo mutex guard that you get back first of all is send So if you hold it across an await point, then it's fine because the mutex guard is send and so the future remains send But the other reason is because when you call await here tokyo knows That this future is actually holding on to that mutex And so if another task comes along and tries to lock that same mutex And it calls await right then tokyo knows Oh, I should put this task to sleep and pick up something else instead Instead of in this case where to block that worker thread and therefore block any other tasks that might be executing So the rule the general rule here is that use a standard library mutex If you have short critical sections like the the Stuff you do while holding the lock is short and there are no awaits in the critical section for anything else use the tokyo mutex Okay, I think those are all the things I wanted to say about the runtimes Let's do questions about the runtime before I move on to io resources um Might it make sense to have two separate tokyo runtimes and spawn tasks of those runtimes to increase performance assuming we assign separate cpu's to those runtimes It usually should not make sense to have two separate tokyo runtimes Usually you're better off just allowing work stealing across your entire task pool There are some exceptions to this so for example If you have an application where you have Some tasks that are all very similar to each other and some other tasks are all very similar to each other You might get enough and the performance requirements are strict enough You might get some benefit from having one task pool That only runs futures of a certain type and another one that only runs futures of a different type So that you get better cache performance on your cpu's But this is one of those like you should be damn sure with performance benchmarks and profiling that that is actually worth the complexity Because once you manage multiple runtimes, it's really easy to shoot yourself in the foot So I would very much default to just having one um Why do they not use implicit return in the tokyo main macro when you expand it? Uh, you know, I don't know I don't know why they don't use uh An implicit return here. I don't think there's a great reason for it Um Is local set slower than regular tasks in the current thread runtime? No local set does not as far as i'm aware come with a performance penalty Uh when sending through channels, you probably need futures to be sent anyway This is for local set. Um, if you want to sort of spawn it on a separate thread. The idea here is that uh There are cases where you can make the description of what to do Be send but have the execution of that thing not be send And then you should be able to still use this channel in local set turkey um I've heard the term green thread, but never fully understood it is just a task in a task q slash event loop Yeah, you can think of a green thread here as basically being equivalent to a task the way i've described it so far Are there any tools you can use to discover or Visualize how often your async tasks are being blocked and if your cpu bond tasks should be awaiting more Yeah, so that is the tool called tokyo console. Um, so if you look up tokyo console um This thing right here Um, it has this this view Under extremely cool and amazing screenshots That shows you for all of the tasks that tokyo knows about how much time has been spent busy How much time have they been spent sort of scheduling? How much time has it been idle? How many times have they been pulled? And it also has information about tasks that have been that are blocking the the runtime for a longer period of time And it shows these little warning icons for those Um, so that's a great great tool for for learning this information about tokyo Do ior in resources include tokyo fs? Yes, it does Uh, is the tokyo lock guard dropped when another await is called in the future? No In fact, it would be unsafe to do so. So, um, if you had Guard is mutex dot lock dot await something dot uh await drop guard Um, if if this await caused this guard to be dropped that would mean you release the lock and then call this thing But that's not the semantics of this code, right? The implication here is the lock is held during this entire call Uh, and so it's not okay for tokyo to drop this guard sort of implicitly just because you're waiting um When using a single threaded runtime and spawn blocking it still spawns a separate os thread Yeah, so spawn blocking will will always happen on a separate thread. I forget whether With the current thread runtime there even is a blocking pool I think there is Um, but i'm not sure you would have to double check that. Um, there are no task priorities in tokyo's you can't say This task is more important than this task will schedule it more regularly. Um, that's not a feature the tokyo currently has Can a worker thread steal from a currently blocked other worker thread It's complicated So if a worker thread is currently blocked because the future called the some kind of blocking system call The queue of that worker can still be stolen from But the currently executed task the one that is blocked Can not be stolen because it's currently active in that worker thread. There are also some other things that are I don't want to say hard to steal, but there's some optimizations. Um, that Means certain other tasks may also not be stealable. Um Uh, nope. I have not switched to emacs. Uh Uh Can there be a deadlock or similar if you share the same task between multiple runtimes? You cannot share a task between runtimes. So a task fundamentally is a single instance of the future trade Um, and so there's no way to share that right Whoever owns it, whichever runtime owns it is the one that will be executing it. Um, so there's no way for one task To be handled by multiple runtimes. Um, if you do have multiple runtimes, you could get into a Deadlock situation between the tasks of the two runtimes. Um, but that's sort of a different concern Uh, since tokyo shares the work across threads and the futures need to be sent This is that overhead every time a future is shared across threads There's no real overhead of futures being works stealable because tokyo will generally avoid stealing futures Unless specifically a core is running out of one worker thread is running out of work And another worker thread has too much work for it And in that case you would rather it be executed even if it's with a little bit of overhead Then just have one thread execute and the other one just sit idle and do nothing So usually it's worth it even though there is a small overhead when you choose to steal Um Is the way to shut down tokyo runtime and wait for all current pending tasks Not to my knowledge. Um, but the way you do this is you implement this yourself And we'll talk about some of the ways in which you can do that There's some utilities in tokyo util for example, which is a separate crate that helps you do this kind of stuff Is parking lot mutex faster than stood mutex, I'll leave that for a separate conversation because it doesn't have to do with async What are the advantages of using the local set is that it allows nonsense futures Um What's the difference between tokyo's instant now and the standard library instant now they're they're the same instant types So there's no difference. Um I got your email. I've just been busy Um Can you use tokyo resources with a different runtime? We'll talk about that when we get to resources Um Okay Yeah, so if you're running on particularly newer machines, uh that have large numbers of cpu cores the cpu cores might physically be located far from each other on what these things that are known as numa nodes And stealing across numa node boundaries can actually be quite expensive Much more expensive than it being worth it. Um, and again tokyo tries really hard to not steal unless it's necessary But but there might be an outsized cost on these high cpu core count machines for stealing There isn't a numa aware scheduler in tokyo at the moment But that is something that i'm guessing tokyo will probably grow over time Uh, tokyo does not have its own instant type Great Okay, uh, I think it's time to move on to resources So we've talked now about the runtime Uh, but the runtime is just about executing futures like calling the poll function and nothing else In reality, you generally want these these futures that you spawn to interact with the real world You want them to do io things like, um Okay, wait someone is insisting that tokyo has his own instant type Uh It is just a standard time instant. There is no difference So yeah, you're right. They do have their own instant type, but it's just a standard library instant type Um, I believe yeah, so here if you look at, uh, the note that explains why Uh, okay So tokyo has, uh, these resources that are intended for interacting with io This is stuff like tcp and udp file system other processes signal handlers All of that kind of stuff. Uh, i'm not really counting synchronization primitives like channels here, but they are in some sense resources as well um, and the The way to think about these is that these are provided by tokyo Because if you let's say you have a tcp stream, right and you want to read from a tcp stream Uh, so you have one of these and you want to call Uh Where is um Async read so async read is the sort of underlying trait that tokyo uses for Things that that can be read from, uh in an asynchronous context So you have a poll read that reads stuff out of you know, whatever type implements it like tcp stream Into a provided buffer and then returns poll to either say, okay I successfully read a bunch of bytes or it returns pending saying, uh, oh I tried to read some bytes, but there weren't any bytes available. And so, you know yield in this future and come back later And ultimately if you think about the sort of stacks of futures that make up a task Generally near the bottom is where you're going to have these resources that ultimately some some giant future Whatever its construction might be at the bottom you have things like read from a socket Write to a socket read from a file write to a file read from standard in Write to standard in or wait for a timer So you have these resources all the way at the bottom that are in they're not actually futures Right. There are things like async read that have a an async a future compatible interface, right? So they they have a poll method They're given asynchronous context and they return poll, but they do not themselves contain futures And all of these leaf resources these leaf, um Future like things are asynchronous primitives. Um are provided by tokyo And the reason for this is because whenever a future is blocked Sorry, I should say whenever a future is is pending It's yielded it it claims that it can't do any more work Then that is usually because one of these resource level calls returned pending That is the ultimate reason why the future can't the make any more progress And that also means that the way in which it can start to make progress again Is if one of those leaf resources suddenly becomes available Like you did a read from a tcp socket and now there is data available on that socket Or you try to read from standard in and now there is data available Or you try to wait on a timer and now that timer has expired like that time has passed or You know, you you were waiting for Uh You're waiting for a process to finish executing and now that process is finished executing And those things those events that indicate that okay now you should try this future again Is the connection between these um these io resources And the future traits that happens through this context thing and you'll see this concept argument is in async read It's also in all of the low level resource primitives that have these poll like calls, but aren't themselves futures It's also passed to future right so the top level call to poll on any given task So on any given future is given a context and that context is passed All the way down the the chain of futures and this mostly happens without you thinking about it So when you call dot await The d sugars to get the current context pass that into poll and then So it gets threaded all the way down on your behalf Which means that the context to get passed in at the top to a task is the same thing that makes it down to the resource at the bottom And context is defined in the standard library and the main thing that a context has is this waker thing and a waker Only has one real method that matters which is wake So a waker gets passed all the way down to the resource and the the idea is When a future returns pending so the whole task becomes pending it's moved to the sort of non-runnable queue and when the waker That was passed into that future when it returns pending when someone calls wake on it That means this future might be able to make progress again And that is so that this is a signal to tokyo that this future should be moved from the non-runnable queue Into the runnable queue and then get picked up by a worker pool thread again As you might wonder well who calls wake and this is where when we looked at runtime uh, you'll see up here that runtime has this i o event loop and this is separate from the scheduler the i o event loop and the scheduler they don't run on different threads necessarily, but they're sort of distinct components or services that are provided by the runtime the scheduler chooses what to pick up next And the i o event loop is looking for these events That might mean that some task can make progress and then calling wake on the appropriate waker If we take an example of something that ultimately reads from a tcp stream, you know the context the the Pole method gets called by the tokyo runtime scheduler. It passes in the context that represents the current task It's passed all the way down to the async read call on the tcp stream That tries to read from the socket and gets told there's no more data What it will do is that internally in tokyo internally in the tcp stream type It'll take that the waker in the context that was given into pole read So the waker that was inside here and it will save that waker inside of a little secret storage area Next to the tcp streams file descriptor And all of that is stored inside of the i o event loop inside tokyo And the i o event loop whenever a worker runs out of stuff to do or in reality every now and again Uh, this event loop is going to be looking for events on all of these little secret compartment file descriptors So look for has anything happened on any Resource that I know about and have a waker for and if so I will go to that thing and call wake on it Which ultimately then kicks off all this machinery of moving things into the runnable queue And so this is what I mean by the runtime does not really have a loop that just calls pole on every future or every task Even if they return pending what actually happens is it calls Pole on all of the ones that are runnable and when it runs out of things that are runnable It will call into the the i o event loop And see whether it can make some of them runnable and in reality It's actually more sophisticated than that it will periodically check whether things should have become runnable So that it doesn't like starve out the ones that are waiting for i o So that's the the sort of low level connection here to resource. Let's see if that makes sense first before we Before we keep going Can tokyo react to mouse or keyboard input and in theory? Yes so tokyo doesn't actually mandate that you use tokyo provided resources the rule is just that If you implement your own resource that represents for example mouse input You need to provide something some resource to implement yourself that will take the The waker out of the context that your pole method is given for let's say mouse and make sure that when the mouse moves the Wake method on the waker is called and tokyo doesn't dictate that you have to do this in a particular way And this isn't even a tokyo requirement. This is the requirement for futures more broadly. So if we go to look at future You'll see here the core method of future pole attempts to resolve the future into a final value This method does not block if the value is not ready Instead the current task is scheduled to be woken up when it's possible to make further progress by polling again The context passed to the pole method can provide a waker which is a handle for waking up the current task That is to say that it is now ready to make progress and the contract here is that Let's see Yeah, the pole function is not called repeated linear type loop instead It should only be called when the future indicates that it's ready to make progress by calling wake And there are a bunch of different ways to do that the way that tokyo does it internally for most of its io resources is to use Something like e-pole, which is an operating level mechanism for saying I have all these file descriptors all these You know utp tcp sockets files whatever Tell me if any of them become readable or writable. That is if they have bytes available Or are ready to be sent on again And so tokyo will handle that sort of io loop for you Would it be possible to implement zero copy using async read? um It depends what you mean by zero copy So do you count a copy from kernel space to user space? because if so, yes The the tricky part is when you really want to give a buffer to the operating system and say the operating system gets to Right into this and this is sort of io u ring kind of setup This interface doesn't let you do that although there are experiments with this and I think it's the tokyo u ring crate um that tries to figure out how this might what this might look like Is there a new waker given to each future? Or async read poll i'm confused about who creates the waker the waker is created by tokyo When it calls poll on the top level task And it's actually pretty cheap to create a waker. Um, it's really just a uh v table like a virtual dispatch table a list of function pointers that it constructs And the idea is that what would wake actually what sort of gets stored in the waker is really just a pointer to the task So that tokyo know which tasks like when wake gets called which task was actually awoken And a function pointer to the tokyo code to execute when wake is called So it's executed by it's created by tokyo at the top before it calls poll So it's not actually you it's not like a waker is created when a task is created and it's stored next to the task It's actually created on the fly for every call to poll because it's so cheap to create um The implementation details of waker are actually kind of interesting the v table stuff It feels like waker could also have been a trait instead of a concrete struct So the reason why waker isn't the trait Is because it also needs to implement Drop Let me see here a local waker Uh Where is the type i'm after raw waker So you see that there's an implementation of drop for waker Uh, which means that you it needs to be able to implement drop so that the runtime for example can keep track of whether a waker Is still around and do cleanup But the drop trait is not object safe because uh Well, it's complicated No, sorry clone. Sorry. It's clone. It's not drop. That's the problem clone is the problem So clone is not object safe because it names the self type Which object safe methods are not allowed to do or object safe traits are not allowed to do And so as a result if you made waker a trait The trait would not be object safe, which means that you couldn't have a box din waker or an arc din waker Um, which is a problem because that's very often what you want And so the only So the alternative would be that you have a generic everywhere for your waker But that gets super annoying because now you have a generic for every future Going all across the the system design and so instead what they did is they have this this raw Uh, this raw waker api that is the vtable that is a vtable that includes uh What call should do and the clone implementation here returns a new raw waker rather than self to get around this problem there is actually a um Uh a waker trait as well Which is this uh task wake trait which basically forcibly wraps it in an arc to get around this problem Um, but there's a little bit of like There's a reason why it's it's not just a trait Um Why does tokyo have async read and async write instead of stood? Ah, so the reason for that is because if you look here at the read trait The read tread trait in the standard library, um Has two problems the first of them is that it takes a mutable reference to self Whereas the async read takes a pin of self. This is important because Uh, I don't want to get into all of pin. But in order to have futures be able to Um have local state that is like local stack variables inside of a future You need to have a pin of self To be able to do that soundly and read does not take a pin of self and therefore wouldn't work in a future context the second reason is because we want this um This poll type that indicates whether the read did something or failed to do something or rather couldn't do it yet The the difference between i'm done and I have more work to do The signature of read in the standard library does not have that type now io result technically if you look at the error There's an error kind Uh, and there's an error kind called wood block and so you could look at that But that gets really really ugly to have to parse out exactly which error kind in every call to read So that's the reason why this is separate trait um So it's a scheduler and the i o event loop kind of always waiting for each other um Sort of right. So the schedule doesn't wait for the i o loop Instead you can think of it as the runtime is running the scheduler and the i o loop at the same time The scheduler just chooses the scheduler is sort of run locally for every worker thread And it just chooses which future to run next and the only thing it really knows about is whether Tasks are runnable or not runnable and it only considers running the ones in the runnable column or the runnable queue The event loop its job is to move things from non-runnable to runnable By calling wake so it doesn't actually know about these two queues All it knows is here's a bunch of stuff that we're waiting for and here's the method I should call whenever they become whenever a resource becomes available It just so happens that the the code that gets called when something some resource is now available Happens to do the move from non-runnable to runnable queues So they actually end up being relatively discrete, but cooperating services Um So who calls wake the event i o event loop calls wake Um Or you can call wake So this is the idea that you can implement your own resource And all you have to do is you need to have a method that looks kind of like this, right? So if we look at something like If we look at the tokyo channel for instance and we look at receiver You'll see that there is a poll receive method and this one is not pin For reasons that i'm not going to get into But it has a poll method that kind of looks like the future poll method It takes a context and returns a poll And so this is really the only signature that you need to provide for your own resource type And then you just need to guarantee somehow that if you ever return poll pending instead of poll ready here Then you have some way to guarantee that the waker inside of this context will eventually be called wake on When you might be able to make progress. That's the contract you have to fulfill for your own resources um Okay, so there's then a question of okay, if async read and read have to be different Why isn't async read in the standard library? The answer to that is a little more complicated But it's basically because we don't know that this is the and I say we like we as a community Don't know what the correct definition of the async read trade should be This one works pretty well, but it doesn't work so well when you get to sort of next generation Inter OS level interfaces for asynchronous i o things like i o u ring It's not clear that this is actually the right interface for async read once you want the ability to do i o u ring type stuff, which is is more efficient And so it's mostly just a matter of The moment we're sure that this is the right interface, then it will move into standard library and we're just not sure yet In some sense, we're sure for now, but we're not sure we want to commit it to it permanently um And there's also this uh, you'll see this read buff stuff versus this which takes a Mutable reference to a slice of u8s. I'm not going to dig into that here But this is another thing where we're not quite sure what the interface should look like um Is the i o loop local to each thread um No The the nature of exactly how the i o event loop gets executed I'm not going to get into here because it's not terribly important and also it's kind of complicated and also It's not guaranteed to stay the same. This is sort of considered an internal implementation detail of tokyo um So the i o event loop periodically runs and checks if awake needs to be called Kind of you the way I would probably think about it is that the i o event loop is kind of always running rather than thinking We are running periodically so it should pick up on um data being available on a socket for example Basically immediately and it should call wake basically immediately It's not like you have to wait for the next time for it to be time to do a an i o check um Okay, great. Um, so let's now talk about the the different Uh i o resource types in a little bit more detail now that we know how the structure of this works um The i o types in tokyo are look mostly like the ones in the standard library, right? You have tcp listeners You have udp stream. You have t or udp socket. You have tcp stream You have uh, you know tokyo fs which has things that are you know files um directories reader you have a bunch of the same methods that you have inside of stdfs similarly for under process You'll see that you have uh A child you have command which is a builder like all of this looks very much like the standard library Except of course for that fact that all of the methods are async And instead of implementing read and write they implement async read and async write So it's really just sort of an async adaptation of mostly the same interface But there are a couple of differences and also surprising similarities to be aware of So let's start with tokyo fs So this is tokyo's implementation of sort of asynchronous implementation of ways to access the file system Now the file system is actually a little bit weird because a lot of operating system operations on the file system Do not support asynchronous access like there just isn't An asynchronous interface to working with files This may be surprising and kind of annoying, but it is the case and so Depending a little bit on your operating system and depending on You know, well the current phase of the moon tokyo will often use a blocking thread like a dedicated blocking thread or set of blocking threads In order to give you asynchronous file system access. There's a little bit of discussion about it here But what it means is for some of the file system operations Even though it looks like an async fn What actually happens is that it does a it has a blocking thread that's going to do the the real operating system Invocation for you and it does sort of spawn blocking and then it just waits on the join handle as a result This works and it does give you an asynchronous interface, but it actually means that for tokyo fs specifically It can be a decent amount slower than the standard library implementation Or or rather, I should say the synchronous implementation And that is because the operating system just doesn't provide an asynchronous interface So we kind of need to fake one by sticking a bunch of overhead on top of the synchronous interface You still usually want to use the asynchronous interface because it gives you things like select and join and you can You know Combine them with with other asynchronous futures And you can stack these inside of futures and not worry about blocking the executor for example or the or the worker pool thread so It's usually still worth it But you should be aware of this cost and as this also talks about at the top There are some cases where if you're seeking the sort of highest level of performance it can actually be worthwhile to Turn your asynchronous file system stuff Into a synchronous thing that you run in a spawn blocking closure So that you get the sort of high performance for all the synchronous file system operations And then it's just the final result that gets turned into something that's async that can be A path you want to take for something where performance of the file system operations specifically Are on your critical path That's the main caveat about tokyo fs apart from that it like mostly looks like all the things you would expect um Yeah, so this is some discussion in chat here about what the async read and write traits would look like in an IO u-ring world and the answer is they just wouldn't be there it just Is completely different and again, if you look if you're curious about this, um There is no that's not what I want. I want uh tokyo IO u-ring tokyo u-ring tokyo u-ring tokyo u-ring Which is basically an attempt to figure out what would tokyo look like on top of IO u-ring for sort of higher performance asynchronous interaction specifically on linux And it just doesn't have async read and async write. They're just not there So one thing you look at if you're curious about this IO u-ring is a high performance asynchronous IO interface that's available in linux 5.10 and later And so this is why it's not the standard thing that's being used because it's a very new Feature of only certain linux kernels its api also looks completely different from the older apis that are used for asynchronous interaction Which allows it to get higher performance? But you have to match the interface well in order to get that performance benefit And so hence the system experimentation on what should abstractions on top of IO u-ring look like in order to maximize the chance You get those performance boost Yeah, so IO u-ring would replace at least part of the IO event loop Okay, uh, so that is tokyo fs Let's then Yeah, so windows IO is also completion based which is so I IO u-ring is a completion based rather than What's the word it's escaping me now The way the linux does asynchronous IO is different from our windows does asynchronous IO Windows has an a completion based IO model, which linux does not IO u-ring, which is a linux thing Does have a completion based interface that's different from the one that's currently on windows But it might be that we actually end up in a world where the completion based interface is what's used on all platforms Down the line But how that looks like is probably not asynchronous right, but something else Hence why it's probably a good idea that we haven't put it in the standard library I guess readiness based or polling based as opposed to completion based is the the wording Okay, uh, next thing I want to talk about is tokyo process This is sort of the equivalent to stud process It looks very similar command is very similar for constructing things the main two differences There's one similarity and one difference to be aware of the similarity is that when you drop A handle to a child process and this is also true for the standard library version of child When you drop it the process is not terminated And this again is similar to stuff like If you have a join handle Both in the standard library you have a like you you spawned a thread and in in tokyo You use tokyo spawn the handle you get back if you drop it The the future that you have a handle to or in this case the process that you have a handle to Continues executing even though you dropped your handle to it This can be surprising even though it's the same as the standard library It can be surprising there too because in general what we expect in futures is that when you drop a future It doesn't do any more work. It's sort of you drop it and then it cancels like it stops doing everything It was doing that's not the case when you have a child handle in there When the future gets dropped when the future gets cancelled the child process will continue executing There are ways you can change this behavior. There's a kill on drop method You can call on command that changes this behavior But you should be aware that this is the default and again that it is the default to match the standard library Those are some stuff about just unix processes that are complicated. I'm not going to talk about this too much But you should just be aware of how unix handles processes if you try to use this in an advanced way The other thing you should be aware of is that On child which is the sort of reference to a child that you might have There is an asynchronous function wait That you can use for waiting the process child itself is not a future So you can't await you can't if you spawn a process You can't just await the child in order to get its output Instead what you have to do is you have to call wait or wait with output and await that thing Um, this is just it's not super complicated. It's just a nuance to be aware of here um Okay So that's the only thing I want to talk about process. You just be aware of this. Um, this particular idiosyncrasy maybe um Okay, two more things here. See this is why I have my notes because there are a lot of things to touch on um When we talk about io So there's async read and async write which we've talked about but there's also the async read ext and async write ext extension traits These are the things that provide you with convenience methods On top of things that implement async read and async write things like read this many bytes or read a single thing of this type um, or you know read to string or Uh, write all which is you know, when you do a write Just like in the standard library when you write to a socket It's it's not guaranteed that it writes the entire buffer you give it It might only write, you know, the first eight bytes or something as you might need to do it in a loop Write all will make sure you actually write all of them before the future resolves so async read ext and async write ext provide Futures on top of the low level interface that async read and async write has It's again the async read interface is just a pull read method And you generally don't want to call those methods directly from the future. And so instead you would use Dot read which returns a read which is a future here that you can await instead um, and there's also variants of this for uh, if you for buff read Which is if you have similar to buff reader and buff writer in the standard library, um, You have the same thing in async world where if you have an async buff read So something that internally knows how to buffer reads you can do additional things like split by line For example and get futures for those So you should be just be aware of these extension traits that make it easier to work with io resources Rather than directly working with the async read and async write traits And these are implemented for anything that implements the underlying traits, right? So if we go look here at um At read for instance Just to see what it looks like uh read Uh Oh boy, I'm gonna regret this aren't I I want to go Oh, maybe it's just actually imported somewhere Yeah, it's uh io util read. All right. Let's go to the repo and see Okay source io util read So the read method like the the actual thing that gets called when you call read here inside of async readxt Is this read type The read type just hold the reference to the reader And the implements future and what it does is it creates a buffer and then calls poll read And then returns you the stuff that got back. So it's it's just a convenient wrapper So that you don't have to call poll read yourself It is there's no magic to it really It is just the the x methods are just so that you don't have to write a bunch of code to call the low level future methods yourself Um and and similarly for writes um, the other thing you should be aware of when it comes to io is that for a lot of these methods, uh, and again we can look at um Hi, we can just look at async read I suppose. Um, you see it takes a mutable reference to self This is the same thing as in the standard library. If you try to read from a tcp stream You need to have a mutable reference to that tcp stream And what we often see unfortunately is that people have, you know, a tcp stream or something And it's in some larger struct somewhere And you have two different futures that both need to access the struct as you end up doing like an arc Mutex tcp stream so that these two different futures can both access this underlying io resource This works Um, but it is very often not what you want in general if you find yourself putting an io resource behind a mutex You can probably do better like you're probably leaving performance on the table because now you're forcing these threads to interlock on the mutex to sort of Interleave their execution on the mutex Um with the other work that they might do in reality. This is a really good fit for things like the actor pattern So the idea here is that instead of storing that tcp stream Inside of this, you know deeply nested struct with the mutex You actually spawn a separate top level task that owns the tcp stream. It doesn't need a mutex at all And it has let's say a channel that people can send things for it to write into or send messages for it to then Read out stuff from that tcp stream The idea being that you have one task that owns the tcp stream and can access it without going through a mutex Without the overhead that that entails and then do the work over it and then choose to either You know fan in in requests that need to be written or fan out things that that have been read Uh, so very very rarely do you actually want a a mutex tcp stream just so that that's something you're aware of um Okay, i'm gonna pause there the next thing i'll i'll talk about is uh the connection between icing greed and icing right and sink and stream Um, but let's take questions here before we continue Yeah, edge triggered versus level triggered is the framing i was thinking of Don't drop a child bro Are there any examples of such actors that own tcp connections or something like that curious about some implementation details um, so It's actually a pretty straightforward pattern right so the idea here is um that you just Uh tx rx is Tokyo sink mpsc channel Uh eight why not um and you tokyo spawn and i guess here i can do a Um tcp is tokyo net tcp Uh tcp stream connect. I don't know one two seven zero zero one on Whatever port 8080 Uh elate unwrap And then what i can do here is async move Uh and in here I do while let Some bites is rx.next.await tcp.writeall Uh bites .await.exp.unwrap Again, i'm not i'm just sort of demonstrating the pattern here And now what I can do is here, you know, I could send I don't know a a veck of I want to write these bites But I could also Tokyo spawn async move Loop this I could have multiple tasks now that are each Individually sending bites for this thing to write If I didn't have this dedicated actor like this thing that owns the io resource I would have to take the tcp stream and stick it behind a mutex and an arc And then have each of these have clones of the arc and then they would each take the mutex Sort of alternating right In order to write out the bites there are trade-offs involved here. This is not always a better strategy This does mean for example that you need to allocate a vector for every sequence of bites you want to write out Or similarly if this was inverted so this would read from the tcp stream and then send it out on a channel We would have to allocate a vector for every every read we did in order to be able to give away ownership But the advantage is that now These don't have to wait for the io to complete in order to continue And also they don't need to have a mutex around the io resource that they both need to sort of compete on contend on and reduce the performance of Yeah, you can still have back pressure if you want to make sure you don't just spam this right you you await this Um, and then you can also adjust how much concurrency you allow here with this Um Ah, and if you want responses, then the thing that this is a pretty common trick as well Uh, you do this send and Here I do Tokyo sync one-shot channel Uh, and the thing I send is a one-shot sender and the bites And then here I do act dot await And so this here will be the number of bytes written So that is the way that you ensure that you make sure that the response is also sent to the one who sent it Um, as you introduce a one-shot channel with with every interaction Oh, yeah ring buffers are nice for this too, but they're they're a little different than what you would use in this case But I do like ring buffers as well And if you haven't looked at thing buff, which is a really cool crate go look at it Um, okay. So last thing I want to talk about when it comes to IO resources, then is the connection or lack thereof between async read and async write and the sync and stream traits. So Uh async read and async write are about Individual sequences of bytes that you want to write into or read out of a resource sync and stream on the other hand stream being the more well known of them are traits specifically for Units coming out of history and think iterator like each iterator yields you a thing a single thing every time you call next um, think of something like again an iterator or A channel where every time you do receive you get a new thing out Um, whereas again async read you read and you read a bunch of bytes of art or like varying size into a buffer So it's a different kind of interface The bytes do not dictate framing. They just say here's a bunch of bytes that I wrote for you And sync is the inverse since it is more like async write We're except that where async write takes a bunch of bytes and tries to write them into a socket sync takes a single element and tries to put it somewhere think something like The most close analogy here is something like a Channel sender where you can send in one thing at a time Stream and sync are in that way the asynchronous versions of iterator and channel send Sync is not very widely used these days. It sort of fell out of favor But stream is still pretty common Um, if you have something that implements async read or async write and you need something that implements sync and stream What you really need to provide is some way to do framing How do you say, you know these this string of bytes is now one element? It's one type one one value And similarly I have a bunch of bytes at what point do I turn them into a single value a single element for the purposes of async And this is essentially what a codec is. This is what framing means It is the the conversion between elements or values that are independent to streams of bytes That hopefully match that framing so that you can turn them back into their individual elements There are some tools in tokyo util like specifically there's a sub module called codec That helps you write adapters between these two traits But just keep in mind that they are separate And this is also not something that's in tokyo itself And part of the reason for that is because the stream trait and the sync trait are also examples of things that We're not quite sure we've arrived at the right abstraction here We're not sure that we know the right definition of these traits and This is why they're not in the standard library There's been a proposal to get stream into the standard library for a while But at least as of the time of this taping I guess Stream is not in the standard library and sync. I don't think it's even being considered at the moment The there is a crate called Tokyo oh my god, tokyo stream which provides you with the Stream trait by re-exporting it from futures core And then implements a bunch of convenience methods around stream But you'll notice that none of the tokyo types implement stream even the ones you would expect like An mpsc channel receiver for example will not implement stream And that is because tokyo does not want to take a public dependency on this futures core Trait or crate rather because if it ever moves into the standard library with a different definition tokyo would have to do a breaking change to align Sorry, uh with that change in the stream trait And so this is why all of tokyo's implementations of stream live in the separate crate called tokyo stream So that it can be upgraded separately from tokyo And so that if stream does move into the standard library The tokyo team could cut a new major release just of this crate, but not of all of tokyo which would be pretty pretty sort of Disturbing to the to the whole ecosystem and just update this one re-export and then all the other types would stay the same What you'll notice about tokyo stream is that it has these wrappers That are essentially types that you take a type you got out of tokyo You stick it into one of these types and now it implements stream Uh, so this is the way that you bridge between tokyo types and stream There is not an equivalent for sync because people seem to generally have agreed that sync is probably not great not that great of an idea um And you'll see the same if you use that tokyo utl codec in order Adapt between async read and async write and stream and sync that it Also operates with stream from the future's core crate and sync from the future sync crate So it too if there was a standard library implementation of stream Would start using the standard library version of stream instead Okay, I think that's all I wanted to talk about when it comes to i or resources. Let's pause there for questions Sync and stream are often used for web sockets. Yeah, I've seen it being used for events or streams as well So tokyo streams certainly The stream trait is pretty useful And that's also why I think it will end up in the standard library The question is just what should its exact type signature be and that's still under some debate Any hints to why sync is a bad idea? Um, I think the observation is that it's not clear that it's valuable to have a trait here um like It's rare. I think the observation is that you actually want to be generic over sync I think you quite often want to be generic over stream either in arguments or in return values But being generic over sync is just Kind of a little weird, uh, and I don't think we've come across very Common needs for for that to be the case And if you don't have particularly like salient concrete needs, it's hard to make sure that the interface is the right one So this is not to say that like sync shouldn't be standardized It's just we don't have enough evidence for what it should look like to be comfortable stabilizing it Uh, tokyo streams are the same as std async iterators, right? um, I mean, you can think of the stream trait as Uh, the asynchronous version of iterators if you will. Yeah, there's actually a really good, um blog post by boats That talks about where is it? the registers of rust That I thought was really cool, um, where they talk about sort of what is the mapping between different concepts in the In the async world versus the non async world versus the iteration world versus the fallibility world And like how does iterator map to stream for example? Um, and they also have a follow-up thing on is it patterns and abstractions where they talk about stream? Uh, no, it's a later. I think it's the poll next one. Uh, in general, like all of these things on asynchrony I recommend reading if you really want to dig further into this. Uh, it's just without dot boats Uh I'm specifically talking about the experimental std async iter async iterator Yeah, so async iterator is The thing that stream is going to become I I don't know whether they've even landed on the name that it really should be async iterator as opposed to stream Uh, there's still some debate about that. It could be that ends up stabilizing under the name async iterator But this is the the same thing as stream just trying to figure out what it should look like when stabilized Okay So now we've talked about resources. Let's then move on to utilities We've started talking a little bit about this when talking about tokyo utl codec For example in tokyo stream, but there are a couple of things inside of tokyo itself that you should be aware of In particular, let's start with the sync module So the sync module provides synchronization primitives that you can use for having different futures Uh different tasks, but also just specifically futures. Um interact with each other coordinate with each other There are a bunch of different patterns you can use here I'm not going to try to talk about all of the ways in which futures can cooperate The doxier do a decent job of talking about the difference ones But you don't want to highlight someone that some of the ones that are easy to miss The most basic one, of course is the mpsc channel the multi producer single consumer channel This maps directly to what you have in the standard library with std sync mpsc um That one i'm not even going to spend a lot of time on it just works. It's great Um, there are variations outside of tokyo that are sometimes faster or work better for particular use cases Again, thing buff is a thing worth looking at here Uh, but ignoring mpsc There are a bunch of other ways in which you can coordinate between futures and usually if you can use one of them Chances are it will work better for you than mpsc. Think of mpsc is sort of a hammer, right? Like a lot of things will look like nails you can solve a lot of things with them But often there are more specialized tools that will perform better have better apis when you can use them The first of these is the one shot channel. I showed you one use of it already So a one shot channel is a channel that you can only send on once And when you receive you only receive once that means it does not implement So the equivalent of stream it just implements future So when you have a one shot receiver all you can do is await it There's no next because it will only ever yield once Um, and one of the things that it's particularly worthwhile to know about the one shot channel Over here. I'll show you Is so there's a receiver and a sender Uh and receiver has a blocking receive So when you have a one shot receiver you can call blocking receive And so this is one way to bridge between asynchronous and synchronous context And the same thing is true for sender. So if you look at sender The send method is not async on one shot. It is a synchronous method And so again, this is a way to communicate from the synchronous world to the asynchronous world Or from the asynchronous world to the synchronous world Um And this is a particularly useful thing once you do things like spin up a blocking thread somewhere To do some computation and and either want to send it some stuff to do Like not regularly, you just want to send it one packet of work or crucially You want it to after it finishes send some stuff back If you can you can just use the join handle for this, right? So remember spawn blocking for example returns you a join handle You can then wait in order to get the uh the result of the closure executing in But if you want more fine-tuned control over it, you could use a one shot sender instead Which you can use from a blocking context Um So so one shot very useful tool for for bridging also very useful tool for anything that's sort of Acknowledgements you very frequently see something like an mpsc Where the thing you send is both some operation And the sender of a one shot for you to send the result of that operation when it's Completed this ends up fitting really well into the the actor model of the world Um The next thing in sync that I want to talk about is and I think these are roughly listed in order I'm skipping up mpsc. Um, so there's a broadcast channel The broadcast channel is just you produce a value and you know that um When you produce that value There's going to be a bunch of copies of the receiver and every consumer receives every value So this is not like a single producer multiple consumer queue Where you know you send in and what whoever reads it first has it It is the thing that ensures that everyone sees every broadcast of a value Not very widely used be useful to know about but the really cool one is watch So watch is kind of like a broadcast channel Except that it doesn't keep every value you send Instead it only keeps the most recent value And the reason this is cool is because imagine that you have a bunch of receivers But they're not necessarily always checking on the value like they're not regularly receiving from the channel But every now and again they are um What watch allows you to do Is not build up like an infinite queue because you had some slow readers and instead When you send you don't really send you just update the broadcast value You update the value that readers will see when they read And then on the reader side you can basically subscribe to changes to the latest value And so if you come back, you know way later and it's changed 20 times You only get woken up once and told about the new value This can be extremely useful for communicating things like config changes The fact that something has changed so now go look somewhere else It's a really useful mechanism where the individual Notifications are not important. What is only what is important? It's just the latest state So it allows you to you can think of it as sort of as batching updates Uh and amortizing that cost of doing a bunch of receives in a row Um, and then the last one no actually I'll talk about two more. Um Because I do want to talk about semaphore, um, but the next one I want to talk about is notify so notify is sort of the Second to lowest Level primitive that you have in sync notify is just a type that you There's an example here. Um, you create a notify you put it in an arc or something and It's sort of like the a condition variable if you will from the synchronous world Except not associated with the mutex. It's not associated with the value at all You can call dot notified and you get a future back that you can await And you continue executing like this await resolves When someone calls notify one As all it takes so this is a way to just tell someone else. Hey wake up This can be a really useful or or simple way to implement wakeers in your own resources Rather than figure out how to do poll calls and distributing this wakeer and stuff You can often get away with just using a notify And so in your resource what you actually do is you just construct a notify and you Give the a clone of the notify away to something or stick it in a queue somewhere That is going to call notify one when there's more work to do And then in the resource wherever you are in your asynchronous function You then just call notify dot await and that way the only thing that you need to be concerned about Is making sure you call notify one from somewhere and you don't ever need to touch the sort of low level poll interface So that's where notify can be really useful. It's useful in other places too, but that's the main place. I've found it to be useful Okay Before I go to semaphors, let's do questions there Uh, does it make sense to use broadcast for a web socket server where the different clients are stored in memory? So the main problem with broadcast is The slow reader's problem, right? So if you have a bunch of web socket connections That are all listening on the broadcast channel Then imagine what happens if you have a particularly slow client Like someone's connected of like 2g network to your server over a web socket So your writes out of that channel are really slow. That means you're draining things out of the broadcast queue very slowly Which means that those can't be dropped because you haven't read them yet Which means the broadcast queue just keeps increasing you can set a Like a bound or a behavior on on broadcast for what should happen If you try to send and the the sort of broadcast buffer is full You can choose for it to drop like the latest thing or just refuse to send Um, but ultimately you have a problem there, right? Which is your application is either going to have to drop things to go out in that web socket That slow web socket or it's going to have to buffer them forever And that's an application design question that you just have to figure out what the right behavior is The the broadcast implementation allows you to choose whatever behavior you want, but you do have to choose Um Oh, yeah, notify can also be really useful for for cancelling although I talked about a potentially better way to do that later um Where imagine that you have a sort of uh You want actually no, I don't want to recommend notify for cancelling because I want to recommend a different thing for cancelling But yes, you can use notify for cancelling um So notify could be used to bridge or wrap sync resources into an async world That's also true, right? So if you look at the method signatures for notify, you'll see that So notified returns the notified and this is a future. So this uh thing is not that useful in The thing you get back from notified is just implements future. So this one is not in and of itself useful In a synchronous world, but notify one is synchronous. So you can use Notified to have an asynchronous wrapper for a synchronous resource So something synchronous is going to call notify one which it can because notify one is not an async function Um, and your future is just going to wait this notified All right. So the the last thing I want to talk about in tokyo sync is semaphore Um, usually you shouldn't need to reach for semaphore semaphore is a very low level primitive But it is useful to know what it does semaphore is a way to have a number uh Think of it as a ticketing system. So You set some maximum number of tickets that are allowed to be active at any point in time Uh, and initially like let's say we say there are four tickets a semaphore Of size four means that someone can take a ticket Someone else can take a ticket someone else can take a ticket someone else can take a ticket And then the next thing the next task the next future that tries to take a ticket is going to Return pending it will not get to proceed Until someone who got a ticket drops that ticket and if that or sort of returns it rather to the pool Then someone else can come in so it is a way to guarantee that there are at most n things where n is configurable active at one point in time There are a bunch of places where this kind of thing goes up. For example a mutex is a single permit semaphore That is in fact how it's implemented, right? It is a single permit semaphore Uh, there are other semaphores like for example, let's say you don't want there to be more than, um So in so many concurrent requests or so in so many concurrent connections you're handling in your web server You could have a semaphore that is the number of connections you allow and every time you're about to accept a connection You first check whether you can grab a semaphore and if you can't then you just do not Accept any more connections until someone gives the ticket back meaning some other connection has been dropped Or if you don't allow, you know more than n concurrent requests on a single connection You have a semaphore with that limit and you before you read a new request from the client all over that connection You try to grab a ticket from the semaphore and only if you get a ticket Do you allow it to send another request? Otherwise it has to wait for one of its pending requests to finish So this is a really useful low-level primitive and you'll find that it's actually used by a bunch of high-level primitives so mutex being an example another example of a place where it's used is in, um, uh, the Uh, concurrent request filter in tower I believe also uses semaphore because it can right this is the kind of pattern that semaphore enables you to do But its interface is very low level. So you'll see here, uh, you when you create it You say how many permits there are how many tickets there are, uh, you can add permits if you want Um, there's an async fn acquire which you try to take a ticket and you see whether you get it or not It's async so you can await it and you'll be notified when you can um, and similarly you can also, uh, So with the semaphore permit you get back Uh The where's my Uh Yeah, so, uh, if no remaining permits are available acquire waits for an outstanding permit to be dropped So the type that you get back Oh, why is the text so long? The type that you get back from acquire the semaphore permit here Holds that ticket until it gets dropped when it gets dropped The ticket sort of gets returned to the pool and then you allow someone who's waiting on acquire to resume And this is also how you get something like a reader writer lock, right? So a reader writer lock is actually one that has n permits Uh, and taking out the read side of the lock just takes one One semaphore, uh, and if you are a writer you have to try to acquire All of the semaphore set once that is how you know that you have exclusive access Um, so yeah semaphore useful thing to know about, uh, not a thing that you should Not a thing that you should generally need to Uh Use very often but it is useful to know that it exists Okay Moving on from tokyo sync The next thing we have is join set um, so a join set Is a collection of tasks that are running at the tokyo runtime and The idea here is that let's say you, um What's a good example of this? Let's say you're writing a web crawler So you have a bunch of URLs that you want to crawl from this website And so you really want to have those crawlers run kind of in parallel And so you want to spawn each one But after you've initiated all of these tasks that are all going to be downloading a different URL You want to wait for all of them to finish But how do you know when they've all finished? Well, you could have a vector and then like every spawn you stick the join handle into the vector And then after you've spawned all of them you loop over the vector and you wait each one That does work, but it's not really what you want to do You can instead use a joint set. So the idea of a joint set is you construct one And then you call spawn and you pass it a future that it will then call tokyo spawn on And it will take the returning join handle and sort of stick it inside itself inside of the joint set And then crucially Join set has a join next method that is asynchronous You can call on the loop in order to get the output results of each spawn future as they resolved And importantly this happens in any order So imagine again that you have this setup where you have these Let's say you're crawling 10 000 URLs you would end up with a vector that's 10 000 Join handles long If you and you then write your for loop What will actually happen is you wait for the first URL that you spawned to finish Then you went for the second URL you spawned to finish then the third and the fourth etc But it could be the url number 18 finished way for before url number one And with this vector approach you wouldn't know that you wouldn't get to print anything until the first one is finished Even if that might be the slowest one to crawl. It's like some giant website or something With joint set when you call join next what that means is Give me the output of the futures that I have spawned in whatever order they become available in So this can be a super useful way to keep track of large numbers of futures And then make sure that you wait for them or collect the results after the fact um That is that is all it really does And it's it's useful to know about Uh, do you notice though that it specifically is for spawning? So this will use tokyo spawn behind the scenes. It will run them as parallel tasks. It is not for Sort of local concurrency. So what I mean by that is it's not for if you have a You already have a top level task and you don't want to spawn them You just want to run them concurrently inside of the current task. Then a joint set does not let you do that Unless I have missed something no For that I don't know, uh, I don't think tokyo has an equivalent to this Let me see just in case they've added it and I'm lying No, it does not so I'm not actually going to recommend that you use this because it has a bunch of problems But there is this type called futures unordered that does not spawn and instead just locally looks at them in the current task Um, so this is a thing you can use instead, but it does have a bunch of problems I'm expecting the one day tokyo will sort of get its own version of this that isn't broken quite, um, a severe way But actually getting this Version right like the version that doesn't spawn right is quite complicated and if you look at the um, the github issues for this type You'll see that there are a bunch of discussions about How should this actually work? And in fact, uh, that's not what I meant to do at all Uh, let me fix that real quick Uh-huh Um, if we look at without boats again, uh, you'll see here Futures unordered in the order of futures, uh, that talks about this exact type and why it's actually pretty complicated to get right So I'd recommend you read that if you really want to dig into it But no, the join set is very often the thing that you want Specifically because usually you do want to spawn these so they can execute In parallel and not just concurrently what I mean by in parallel is that The different tasks that you spawn get to work on different worker threads at the same time Whereas if you don't spawn them if you just sort of join them, then they're only executing on a single Uh worker thread at a time um Okay questions on Uh, join set before I move on Yeah, so there's also here in, um In fact, I'll show it in task You'll see that under task there is Am I lying? I am lying Uh, uh in the root of tokyo, there's a macro called join and there's a macro called try join Um, and these you actually give a list of futures and it will wait for all of them to to resolve Um, and then it will give you back a tuple of the results in the same order you pass them in So this is a way for you to run multiple futures Concurrently but not in parallel so within the context of a single task without using any tokyo spawn um But the the downside of using join here is that it requires that you have a static number Like you you know at compile time how many different futures you're joining From uh, and but they are allowed to have different return values different output values Uh, try join this the same just they're allowed to be fallible um If you don't spawn them doesn't tokyo still work still if needed. No So if you use join here, um, or futures unordered That means you're not spawning them if you're not spawning them That means they're just nested futures inside of whatever future you're currently in so they're nested futures They're not top-level tasks and because tokyo only gets to schedule top-level tasks It cannot work steal one of these sort of embedded futures down here And so if you use join or futures unordered without spawning those futures will execute concurrently within this task But they will not get to execute in parallel. They will not get to be work stolen um The spawning a tokyo task allocate a new stack like gore beam instead of using memory as needed Yes, you can think of that as as roughly right that Rust futures are kind of stackless like they don't Truly have a stack. They're more like state machines um But when you spawn a task it does get its own heap allocation for the task and for the task state machine um What's a good way to pause all tasks on join all temporarily and so until something is done for it to be resumed You'd want to use join plus select. I'll talk about select in a moment okay, um Moving on to select in fact. All right. So select is One of the coolest features of async and also the easiest one to burn yourself on the idea behind um Select is that you have a bunch of different futures And you want to wait for one of them to complete This could be or sort of the first one to complete Another name for select that you might think about in your head is race So the idea is that you have multiple futures and they're racing and you just want whichever one finished first You want to do something when that one finishes and then forget about the other ones common examples of these are things like um wait for A tcp packet or the user to press ctrl c Or wait for an input on This tcp channel or on standard in Or wait for A new Message on this channel or for this right to complete Or you know wait for Input on this channel or for this notified to complete that tells me that I should shut down early Select embodies all of these When you write a select the idea is that you write a bunch of arms Inside of the select, uh, I wish they had an actual example here. Ah, they do So when you write a select you write a bunch of arms for the select Each arm has something that needs to be a future So in this case do stuff async is some method that runs a future and more async work is also a function that produces a future and When you use select in this way What's going to happen under the hood is you construct each of each of those futures They're embedded futures, so they're not top-level tasks embedded futures. They get to run concurrently And whichever one finishes first We're going to execute the block under that one And then we're going to drop all the other ones all the other ones We're just going to drop them not do anything more with them So let's say this one finishes first then we're going to go into this arm We're going to execute this code and then we're going to continue from after the select So for example here you can see we are Selecting over two different streams And if you go into this one then the v will be the v we bound back from this next call And you get to access that in there and similarly here Um And you can do the same over things like a sleep, right? So this this is one way to implement a timeout Is that you call stream dot next in a select and the other arm is you're waiting on Some sleep future, right? And so if the sleep finishes first then you go into this arm And if the stream dot next finishes first you go into this arm And so what that will be is a timeout So select useful for all sorts of things anything where you want to wait for multiple different kinds of things And do different kinds of things in return or in response to them finishing Before I talk about the complications around select and about cancellation Does the basic premise of select make sense Large select blocks are a pain because the lsp basically doesn't work inside them Yeah, so one of the pains of select is that it's a giant macro. Uh, and that means that Like id integration often doesn't understand the syntax because it's as far as it can see It's just arbitrary macro input code. So it doesn't actually know what it turns into So large select macros tend to be pretty annoying Um, usually you'll want to keep the select pretty slim And then move the actual logic into separate functions Um Why is there no await on the asian functions inside of the select house run rust run them without the await So the answer to that the like why is there not a dot await here? And the answer to that is because Tokyo's select macro expects that everything that's passed here is a future And so it will the the expanded version of the macro will have a dot await in there But you don't write the await yourself and the reason for that is because if you wrote the await yourself You could in theory pass in something that wasn't a future and then all of this would get really weird The slightly more complicated answer is that it doesn't actually call dot await for you It doesn't add a dot await. It has to deal with the underlying poll calls for various reasons But the net way to think about it is the things you passed in must be futures Therefore because tokyo knows they must be futures. You don't have to write the dot await yourself Uh, what's the behavior of select when multiple futures are ready with the result? Um, yeah, so so this can happen um either Usually this happens because multiple of the futures were actually ready before the select even started But it could also be that like during the execution multiple of them happened to technically be ready But because this is all single threaded like there's no actual parallelism is just concurrency What select is really doing is it's calling poll on all of the futures over and over The it's not actually that dumb it has optimization So it doesn't do it unnecessarily But you can think of it as it calls poll on the first future If that's not ready it calls poll on the second future If that's not ready it calls poll on the third future if the third future is ready Then it it goes into that arm executes that code and then exits from the select Otherwise it goes to check whether the fourth Pulling the fourth future is ready, etc And then it if it gets to the bottom it goes back to the first one again That's sort of how you can think of the implementation of select even though it's in reality it is smarter than that Um, but in terms of fairness, it's a good question Like what if multiple futures are in fact ready? Which one does it choose and the answer to that is you'll see here There's a section on fairness and by default select randomly chooses which branch to execute first So you you get an a random branch that is ready So that if you call select in a group, for example, you don't always end up with the first branch being the one that's taken There are ways to override this like you can make a An a branch be biased to say always choose this arm if it's ready Otherwise just do fair among the the remaining ones but this is like There are some subtleties involved here And if you actually care about exactly which arm gets executed when multiple already you should read this paragraph Or this this sort of little section to figure out what's right for you Is tokyo selecting a different from future select, uh, I don't remember the exact details of future select I think they're fairly similar But there are a couple of ways in which they're different such as I think future select. I think requires a weight was the tokyo one does not But they're they're mostly similar you you can think of them as mostly similar Bias can only be added once to the top of the macro not per arm Ah, sorry. Yes biased is Prefer them in the order they're listed So you set it once for the whole selector you don't get to choose which arms are biased. Totally right Okay, uh, so that's the reasons why select is nice Then we get to the reason why select is complicated and that has to do with something called cancellation safety so Let's imagine that we have the following. Um Let's do a loop We have a select And inside of the select we have one arm that is Uh Read a surrogate just in value Does something we have another that is Tokyo time sleep Okay, so let's say I have this Uh And let's assume that I Let's find I just write this so we have read a surty jason value Uh fn and it returns a Everyone knows that a surty jason value is actually just a string And this is a to do Okay, so let's say that we have this use of the select macro in a loop The idea here that the intent of the loop right is that Um, I want to be reading stuff into a surty jason value But every second I want to write zzc I just want to do that and then I want to keep going around until I get a surty jason value out And it's only when I actually get the value out that I'm ready to print something Um, in fact, let's make this a little bit more interesting and make it be a No string is fine. Um So now the question becomes What goes in here and let me write something that's not cancellation safe to show you what can go wrong Uh, I have a buff here Uh, that is you know, 1024. Um, and I do something like, uh Let's say I have a tcp here Oh Tcp stream connect 1 2 7 0 0 1 4 80 80 await unwrap So here this takes a, uh New tcp stream And down here I do tcp dot read mute buff await unwrap Uh, and then What I actually want to do here is loop Uh Actually, let me just do s is string new Like this and then I do s dot Pruster or push Uh buff n And then if s dot len is greater than 100 Then I return, uh s 0 8 And bring me a tokyo tcp stream And bring me read async reduxed And That's fine. We'll do, uh We'll just assert that this is utf 8. Why not? All right So now I've written a read a sturdy json value that just returns a string It doesn't I guess sturdy json value here's a lie. This is really read a string Read a string of 100 cares That's really what I've written. So let's let's name it like that um So this all looks fine, right for sort of naively, right This asynchronous function creates a string. It has a buffer. It reads into the buffer. It keeps Pushing on to the string. I don't actually need the end here. No, I do need the end here It keeps reading bytes out of the tcp stream Parsing them as a string pushing them on to the string It's accumulating and once the string is greater than 100 characters, then it returns us looks great, right? The problem here is what happens if this connection is slow So imagine that we go through this loop So we enter the loop first we go into the select and then the select tries to execute this function This function runs it allocates a buffer. It does a tcp read And let's say it gets I don't know it gets 10 bytes 10 characters So it takes 10 characters. It pushes it on to the string The length is not greater than 100. So it doesn't return it loops It goes back here and now it tries to do a tcp read again And remembering now this now has 10 characters in it, right? But this await realizes there's no more data to read at the moment. So it goes to sleep and yields Okay, so that means that we end up back here up at the select the select tried to execute this But this future isn't ready yet. So it then goes to await the next future. Okay, so that's a sleep Great. So it now tries to await the sleep So now we have two futures that are and you know, it hasn't a second hasn't passed yet So now it's two futures. Both of those futures are not runnable And so now the select kind of just waits for one of them to become runnable So far so good Now let's imagine that One second passes so this future now becomes runnable like the io event queue or like the thing that drives timers rather Now has a full second expired. It realizes that this particular sleep future should now be awoken It calls wake on that on the waker for this future that moves that Future from non-runnable to runnable the select picks up on that and so it goes around again It might even try to pull this future again, but it still says I don't have any more bytes So it goes to this one. This one now resolves. So we enter this block Now that we've entered this block, we do this print and then we finish the select and when we finish the select what happens All the other futures that didn't finish get dropped. That means that this future gets dropped And then we go back around the loop and then we do it again And we construct a new future for this and a new future for that The problem is these 10 characters that we've already read Because they were in this future the first time around the loop And then when we dropped this future At the point of finishing this block then those 10 characters are dropped because they were contained within the future And so once that happened then we dropped those characters. They're gone forever So when we go back around the loop and then we create a new instance of this future That means we call this function a fresh It starts a new string with no characters in it and then it starts reading into this So we've lost we've dropped on the floor 10 characters that we're never getting back And so we might actually if this connection is slow enough We might never get into this branch because we keep reading a couple of characters But then we keep hitting the time out which then causes us to Exit the select which causes us to drop the future which causes us to drop all the characters we read and then we loop back around This is an example of The problem of cancellation the problem of cancellation when you're using selects is that you have a future that got cancelled It got aborted it got dropped and it had some state that was actually important that went away These 10 bytes are just as if they were never read and this can lead you to really problematic situations like Imagine that this was not reading just a string of characters. It would actually reason reading adjacent object Maybe you now dropped the opening curly bracket And so now as you keep going around you actually never end up with a valid jason object Because you never got the opening bracket or rather you got it, but then you dropped it This leads to enormous problems down the line where you just silently drop state of futures And this is the really big thing to be aware of whenever you use selects Is that either your futures need to be what's known as cancelation safe meaning that they can be resumed without having lost any state or You need to write your select so that it doesn't matter like these futures are never Attempted to be reused but cancellation safety is usually the thing that you want. There are two ways to achieve cancellation safety and the first of them is that you'll see on a bunch of tokyo types like If you look at mpc receiver for instance, you'll see that the receive method says under cancel safety This method is cancel save If receive is used as an event in tokyo select statement and some other branch completes first It is guaranteed that no messages were received on this channel The implication being that there are no Messages that might be received and then get lost forever as a result of that future of receive being dropped So receive for an mpc channel is safe to use in this kind of context This asynchronous future is not cancellation safe If you drop its future and then try to resume You will have lost state or you may have lost state, you know, if it if it never did any read, of course you haven't lost anything So In general you want to read through anything that you use in a select and make sure that they're all cancellation safe And the way to make something cancellation safe if it if it isn't is to construct the future outside of the loop like this And then in the select Use a mutable reference to the future instead Because that way you're never dropping the future when you get to the select because the future was never owned here in the first place And so that way when you go around you're actually resuming the same future This doesn't quite work because of pinning So the actual way that you need to do this and I think there's an example of this down here Oh, yeah, these are the things that are cancellation safe. These are not cancellation safe. You need to be careful But there's an example here of how you can avoid this problem Where you pin it outside of the of the loop and then you take a mutable reference This pinning is kind of annoying, right? You just but it won't compile without it. So at least there's that pin And I think actually we can use std future pin here Std task pin std pin pin. Okay, great std pin pin And then you can pass a mutable reference to it here instead But you do need to pin it in order to use this trick But once you do then now you you don't have to worry about the cancellation safety because you're not ever canceling that future So this is what we mean by cancellation safety I have more to say on this but I'll pause here for for questions first It's just only because selected is in a loop If select is not in a loop this tends to not really be a problem because it kind of it usually means That you don't care about the other futures being dropped Because they were constructed in here and you're never really resuming them because you're not running that line of code again So it only really tends to come up as a problem in a loop You could probably write some code where this problem arises even though you're not using a loop But that is much less common What does pinning do under the hood does it change the type of feud? Yeah So the future type here actually becomes a pin mutable reference to The future that comes in here. So it turns this into it's no longer just this future It is actually a pin a pinned mutable reference to that future Does this mean that ideally we need to make sure that all the processes and the http handler is cancel safe I'm not sure what you mean by http handler. This only applies to things that are awaited on in a select In the the arms the the arm definitions of the selects Yeah pain was a good freedom slip here. I agree um Yeah, so the behavior is different now Uh, which is good totally correctly pointed out here that here we will now only ever read one of these But that's kind of what we intended right because we had a break here So we intended for the loop not to continue anyway the moment this resolves Select is more like a future than a loop, right? We'll we'll select is A future that internally loops over all of its arms So kind of both You should think of select as being a future it isn't But that is sort of what it is Um, it is also internally kind of like a loop In this scenario the future is never canceled and it's just ignored Um Well in the in the pin case the future is never canceled in the old case uh When we say, um cancellation safety and when we say a future is canceled What we mean is that is dropped before it got to finish So it got canceled it got aborted right it started doing work It accumulated some state and then we dropped it rather than letting it complete in other words. It was canceled So in this case we did cancel it Uh in the pin case it does not get canceled because Dropping it here does not mean that it doesn't get to do any more work the next time around the loop Um If we pin the future once it resolves won't that make it a one shot future. I'm not sure I understand So one shots are different from this one shot is a mechanism for having two different futures communicate with each other So that one can send a value to the other Uh, it has nothing to do with a future only resolving ones all futures only ever resolve ones This versus calling timeout on fut in a loop Not sure I follow Uh, could we have made the future cancellation safe by passing in the string that we write to? Yeah, so that is another way that you could solve this problem. So instead of pinning here You could do this and then do At mute s And then have this like I guess not return and instead it'll print s here This will take an s which is a mutable reference to a string Uh, this no longer gives s here Uh, like so yeah, so this is an example of something that would also make it be cancellation safe Um, a but that assumes that that's only true if read is cancellation safe Uh, and we can check whether read is cancellation safe. So if we go to read, uh As in greed xed Cancelation safety this method is cancel safe if you use it as an event in tokyo select and some other branch completes first Then it's guaranteed that no data was read. So only because of this clause is this safe But given that clause, this is cancellation safe And usually I tend to tell people always document the cancellation safety of anything that you call from a select If I have some work after the loop that does not depend on the food Uh, we need to manually drop it to avoid unnecessary background processes. Um, okay, so this is in the This is in the pin case So if we have this then yes down here, you might want to drop feud Um, but it doesn't actually matter because remember this is not an actively running thing This is just the state of the future And so you're not really wasting system resources. You're wasting a little bit of stack memory You're not really like wasting compute resources because this future if it's not being awaited or run in a select It doesn't do anything. So you're not wasting compute Um Is whatever tokyo sync mpsc uses for cancellation safety the same or related to this mute technique? No, so the way that you can implement cancellation safety internally enough in In some kind of resource is just by having to do atomic things I don't necessarily mean cpu atomics, although that that can be the way to do it so that You basically don't have any wait points between when you consume a value or like consumer Resource like reading bytes from a file socket and when you return them So you know that if you consume the resource then you definitely also return it That is the way that you guarantee cancellation safety If you consume some resources like you read some bytes off of the wire And then you have some weight in there before you return that value then you're not cancellation safe Because you could be dropped the at any await point you could be dropped um It's funny how cancellation unsafe code is still safe rust. So safe rust is a very particular meaning right safe rust means that it cannot have Memory on safety things like one type being treated as another or have undefined behavior There's no undefined behavior here. You just dropped a bunch of data, right? So it's wrong, but it's not memory on safety and that's what we mean by safe and rust Uh, so this is more of a definitional question of what the safe mean Um, and when we say that something is unsafe Do we mean the formal sense of rust unsafety or do we mean unsafe? And this is in general like this could lead to buggy behavior and those are two different meanings of the word Um What I mean by one shot is that it can't be reused in the select This is only useful in the case where you only want to use this kind of future once um Well, so sort of right so you could hear uh, if you end up in this branch you could then say um You could reassign to a few tier you couldn't actually do it this way because of the pin But you could imagine that you reconstruct the future in here. Um, in fact, I think you would use Is it uh Dot replace I can never quite remember what Um Set so you could do this in order to Uh Once you've read a thing you if you actually want to keep looping around then yeah, you would replace it with a new future So that but inside of the pin so that it itself Remains cancellation safe Um So if you want something to happen in one select arm on interval then also the interval should be to find outside the loop Right. Yes. So that's also true. So this thing will not happen every second It will happen a second after the start of every select So if you actually wanted something to happen on an interval, well, you would probably use the interval. So tokyo has a Inside of the time module um I haven't really talked about time because it it's fairly straightforward the only thing to know about here is that there's a Minimum limit on how long intervals can be in tokyo like it doesn't really support. I think intervals shorter than 100 milliseconds or something. Um But you can have an interval that you can construct And then what you would call inside of here is dot tick on that interval But you would construct the interval outside the loop precisely for this reason And in fact if you look at tick cancel safety this method is cancellation saved So make sure to always check that um I wonder if a different select API could exist that makes it less likely to make a mistake here You are not the first person to wonder that and we haven't really come up with one yet If you have a good idea then absolutely, but It it's actually surprisingly hard to figure out what should happen here. One of the questions is should futures Be cancelled on drop like what it should the drop behavior be for a future Currently it means cancelled, but it's not clear whether that should be the case. Um, there's this also ties into the question of Asynchronous drop so currently If you have a future and it implements drop then your implementation of drop is not async It like drop is a synchronous method as that means you can't do asynchronous things in your drop But trying to fix that is all sorts of complicated for other reasons. Um So this one's uh, this one's tricky um All right, so uh, that is Cancellation safety which mostly comes up in the context of select. Um Last thing then to talk about in terms of utilities is Is tokyo util so i've talked a little bit about this already so codec is the way that you connect between async read and async write and uh stream and sync Um, so that one we've already talked about i'm not going to dive into how it works Uh, I think if you need that you can go look at it This is after after all tokyo de-crossed not the tokyo ecosystem de-crossed um Compat is a way to have interoperability between the async read and async write traits from tokyo And the async read and async write traits from the futures i o crate which is used by some other asynchronous runtime in some other crates um There are a couple of useful helpers here inside i o and net for things like, um Um Things like bridging between synchronous and asynchronous i o uh things like copying things in and out of bytes Things like having um Uh, I don't even know what this thing does. Um, but under sync. I think there's some interesting stuff Uh, I'll talk about cancellation token in a second. Um delay queues for like basically things that are bigger Bigger and also a little bit more unstable constructions on top of the tokyo primitives And tokyo primitives are usually sort of lower level things building blocks that you can use Where's tokyo utils tries to provide higher level abstractions on top of that But there are a couple of things that I want to call out from here Um And they're both unknown one is under task and the other is under Where is the other one? Where is task tracker? Ha, it's even deeper. Okay So the first one of these is cancellation token. Um cancellation token is unrelated to cancellation in terms of select What cancellation token is for is specifically imagine that you have this like giant system You have a bunch of tokyo spawns like you have a bunch of different background tasks running And at some point like the user presses control c And you want to have graceful shutdown you want your entire program to like you want all of those different Uh tasks that you have to start exiting One of the ways you could do this is you could as we talked about on the runtime call Um shutdown and when you do that The next time every task yields it gets dropped and then eventually all the tasks are dropped. And so eventually you're your shutdown Returns because all the tasks are gone and now you can shut down But in reality, you didn't really let those futures finish up their work You just kind of dropped them when they were in some kind of pending state One way that you can improve on this situation is to use a cancellation token What this type does is that you can It's kind of like a notify and in fact, I think it's implemented using notify internally But you construct one you can clone them and give them out to all of your different tasks that you spawn and the idea with it is Every cancellation token has a canceled method that is a an asynchronous function That will only return the moment someone calls cancel on the cancellation token But all of the consulate all of the clones of a cancellation token are connected So if you call cancel on any clone of the cancellation token Then the canceled future on all the cancellation tokens will suddenly yield So the idea here is that for all of the tasks that you spawn in various places You have them do a select over the main work that they do and canceled And so that way whenever they get this canceled signal, then they can choose what to do in order to clean up and then they can choose to exit And so this is a way for you to have one place in your program where you say, okay time to shut down and then at that point All of the threads start hitting this clause of their select at that point they can choose what to do Maybe they exit their loop. Maybe they do some cleanup whatever But eventually when they get that signal, they should know that it's time to wrap up Stop reading from channels You know, stop reading from tcp sockets, whatever it might be and just sort of finish up and then eventually return And hopefully at some point after that your runtime should become idle There should no longer be any runnable tasks Because they should all have sort of reacted to either this cancellation signal or Cascading signals from that like, you know, they were reading from a channel and the sender of Of the sender end of the channel they were reading from has now been closed because that thing saw the cancellation signal So eventually it should sort of propagate all the way down until all of your tasks have gracefully exited and the cancellation token is just a handy means to have the trigger to start a shutdown and the Means in which to react to that shutdown signal There is also a Stream cancel there's an equivalent type specifically for streams um that has uh Sort of this this trip wire thing where you have you can take a stream and you can wrap it with A thing that will close the stream like have the stream start to yield none The moment someone calls cancel on this trigger and so it's the same kind of construction It just forcibly makes a stream start returning none with the same ultimate purpose Um So this can be a a really nice way to Give your application the ability to do graceful shutdown But it does require that you thread this trip wire or this cancellation token all the way through your program So that every task has a way to know that okay, it's time to shut down now. Let's see Can you keep reading from a channel after the sender has been dropped? So it depends If you look at The tokyo mpsc Channel right it is a multi producer single consumer And so the rule is that when all the sender handles have been dropped Then it's no longer possible to send values into the channel This is considered the termination of the stream. And so then poll will start to return none But if one recender is dropped, then that doesn't mean that the channel is closed and you you Will still block waiting for more values There is a way to specifically say um Where is it? No, there's not i am lying But yeah, so so the the answer to this is like the a channel If all the senders have been dropped then the receiver will still receive whatever may be in the channel But after there is no more it'll get a none and then at that point no to stop reading similar to an iterator really um What if you're writing a library like an actor library? Then it's hard to share of the cancellation tokens as the user might create their own one It's true. And and this is one of the ways in which it would actually be handy if Cancellation tokens was in the standard library so that everyone could agree on how to signal It's time to finish now Until that happens, and I don't know whether it will it would probably go inside the context maybe um Until we get something like that if you're a library author You could have sort of two apis right one that takes the cancellation token And one that does not and if you don't get one you just do whatever you normally do which is you block forever or you um Your awaits might take forever during shutdown. You might just get dropped But if the user provides you with a cancellation token, then you're willing to respect it um That of course would mean that you would take a public dependency on the tokyo util crate Which you might not be willing to do But you're sort of caught between a rock and a hard place here. There's no great solution for having libraries that support being Uh gracefully shut down sadly Unless you just write your own type that wraps this that you tell people to write in and then they need to figure out How to bridge between them It's unfortunate Because token is not mutable Can I use it as a static to avoid needing to pass it to every function that's going to depend on it? um You know that is an interesting question um I think you can actually because I think it is yeah, it's both send and sync So yeah, I mean I think in principle you could you could have a You uh, it couldn't actually be a normal static because the con unless constructor is const Yeah, the constructor is not const And so you would need to use something like a once lock in a static But if you did then yeah, you could use a static for this It might get really annoying for things like tests right because all of your tests will use the same static So if you go cancel in any test all of the tests will exit and this is the the perpetual problem with statics Right is that they truly are global But if you don't have tests then you could probably do this. I don't know that I would recommend it Well async drop replace cancellation tokens No, even if we had async drop That would be convenient But it wouldn't replace cancellation tokens because async drop is just to allow you to do Asynchronous operations when you are dropped But cancellation tokens are the way that you learn that you should exit and should start to drop yourself Um Yeah, often the way you would do um Cancellation and libraries you would have like Give me something that implements future and I will treat that as my cancellation So you just take anything that implements future with the output type as unit and Someone could pass in a cancellation token or whatever their own means are It doesn't really matter right so so they could just take Call cancelled and just pass this type in because as long as you're generic over what you take you're okay If you pass the cancel token every function does not behave just like go context. Yeah in some ways does um All right Uh This is a type called task tracker that can be useful to know about in in tokyo util It's sort of a combination with cancellation token. Uh, you can use it to get graceful shutdown I'm not actually going to talk about this in detail, but there's a decent Uh Blog post on this on the tokyo blog on graceful shutdown that I recommend you read talks a lot about exactly this And how to wire it all together correctly and how you can use task tracker to make the make your life a little bit simpler Um All right We're on the home stretch. I think those are all the utilities I wanted to talk about Now the only thing that remains is a couple of like Foot guns that are particularly common that people run into all the time Um, and I'll go through these pretty rapidly because they're they're not huge topics They're just things that you just need to be aware of so that you don't shoot yourself in the foot with them um Okay, so number one took your spawn When you spawn something Uh, and I think I mentioned this earlier briefly, but I want to reiterate it the join handle that you get back When you drop it, you're not waiting for that task that task continues to run in the background And there are two implications of this the first one is Um, when you drop the join handle, you're not guaranteed that that task has finished doing in the work But also it continues to run it continues to use resources And so because it spawned because it's running in the background Just dropping it is not sufficient if you want your system to not be wasting resources over time So if you truly want it to stop then you need to do something like call abort So that tokyo knows that the next time that future yields, it shouldn't resume it. It should drop it It should cancel it instead So that's the first aspect of this that's important. The second aspect of this is When you return from an asynchronous function that happened to run tokyo spawn You now have this thing that's running in the background that might be doing something important But the from the outside from the caller's perspective of the function that internally did in a um a tokyo spawn The caller doesn't know that there's now this background thing, which means crucially it doesn't know that it might have to wait for it So imagine that you have a background thread that's like, I don't know you tokyo spawned doing a uh A file copy Well, if your main your asynchronous main function calls the thing that calls the thing that calls the thing that ultimately spawns this thing that copies the files But nothing awaits the join handle Then what's going to happen is the thing that spawned the file copy returns And then we return return return return all the way back to main main also doesn't know about this background spawn That's going on so main finishes and at that point all outstanding tasks are Dropped because main exited and at that point your file copy stops in the middle And so it's really important to remember Whether you want to do something whenever you do a spawn think about the fact Or consider every time you do a spawn assign it to a variable and decide what to do with that variable If you truly want it to be dropped Then you can drop it. That's fine But you should make a conscious decision about whether dropping is the correct choice or whether it should actually be awaited Or whether it should actually be communicated further up or whether it should actually be aborted But make a conscious decision rather than just letting the the default behavior come back to bite you later um Yeah, it's not as though a background task is necessarily useless, right? It could be you actually wanted to run in the background. You don't want it to be aborted You don't want it to be awaited like it's a I don't know it's Your logger or it's um forwarding something from a channel or reading something from a channel So it could be the correct decision is to sort of forget the handle like drop the handle and just let the background task running It's more that that it should be a conscious choice. What do you want to happen? Uh, I think the default behavior is actually reasonable as a default It's more that you should really think about whether that default applies to you Um, okay Second thing to be aware of is the tasks are single threaded So when you do a tokyo spawn or whatever you do in your async fn main Is a single task, right? We've talked about what tasks are It's a single task which means that it's being executed by a single worker pool thread If you do a bunch of work in different futures, but they're all in a single top level task That means all of that work is only being executed by a single thread If you want work to be spread across multiple threads Then you need to spawn those futures so they become top level tasks so that they can be executed in parallel if you just use something like select or join or Well, not a join set because it spawns for you for exactly this reason, but if you use a join or a select Or a futures unordered for that matter Um, what you end up with is a bunch of futures are executing concurrently but not in parallel So they're all being executed by this one thread But only one of the futures being run at any given point in time That may or may not be what you want Sometimes you do actually want to multiplex them on one thread But you should be aware of the fact that futures embedded in futures are always single threaded You need to spawn them and turn them into tasks in order to get parallelism uh And then the final thing I want to talk about is uh a footgun that I see pretty often in an overreliance on D channels So we see those in particular with the actor model actually where you know, you spin up something that's an actor It owns let's say a tcp connection And what you do is you keep sending messages to the actor for things like write these bytes write these bytes write these bytes or Handle this request handle this request The actor is again a single top level task As a result it only has a single thread and as a result That single thread has to handle all of those requests one at a time Now that's not inherently a problem, right? It might be that one at a time is the fastest you can write out on this tcp stream anyway And if you had multiple Tasks all trying to write out to the same tcp stream they would need a mutex or something So that's not inherently bad The problem you run into is that when you have a Sort of fan in pattern into a single actor a single task that needs to do something about them It needs to pay a cost for every channel receive And again, if you think of these receives as being like write these bytes or handle these requests Sometimes that's reasonable, but very often the loop on the actor is probably going to look something like you know while While let some equals channel dot receive and then you know do a bunch of work with that message That means that by the time it comes around Depending on how much work there's in the loop by the time it comes back around to read from the channel again There might be five new things in that channel But it needs to read them one at a time even though it might be there might be more efficient ways to handle those incoming requests in batch How you choose to do that batching? There are a bunch of different ways to do so. Um, you could use things like a Well watch is not really appropriate here But you could imagine for example that instead of having a channel to this actor The input to this actor is actually a mutex over a vacuate And so if you want to send something to the actor what you do is you take the mutex You extend the vector with the bytes you wanted to write and then you drop the mutex And then the loop on the actor's side no longer looks like Get you know one small set of bytes and then go again and get a small set of bytes Instead it takes the mutex It takes the entire vector out and leaves an empty vector behind and then it writes all of those bytes out at once That might be way more efficient And allow it to get to much higher bandwidth Because it allows it to to batch the cost across multiple calls into the actor This pattern doesn't always work. It doesn't always fit what the actor needs to do But but in general there's a tendency I see to over rely on mpsc channels, especially for actors Where it's often the right call for the simplicity of actors, especially at lower throughput, but when you get to Systems that strain the resources more You might want you might want to start to think about these kinds of optimizations to amortize the the cost of communication and synchronization here I think I think Those are all the things I want to say about tokyo But I'll I'll spend some time in chat and see if there are more things again Doesn't have to be about what about the sort of foot guns I just talked about but across the whole stream Let's do sort of quick. I don't want to say q&a, but a quick Everything I talked about at tokyo. Are there things that you still get confused by? You want more detail on Things that have bit you in the past to sort of round off the stream here uh Sounds a bit like a ring buffer. Yeah, it is kind of like a ring buffer In fact, you can use a ring buffer for this if you're willing to do the copy Um to copy out of it rather than a steal um What things in tokyo are actually parallel not just concurrent. It's only tokyo spawn um And of course runtime spawn, which is the same spawn blocking is parallel because it runs on a different thread Joint set is parallel because all of the all of when you Give a future to a joint set it spawns them. So basically the answer is only things that are spawned are parallel And the i o event loop is is parallel um When doing batching usually you want to paramerize a parameterize it with two inputs the max batch size and the max wait time for a batch to fill Um, is there something in the tokyo ecosystem that implements such a pattern? Not that I know of but we're talking like five lines of code. So I I I don't think so, but maybe Request says he wants a tokyo 1.x runtime when running with another runtime is tokyo doing something special their quest needs um, well, so this this uh This gets at the fact that In order for tokyo to know how to move when futures need to move from the non runnable queue to the runnable queue It needs to know which things those futures are waiting for Which means that when those futures use a for example a tokyo tcp stream Um, that's how tokyo knows is because tokyo implemented the tcp stream So it knows what to wait for so it knows when to move them and basically how to implement wake um If you try to use a non tokyo runtime, but you're trying to use the tokyo resources You run into a problem because the tokyo resources like like a tcp stream When you call poll on it and it realizes like let's say you call poll read on a tcp stream It knows that oh, I didn't have any bytes to read. I need to register myself somewhere so that the runtime So so that I can notify the runtime when I need to resume Basically, I need to register myself with the i o event loop So it looks for an i o event loop Specifically the tokyo i o event loop and it doesn't find one Then it doesn't know where to put its waker and its file descriptor for something to be picked up later Uh, and so that's why when you use the tokyo i o resources You need to also use the tokyo runtime because the i o resources are tied to the i o event loop Um, if you use let's say the the async stood runtime or something But you're just trying to use a tokyo resource Then when a poll read happens it doesn't have anywhere to to put its state that it it can rely on will call wake later And so Request I believe uses tokyo resources All the way down like it uses tokyo tcp streams for example And you can't run that in a non tokyo runtime Because that connection with the i o event loop would be lost Um Does tokyo not do mpmc? No tokyo does not have a multi producer multi consumer channel Uh, if this hasn't been asked already, why does the async write x have implementations for each individual type such as write u8 instead of a generic time parameter Um, I think that one is because it's not clear what it would be generic over Right, let's say it's you had a right Number right and it's generic over t Where t is what? Right, like what what is the bound that allows it to take a number and turn it into a sequence of bytes? And this is it gets complicated, right? Like if you get a u128 The order in which you write the individual bytes of a u128 out to out to the writer depends on the endianness for example and so you you There's no there's no generic bound to really express that and so that's why you end up with individual methods Uh Can you talk about broadcast and the overhead that comes with having a bunch of receivers and senders? um So there's there shouldn't be that much overhead when using broadcast with with multiple receivers I don't think broadcast allows multiple sender Right is it uh Sender oh, yeah send takes a reference to self. Okay, um So there's not I think a huge overhead to having multiple The main problem you run into with broadcast is if you have slow readers They hold up all of the readers or rather it's not quite true They don't hold up the readers, but they either hold up the sender or they cause rights to start to get dropped and they start losing messages Um, but that's just sort of an an inherent problem here. This is the the lagging problem that you see um Does tokyo spawn create an os thread or a green thread it creates a green thread Um, what about keeping a mutex guard between the weights? We already talked about that earlier Is there a way to access the blocking thread pool of the runtime? Can I get it to a handle to it and send it somewhere else? Um, sort of so if you look at runtime Uh, there is a method called handle that gives you a handle and on a handle Uh, which is clone there is a spawn blocking and spawn blocking lets you run something on the pool and get an an asynchronous Future back So so you can pass a handle around and use that to spawn things onto the blocking pool But that is your handle to the blocking pool Uh, can you talk about how you deal with handling what would normally require async drop like losing Third party async libraries you would like to clean up on drop, but those cleanups would be async There's not a great answer to this async drop. Just it sucks that we don't have it and it's really hard to get You will see that for a bunch of types not close as a bad example of this, but um What is a good example of this Um, I guess actually maybe file Um, no file is also a bad example here. Um I guess tcp stream maybe Or buff stream Maybe buff stream is the one I want to talk about. Um Although maybe that doesn't solve the problem either I guess maybe there isn't an example of this in tokyo So what I'm looking for is you'll see a bunch of asynchronous libraries have An asynchronous function called close or something like it Um, and the idea is that If you know that you want to drop this type then you should call This function in order to drop it So it's a close would be an asynchronous function that consumes self And so the idea is that if you have the ability to manually drop it Like you know that you're going to drop this thing rather than relying on the drop trait Like it just going out of scope call this method instead and then it will actually get to do its asynchronous cleanup The you also see this actually in in synchronous methods where, um You might have a close function just because the close thing might error And so if if you just let the thing be dropped that was the error from the result is just silence is forgotten So you are encouraged to call the close method if you can in order to recover that error or see that error And not have it just be silenced And you can the the same thing applies for async that you can provide an async function close That doesn't perfectly solve the problem right because there are cases where It's just going to be dropped and you don't really have control over logic that happens on drop except through the drop trait when that happens that the best thing that I know of here is um So there's a handle current or try current that lets you get a handle to the current Tokyo runtime and then When you get that handle you can then spawn code that would get to do your cleanup stuff It's not beautiful It means for example that if you get dropped off of the runtime Then now you don't get to execute anything to do that cleanup properly But it is sort of a best effort asynchronous cleanup Ultimately though this is a problem and we just don't have a great solution for it um Tokyo sync broadcast is not in multi producer multi consumer. No broadcast. So broadcast Is specifically not a multi producer multi consumer Q it is a broadcast Q what that means is and the distinction is an mpsc an mpmc channel is Any sender can send a thing and one of the consumers will get that thing A broadcast is any sender can send a thing All consumers will get that thing. So the semantics are different Uh, does the can do the cancel safe Futures or methods internally revert state? How do they know when to do that if they can become cancelled in the middle? Um, it depends actually So some of them are just written in such a way that they are cancellation safe So they don't actually have to revert anything. They're just written in such a way that there's no a wait point between the The sort of commit point like the point where they change externally visible state and when they return And so therefore they're just naturally cancellation safe and some of them will actually have a sort of drop guard So they'll construct a type Internally that they create an instance of before they do some stuff And so if the whole thing gets dropped the drop implementation of that instance will do some revert Stuff that is a much more complicated pattern, but we do see it sometimes Um You think tokyo will remain the most common asynchronous runtime in the future um At least for the foreseeable future. I don't see any particularly relevant general purpose runtimes at least um There are runtimes that are better suited for sort of specific domains, for example, like embedded But I don't know of any that are sort of compelling as a as a general purpose executor at the moment Um, there's a send method on channels called wake. Yes That is that is exactly the way in which they uh allow the receiver to Realize that oh this future has now moved from non-runnable to runnable Um Why doesn't tokyo implement unix sockets of the type sock sec packet? I don't know. I don't know what that Variant of unix sockets are there are a bunch of like utility crates around tokyo that That provide like alternate socket types, for example Um, I think also there is uh, so this is a thing called async fd Which is basically a wrapper type specifically on linux on anything that has a file descriptor Um that implements the standard Unix sort of read and write interface. Um, so you might be able to use that but tokyo is never going to support like Every single possible resource Directly in the library that would just not be feasible But the idea is that you should be able to write Additional resources as other crates that people can use and they will interoperate nicely through things like the waker Um Would changing an asynchronous function that is pub from cancellation safe to non cancellation safe be considered a breaking change for sember That's a very good question. Uh I There's no well-defined answer to this I would claim that the answer is yes because you have changed the correctness contract of the function Right like someone has been using your code correctly Uh, and now you have made their code incorrect By changing your code and they were relying on something that was At least assuming the cancellation safety was publicly documented before then it would be a breaking change If you never promised anything about cancellation safety, then no, I don't think it's a breaking change So this becomes a question of documentation really did you ever promise that it was cancellation safe in general? When you call third-party libraries for things you should be pessimistic about what they promise you So if they don't guarantee a thing, you're not allowed to assume that thing or if you assume it You should also specifically check it because it's allowed to change um Any plans on de-crossing the small ecosystem? No I don't think it's particularly relevant um Tokyo is not really good for real-time applications. I mean That's generally true for Anything that doesn't use a real-time operating system So but but but this sort of gets to what I was implying with embedded, right? Is that there are runtimes that are better for specific kinds of use cases? Um, so I was more referring to like a general purpose asynchronous runtime um Are there any workarounds for synchronous to asynchronous to synchronous to asynchronous call stacks? I mean the answer is basically don't do it. It will come back to bite you the slightly less um slightly more helpful answer is Channels are usually the way that you do this Whether one shot or mpsc. They are the best means we have for really bridging that gap um things like block in place are Crutches where if you use them a lot or similar like handle current if you use them a lot You're going to run into really sad times as a result Because those those nested patterns end up just performing really poorly having really hard to debug runtime failures You you don't generally want those like sync async sandwiches um Okay, I think I got to the end of chat Oh Okay, so I hate this so much YouTube has this feature now So I have the chat window up on the side and it defaults to top chat not live chat top chat Which means that I might have missed a bunch of messages. Let me change that to live chat and see what I've missed Uh Okay, it looks like I haven't really missed anything great. I am not an elixir user. No um What are good strategies for migrating from sync to async? I find that trying to write certain modules using a runtime and throwing handles around is difficult. Um There's not a General purpose solution here for rewriting things from sync to async Um, I think I've usually found that you kind of want to go outside in rather than inside out Like trying to write wrap asynchronous things with synchronous things um tends to be a bunch of pain because um You now end up spawning multiple run times in each place where there's an interface if you start from the top You can have one asynchronous runtime And initially you're going to have things that block the runtime and you can use block in place or spawn blocking and so that Direction of the port tends to be easier and then you just push async more and more down so that tends to work better um The problem with real-time async is I don't think rust is the right abstraction since real-time requires to some extent detail how to make a cpu Uh How much cpu a process is going to consume it's unclear whether in real-time operating systems You this is the right interface. It might be um, but it's much less clear um At the domain is without boats without dot boats Uh Are you a heavy user of tokyo? At helzing. I mean, yeah, tokyo is the asynchronous runtime like I don't for for general purpose Uh asynchronous code. I don't see a reason to use anything else um Okay I think that's it. I think we're done and hey Okay, I start when I started two three hours and 33 minutes and 33 seconds I think I guessed three to three and a half hours. So I'm pretty happy with that All right, great everyone. Thank you for coming out and I hope you found that interesting and I'll see you for the next stream