 Hi. Hello, EuroPyton. I really enjoy to be here today. A little bit about me. As you already mentioned, I'm a contributor of Apache Airflow and other smaller libraries. I'm a speaker. I'm originally from Ukraine and today I will start joining with you about some development and debugging asynchronous programs. Software development is hard because of different factors. In one of the factors, it's hard to debug. So I extended very known eight stages of debugging. So probably every day you observe your program and first question that, oh, it can't happen. It does not happen on my machine. That should happen. And why does it happen? Oh, I see. How did it ever work on production? Who wrote this? Oh, wait. It was me. So there are many books. How to write programs in Python. But there is no enough information how efficiently we can debug your programs. I feel like this when I coding and when I'm debugging. So it's still hard. Sometimes I feel I spend most of the time for debugging, testing rather than development itself. And compared to asynchronous programming, it still is challenging. So the interest of I think programming has grown dramatically in the recent years. And it's slightly different from sequential programming approach, which means it's even more harder to debug because I think programming added another complexity which makes debugging asynchronous programs harder than sequential programs. So I have a question to the audience. How can I debug my Python program? Any ideas? Correct. Correct. Print statements is the right answers. And usually, yeah, you can just add a bunch of print statements and you can run your program. You can see. And, okay, can you raise a hand who is using on daily basis print statements? So, okay, yeah. By the way, yesterday I saw a talk about PySnooper. So you can check it. It's very nice from Ram Ratchoon. Yeah, so it helps you to use better print statements. Okay. Any other ideas? So I typically also use Doug debugging. So it's quite nice. So when I can start talking with this small Doug, which explaining, I'm trying to explain to this small Doug how it works, et cetera, it helps me a lot compared to print statements. Any other thoughts how we can do it? Correct. So we can do it with the buggers. But the buggers of sequential programs consist of, like, it involves repeatedly stopping our program execution and then, like, oh, click next, like this. So we can put breakpoints and we can jump to the lines and we can continue and we can repeat it infinitely until we can find root cause. So this kind of debugging style we call cycling debugging. So, yeah, as I mentioned, you can click step by step so many times. Okay. Unfortunately, programming and parallel programs do not always have reproducible behavior even when they run with the same inputs, which can be different rather than sequential debugging. Okay. Let's back to the Python. In Python, traditionally, we write our programs in a sync manner using this async-weight style, using async.io library with the sync.io loop, et cetera. And even if you are not aware of this async.io, you can use some high-level frameworks which are more likely under the hood using async.io. So async.io provides for us debug mode, so which can help us to find not awaited coroutines, like mitigated, forgotten awaits. We can find non-thread safe API exceptions. We can log if something running more than we expected, same with callbacks. How we can enable this debug mode, it can be done in different approaches, like we are environment-reable, using Python dev mode, using just the keyword debug through for sync.io run, or just using method on loop, on current running loops at the back. Okay. So what it does, it helps us... So this is how you can see when debug mode is disabled by default. So if your program has logs, it's just printing something. But if you enable this debug mode on the line number seven, you can see it prints that something executing more than 103 milliseconds, while it's printing it by default. I think you said like threshold to 100 milliseconds, so anything above will be printed, anything faster just hide. So you can tweak it if you're looking for even faster coroutines execution, you can tweak it. Okay. Another reasonable question, okay, we enable debug mode. Okay. We can see some extra information. But returning back to our day-to-day basis, probably how many of you are using REPL on daily basis in Python? Raise your hand. Yeah. So Python is interpreter language, which has this redevelop print loop. So you can run any program... So you can run something in REPL, see result, and understand how it works. Alternative to debugging, we can use this REPL. But the problem with REPL currently, if you run it in Python 3.6, for instance, you can't use evade, async evade inside REPL. So we just show you syntax error, and you can say, wow, evade is a valuable keyword. Why it does not work? Here's a Python bug, which explains the problem. And you receive one of created APR. It has been merged, which brings like alternative REPL. You can run it just Python minus M async IO. And inside this REPL, you can do evade. So there's a drawback of such approach. You need to copy entire your program in REPL and execute this evade somewhere. IPython REPL, for instance, for Python 3.6, already includes this feature with lots of hacks, but it works. So as you can see, you can just do like KPI calls using a HTTP, and you can do evade, which is nice. For instance, if you're a Jupyter user, and you are using, since Jupyter runs under the hood, using IPython, IPycernel, so you can do anything you can do with a Python and evade syntax inside your notebook, so you can put breakpoints, you can evade something, and it can work. Okay, but let's return back. As somebody mentioned in the audience, we use a lot PDB or IPDB, or maybe you can use PySharm, but it doesn't matter. So PDB defines an interactive code debugger in Python. It's very well known. If you're still not using it, I highly recommend. So what it does? You can put breakpoint in any line by this new keyword. I don't remember since Python 3.something. Or you can just type import PDB set trace, and then you can run it, and it will stop on this line, and you can, I don't know, print entire function using list. You can, again, do next, you can do continue, you can do anything you want, and it works with async def, so you can dig into any asynchronous function. There's no any issues. But still, yeah, and still you can extend it. Standard PDB has very limited from user experience features, so you can use any extension like PDB++ or IPDB, so I like PDBPP++. For instance, it has very nice sticky command, which can, like, if you're a terminal guy like me, you can just feel like you're in PySharm, so it just sticks this line. When you do next, it will move line to this line and like throws your program on the screen, which looks very nice. And it's just a sugar, nothing useful. So, but PDB++ has lots of extensions. You can find in docs, it's very handy. So, yeah, you can do same stuff as with PDB. You can dig into any await function. Next one is IPDB, same as PDB++. You can install it. The only tip I can show you right now, if you use this Python breakpoint hook, you can use breakpoint keyword and Python will inject IPDB instead of PDB, which is also handy, so you can use native Python keywords and stop your program just using breakpoint instead of typing import IPDB. So, everything is okay, but the problem, still, all these PDB-like tools does not support any await syntax. So, if you stop in any, like, I don't know, you have a handler, which is asynchronous, and you have await, and you try to await something, you can do it, because IPDB does not support it. Here's the issue. It's closed. It's not implemented since 2019. Even IPython supports it. So, what is IPDB? IPDB brings IPython rappel into your, like, debugging session, so you can use, like, IPython. But, still, it does not support await. Why does not support await? It's a reasonable question, and you can ask me how I can await a car routine in PDB. There's always workaround, right? So, we always can find workaround. And then, why it's workaround? Because by design, I think it does not allow us nest event loops. So, when you're in this debug session, you should keep in mind you have running event loop of your program. When you want to await something, it means you would like to emulate something like, I think, how you run this car routine, which is not allowed by design, because it has practical problem. For instance... So, typically it shows you runtime error. So, this event loop is already running, and, yeah. But, as I said, there is, like, a workaround which has a name nested async.io. So, what it does, it patches async.io library. So, this is more or less real-world example. You're using io file, which... What it does, it just gives you ability to read or write files asynchronously using, if you try to keep your code base, like, equivalent. If you are using a sync code, then it makes sense to use another sync code inside. But, for instance, I put breakpoint, and I would like to, like, await fpread and see what it should look like. So, I put some data to the file, and now I would like to see results, right? So, it's, I don't know. You would like to evaluate any async function in your REPL debugging session, like, on this slide. So, for instance, I try to await read, and you can see you can't use await. Then, okay, how I can do it? I can get current event loop, and I can run until complete, and it will print hello world. So, it works. But, again, it works because we just patched async.io. So, it's only useful for debugging. It's not for production. Yeah, it requires modifications, but still, it helps you to do it. For instance, in Node.js, they also have a REPL, and it supports await. So, by design, just to compare. Okay, if you are on Kubernetes or Docker environment, you not always can monitor your stuff. So, there is another library, which has named ioMonitor, which helps you to wrap your running web application. For instance, ioShttp, or other, and expose port. So, then you need to tweak your Docker file to open this port, if you want to debug it somehow. And then, you can connect to the host and port. And, yeah. And then, you can see some useful information. So, it's kind of, like, top or something, I don't know. So, this means, like, print all coroutines. So, you see, currently I have one painting coroutine and other stuff. It's also very useful to, like, for instance, one use case can be, if your program is hanging for a while, you just restart it, it works. Then again, it's hanging randomly in the night, you again need to restart it. But if you try to expose a port and connect, you can at least see where we sleep, what we are waiting for. So, very useful. Okay, and last but not least, it's ioConsole. It's just, like, a very nice alternative to what I already demonstrated about Rapples. It's just an alternative approach, so you also can do a wait. So, if you run your program with iPython, airPython, so, yeah, you can do a wait. Yeah, we don't have too much time. I have few more important examples. One production, let's say, very productionized. One is find deadlocks. So, the problem of deadlocks in Python is also painful, like in other languages. So, what is deadlock? So, I will give you example. So, if you have, like, two sync functions and one, like, rely on another and you can run it in parallel. So, it just, yeah, it just can all hang or just crash out of memory with recursive error. You can find very nice issue. Yeah, right now it's already in GitHub issues, but previously it was in bugs Python org. So, it's how we can handle a sync of your deadlocks. So, it can be very useful and practical to have a deadlock detector, like in other programming languages, like Golang or anything else, inside Python. But still, it does not exist. So, there is, like, one snippet, which you can find on this link. Yeah, and what it does, it just helps you to cancel your waiting deadlock and, like, try to find these deadlocks. So, it also requires some modifications of your source code, of course, but it helps you to, at least, because sometimes you have memory and it will just hang. So, you can wait, like, I don't know, one hour when it can be killed. And last but not least today, it's library with funny name is await what. So, it's also trying to solve production problem. Assume you have long-running sync.io program that appears to get stuck about once for a period of time. Again, you can probably kill it and run it again. It's typical, like Kubernetes does, right? So, if it's see some, I don't know, out of memory, it kills pod and start again. But if you want to find root cause, you probably need to find at least line or place. So, assume you have, like, a sync function which reads something from socket and you probably wait for a bit in this line. And then you have a long chain of other functions which run one then another. And as you can see, we just run it. So, when we run it just with vanilla Python, you can see that we, like, wait for pending coroutine and just stack trace. So, if you read it carefully, you can see, okay, but wait for foo. What is foo? Foo is a line for, okay. So, it's absolutely unclear because you probably don't know your code base well, especially code base of your third parties, right? And what await what does. So, it's also do some tricks. As you can see, it's introduced some debugging information. So, it prints the full stack trace of chain of your coroutine. So, if you're like, if it was foo, bar, base, until leaf. And now, you can see, okay, I am in the leaf. So, leaf, it's waiting for socket or even if it's sleep, like in my dummy example. So, this is a problem. And that's how it helps you to identify the problem. You can fix it and you're happy. As a bonus, this library also can draw your fancy graph if you like. Okay, last one. It's just for any production Python code. Just recommend you this IO debug. What it does is just helps you to print or even on callback, just to tweak, do something when, I don't know, slow coroutine exists or running more than duration. So, you can send it to studsd and you can then draw final pictures. Okay, final sort. Enable debug mode for sync IO. It's the first step you can always do. For production, of course, expose metrics and monitor everything. Use debuggers if possible with some modifications. IO monitor is very nice for container environment. And questions, thanks. If there are any questions in the room, please line up next to the microphone. And people online, please don't hesitate to let the operator know if you have a question. Thank you. I was wondering which tools and libraries do you use to test your async IO programs and how well do they work for you? So, PyTest async IO plugin works well for just writing functional tests. And enable logs when you run your test suite for entire programs also looks good and check if there is any warnings. Just don't hesitate to ignore it and fix it. Probably it can be the root cause to why you're waiting in production for something later. Any other questions? Don't be shy. All right. If not, oh, we do. Go ahead. Sorry, do you have any strategies for tracking down raise conditions? Yeah, as I demonstrated, you can use this like wrapper. So we just wrap your coroutines and it's just if you dig into this snippet, I will share slides. You can understand how it works. But it's just what it does, it's just cancel your coroutine early and not waiting for recursion error exists. But I'm usually, I mean, it's very tricky. So it's case by case. I mean, there is no one common approach. But it would be nice to have this raise detector for Python itself. Since like library exists, we introduce this new subset of features. It makes sense to introduce raise detector too because you can easily find, there is a good Stack Overflow answer for similar question. So where you can easily like shoot to your leg like when write a quote and do deadlock. Any other questions? All right. Let's thank Andrei again.