 I also want to thank the organizers of EuroPyton and the volunteers. This is my first EuroPyton. It has been awesome so far and I really hope you are all enjoying it too. I am very excited to talk about how thinking in Python made me a better software engineer. This is a subject I really like, like the subject of thinking in a programming language and I really hope I can interest you too. So my name is Johnny Dude and this is my real name. I work at Toga Networks, I design software and algorithms. Python is my favorite programming language but I usually use it for prototyping. My team is responsible for C++ code for the most part and that's what I've been doing in the past decade. So I'm going to use C++ as a contrast for Python in some parts of this talk. This talk is divided into three parts. I will first go over some psychological concepts and then I'll try to show how it relates to the development process and I'm going to attend in about an experiment I did and its results. So first couple of psychological concepts and I really hope you can keep these in mind throughout the talk because I'm going to refer to them. Imagine you were sitting in a pub drinking wine with a friend and out of the blue your friend asked you not to think about a pink elephant. Ironically you will think about a pink elephant a few times in the next few minutes and this is called the ironic process theory. Try not to think about something makes thinking about it more likely. Next is the magical number seven. As Wikipedia describes it is often interpreted to argue that the number of objects an average human can hold in memory is seven. Give or take two. There is nothing we can do about it. We can however keep that space available for when it is needed the most because anything that occupies your working memory reduces your ability to think. This is an example of priming. By reading capital of France Paris should pop up in your mind. I almost said Paris there. Priming is a technique whereby exposure to one stimulus influences a response to a subsequent stimulus without conscious guidance or intention. So basically what it means that things that happen influence your mind as well consciously or subconsciously and it can be helpful at some times but very much like the pink elephant sometimes you may not want it but you cannot prevent it. Think most software engineers if not all will agree with me that task switching reduces their ability to think about their productivity time. Psychology research shows that switching between tasks of different types like reading and writing for example costs more in terms of productivity and can cost almost half of the subject productivity time. Last is fluency which is the ability to do an activity with little or no effort at all. People for example are fluent in walking. When you compare a walking meeting and a sitting meeting both you take the same time and conscious effort. So those are the things I'm going to refer to throughout the talk. Now let's see how it relates to the development process with some examples. Let's start by getting immediate feedback from the code you write. And this is very in development. I got the habit of using idle both as code editor and interpreter when I started coding Python professionally that was about 2005. There was no IDE back then and that was what the team used and real programmer didn't use mouse and window but it looked something like that. You evaluate and check each expression you write as you write it getting immediate feedback from the code you write. Now you find bugs earlier, you learn faster and you have confidence in the code by letting the computer execute it in its head and not in your head. So in this example we have a short function and it aims to find the biggest files in the folder. And this system function returns the standard output. We use it to call disk usage and split line will return us a Python list of lines. And split each line by tab to get a size and name, pairs. And then enlarges will give us the enlarges pairs and descending by size. We drop the size, return the file names. It seems simple but there is a bug here and it's a silly bug. And my point is it doesn't matter how senior you are, sometimes you will write silly code. So let's use the interpreter alongside writing this code and see how it works. First we define an example argument and then we execute a statement and we try to sample the result. Once we see it is okay, we move on to the next statement. And here we can see that the numbers are actually strings and not numbers. What does it mean to enlargest? If we manage to miss it, it would be glaringly obvious in the next statement that 96 is not bigger than 912. So we can fix it without task switching and with the same information in our working memory and move on. And the benefits are that you learn faster. Human and not only humans tend to learn faster and achieve higher degrees of expertise when they get feedback. The sooner, the better. Catching bugs earlier reduces task switching. And it is important here to note that task switching is not window switching. So when you write code you need to consider how it will be executed. And I do not consider actually executing it in another window as a different task or a different state of mind. Writing it blindly, however, and then finding the bug in the unit test forces us to task switch into a debugging mode. And remember this can cost half of your productivity time. And with immediate feedback you also gain confidence that your code works by executing it at least once successfully without spending extra conscious effort. Another example of confidence, when you learn C, you learn C loops and you learn not to get an index out of range. This is a must-have skill for C developers because an index out of range might cause segmentation fault and then you will need to debug it, which is painful, hard to do, and hard to forget. So you will spend years double and triple checking every edge case of every loop you will ever write until you are fluent in C loops, if ever. Executing it all in your head and despite all this effort, you will still have an index out of range bug once in a while. Okay, standard representation. Object representation is often underestimated. Can you tell me how old bread pit is? You should be able to find the answer quickly because you should be fluent in reading such data structure. If we want the list of strings optimized for filtering items matching a regular expression, we can use a line-separated text, just like most Linux tools do. And then we can use any Linux tool or regular expression library for high-performance matching. But it is not easy to read. A list is. And you don't get to wonder who is Numa Thurman. My point is, regardless of the implementation of your data structure, keep a standard representation for the user, which is often yourself. Then you can actually visualize it in your mind and think about it effortlessly. So I'll take it a step forward. What if we wanted a dictionary with keys that can be searchable by regular expressions? In this case, we can use the line-separated text for the keys. And those keys may be long, so we can use the offset of the key in the text as a dictionary key. It saves space and it saves CPU and hashing, but it looks like that. Can you tell me now if Mia Wallace is alive or dead? How about now? Again, regardless of the implementation of your data structure, you need to keep a standard representation for the user. In GDB, when debugging C++ code, we have modern Python scripts that allow us to pretty print the C++ standard library data structures. And the problem is usually when you write your own structure. If I write a structure like this, it would look like that. In Python, we have a wrapper. But in Python as well, I often see user-implemented objects without a wrapper or a non-standard one. And remember, if you say, I will write wrapper later, it won't get done. It usually won't get done. And if you don't use standards, it is more confusing than helpful. So if you can read it, you can visualize it in your mind and you can think about it effortlessly. Discuss it with other developers and you will find it easier to work with and less of a distraction from your main task. For me, it's more important than the plain readability. It's the ability to visualize and think of the data structure effortlessly. Standard API is also important. And a very good example for a standard API is the counter. You can read it immediately and be alone, but the number of walking dead. And the counter uses a very well-known data structure with a very well-known API. And the counter also tells you explicitly what it is. The only difference is the constructor. Using a familiar data structure and API is the base of keeping your mind free to think about many other things. One thing about the magical number seven is that it can hold entities of a limitless amount of entropy. So it is harder to remember nine random digits than two random words, which in terms of entropy holds more information. And in the same way, it is more... it is easier to remember a counter which is a dictionary with a different constructor than a counter which has no explicit reference to a data structure or an API. Speaking about dictionaries. If I tell you that I want to store something in a dictionary, it keeps your mind free and allows you to give undivided attention to the explanation that follows. But if I say to you that I want to store something in a map, well, if you are a C++ developer, you will start wondering about things like why do we need a keys order or why not use a hash map or why is there no good hash map implementation in the standard C++ library? And even if you don't think about it consciously, your mind will prepare itself for answering such questions. Now, composability. And another silly example. Here I have a function who returns a list that contains a string of a number for a number in sorted numbers if the number is valid. And Python here allows me to explicitly describe it. And it also dictates how it may sound in the voice inside of my head or in a talk in front of an audience. C++ without diving into details does not allow composition of standard library iterators and algorithms. And in some cases it forces the user to create new copies and in other cases it forces the user to modify data in place. So it may look something like that. And for me it looks like a regular recipe. If you think about a recipe, an actual recipe, like this one, you have mixed ingredients in a bowl, period. Pour the bowl content into a mold, period. And bake the mold along with its content, period. And those were three sentences with intermediate variable and a lot of useless, obvious information. A baker reading it may just translate it in his mind to bake the mixed ingredients because that is all they need to keep in mind. So this example showed a functional composition and it works well for me because I like to think functionally. But Python allows you to be consistent and write imperatively or object-oriented if you want. You can say I want this list sorted and contain only valid stringified numbers. And in the recipes case you can just say mix and bake. And by the way, this is like C++ because actual C++ looks like this, which is a topic for a different discussion, I think. But my point is this. It is easier to think with composable tools. And generally, Python tools and modules in the rich standard library are consistent and composable. And my favorite example is the unit-test mock library, which is written for the unit-test framework but works with any test framework. Simple is better than complicated. I skip the complex part and you'll see why soon. In C++, when you define a function, you can pass parameter by reference, by value, by row pointer and recently by rvalue and smart pointers. You have shared pointer for shared ownership and unique pointer for unique ownership. This is something you don't have in Python. Everything is shared object. Or like shared object. So if you have a project and you don't care about the micro-performance it's written in C++, you may think why not just use shared pointers everywhere? It's easier and as you know, simple is better than complex. So you tell everyone in your team, from now on we're going to use only shared object everywhere. And then somebody decided they want to use a unique pointer. Why? Because of the C++ core guidelines. It's a 450-page guide that any good C++ engineer follows. And now we have a problem. What if we want a function that receives a smart pointer but we don't care which? In this case, the C++ core guidelines have an advice and it tells us to use raw pointer and make it const, which means immutable in that scope. So you can put the const keyword before the type declaration, it makes the object immutable, you can put it after, it makes the pointer immutable, and you can make both. And then you read code and you see the const in the middle. And the C++ core guidelines actually have a section about it. It says don't do that. It works. It does something, but please don't do that. And then you remember that complex is not that bad. Complex is better than complicated. So you decide to throw away the C++ core guideline and just use shared objects everywhere. And then you spend a week debugging a segmentation fault. Remember, the hard to do thing because of a line like that. Pass a shared pointer by reference, which doesn't make sense in C. And this is not funny. This is like the culture of the C++. Every time we read code, write code, or do anything with code, we just want to optimize everything in the most absurd and unnecessary way. And we also care about parameter passing. So we can use shared pointers everywhere, but we cannot stop thinking about pink elephants and every other option in the C++ language. And here Python makes it easier to think about other things by offering just one option. Yeah, I wanted to mention type hints. Not that part. That got me laughing, though, until I got to the 2039 part. Yeah, so now we're in 2019, and static type checking made a huge comeback in the Python community. So I had to try it. And it took me like five minutes to ask myself, do we really want to define types and structure before understanding the problem and the solution? And Python here reminded me that it is possible to do it separately. So I wrote code and added hints later. And then I realized something else that I didn't notice before. Type checking and describing processes are completely different tasks, which uses completely different systems and require attention to completely different details. So when we are writing in a statically typed language, we are constantly task switching between coding and type defining. And it's actually worse for statically typed languages because just by defining integer, it raises the question of how many bits would I like this integer to have? And it's important because what happens when you are wrong? Or you can say, let's just use an integer. It's the default. We don't need to think about it. We will deal with it later. So you can say we don't need to think about it. But can you actually do it? Okay, this is the last example for today, and it is a topic I am very familiar with. It puts everything together because prototyping is actually a way of thinking. Prototype is a model built to test a concept and to be learned from. You write it once, gain an experience in both understanding the problem and a solution. And this is a solution you are trying. And if you are satisfied with the result, you write it again with less thing to worry about and attention to finer details. Writing it again after it is organized in your mind will make sure it is more readable. And by solving all the sub-problems before, you have less thing to think about. And then you will have a little bit more space in your mind to think about future tasks and maintainability. And maybe find a bug or two you had no capacity of noticing before. But it does cost you a little bit more time. So prototyping should be a little bit faster. And coding anything the second time should be about twice as fast. For me, it takes about 25% more time. But this will eventually be profitable for the entire team's productivity because it will reduce the complexities of future tasks. Now, this is true for any programming language. But as I mentioned before, some things you can only do in Python. In Python, you can use a dictionary without considering its implementation. Implementation, sorry. You can define a function without caring about the arguments types or bits. You can think in the language you write and not work hard to translate your thoughts. And if your production code is in Python, you can handle the type checker later, separately. So along with many other things that have nothing to do with thinking, developing in Python is about four times faster than developing in C++. So prototyping should be at least four times faster. And there is an empirical study here, a reference downstairs. And it shows an example in one domain how it's actually faster. It is a nice old empirical research that I like. This is so counterintuitive, I often forget it when I am busy working. I have a task to implement something in C++. By adding another task of prototyping before I write the C++ code, I save the overall time I need to spend to finish the implementation. So my question is, how much of this speed-up do we get from thinking faster and not from fancy features and other technical qualities that makes development in Python faster, as you all may be aware? And that gets us to the last part. I want to tell you a story about a personal experiment that I did, and it is not a valid research, not even close. But it is something that you can try if you have free time. And for me, it demonstrated the power of thinking in Python. So in the beginning of this year, we were interviewing candidates at my workplace, and we gave them a well-known problem with a well-known solution. Find the shortest path in a text maze. On the left, you can see an example input, and on the right, you can see an example expected output. And what we wanted was to understand the difference in work times it takes to solve this in Python compared to C++. So I implemented it a few times in that order. And I measured the times. And as you can clearly see, it takes much more time to do it in C++. And I remember that empirical research that I talked about before, and it actually correlated source lines of code to work hours. So I counted the source lines of code, and I found no reasonable correlation. So I actually had to read the code and see what's going on. And what bothered me most was the last two implementations. So as you can see, I chose the question, and I consider possible solution before even writing the first implementation. So by the time I got to the fourth implementation, I was really bored. I knew what I was doing. I knew what pitfalls to avoid. I didn't use an interpreter. I didn't have compilation error. Everything worked perfectly well. And both of the solution looks very much the same. In fact, both versions had exactly the same algorithm, data structures, functions, and even names. And the only difference was the number of characters. So if it wasn't boring so far, I re-typed the solution and reduced the time it took me to write it in C++. And it still took me twice the time. So this is one example function from the implementation. And as you can see, I didn't use any fancy Python feature. I did use a generator, I think from Habit, but I didn't use its true power, its composability. So I just populated the set. And you have to understand that I am significantly more fluent in C++. So I should be doing the C++ implementation faster, and yet it took me twice the time. And the question is why? Why did it take so long? And failing to find a solution that relates to the development process, the question literally became, what was I thinking about? And these are possible answers that could fit a 30-minute talk. But we're not done yet. One more thing. It is fun to experiment. And you don't need to go out of your way to do it. So in this case, I just needed to understand what process will the candidates go through. And after the first two implementation, it didn't make sense to me. I had to implement it again and see what happens if I implement it again in Python. For some reason, I did it again in Python and then I needed to see what's going on with the C++ version. And so I was not performing a supervised experiment. I was just following my mood for very interesting results. So maybe you two sometimes choose different approaches. For me, sometimes I think like something takes me, it should take me 10 minutes of work. I don't need an interpreter or I want to use an interpreter even though it's 10 minutes. So I don't need to change what I want to do. I just need to measure and with time, I will get enough samples to hopefully answer the question of which process works best for me. Okay, so what have I been talking so far? I was talking about immediate feedback and its importance for acquiring fluency, reducing task switching and improving confidence. And in Python, it is just as easy as moving from the usual blind writing mode into an interpreter. In C++, I learned how to remove dependencies and work on isolated chunks of code so I can execute the code almost immediately, as often as I would like. I also make sure to have a standard and composable representation in API in every project I'm involved in. This helps the entire team understand each other's data structures and helps them even to debug each other's code. Also, Python here taught me the importance of good architecture and API design because it's one thing to read a book about it and another actually using a very good example daily for a while. I can say the same thing about composability. One second. I lost my notes. Keeping consistent and composable structures, objects and modules, help you become more fluent in the framework you work in faster. And it also keeps your mind free of things that becomes obvious. And prototyping Python. Even if you write in other languages. To prototype properly, you actually need to think without destruction and it doesn't usually work very well with most other languages. And if you write Python code, try rewriting it instead of refactoring it. It works for me. Maybe it will work for you just as well. Okay, so, let's see. If I want you to take anything from this talk, I hope you will start thinking about the way you work. Think about the way you think. Try to think in Python. I know it sounds insane. For me, sometimes I actually think about some solution or something and I can just write it in the same way I thought of it. And don't just think. Experiment. It's fun. So, please let me know what you think. It doesn't matter if you're a junior or a senior, if you write only in Python or in many other languages. If you agree with me or you don't agree with me, I would very much like to hear what you have to say. You have a link for the slides here. I have a thanks section for people who helped me write it and some inspiration and references that you can see. And that's it. Thank you very, very much for coming and for listening. A insightful talk. We have approximately one minute for questions for Johnny. Yes, let me walk to you. Thanks for a nice talk. One small question about immediate feedback. Is there a particular reason for not doing it in tests, like unit tests, instead of switching to the Rappel? Why would you want to write the same code twice? When you write a unit test, you actually think differently. Once you have finished written the function, as short as it may be, and my example actually demonstrated it, when you write a unit test, you need to switch to a different mode and think about what will I do when I unit test the function. The function doesn't even return the size of the files, so you need to think about how to test it. And if you find a problem with it, you actually need to switch to a debugging mode and think about how am I going to debug this function now. And you're in a completely different context and think about completely different things that you will need to come back to later. I hope it answers your question. You can experiment and then copy it to your code if you want. That works just as well. Thank you very much. Unfortunately, we run out of time for questions, but Johnny is available afterwards for more questions. Thank you.