 Hi, my name is Lucas Langa. I'm the CPython core developer with the core developer in residence and I'm Hosting this event, which I was tricked to do for the second year in a row. So I guess you know like Fool me once, you know in any case I have six interesting people with me Starting, you know to the closest to me Mark Shannon who is the tech lead of the faster Python Team at Microsoft another Microsofty next to him who is our Windows expert We have Peter Victorin. I have a name. He does have a name. He's Steve Dawa Yes, we have Peter Victorin who you probably met in the morning. He is our Fedora expert and You know knows a lot about the C API and has even more opinions about it Then we have Larry Hastings Who was the release manager for Python 3.4 and 3.5 and attempted a Gelectomy at some point in the past And survived to tell the tale then we have Pablo Galindo Salgado who you could also meet in this room Giving a talk before with Martha Gomez on f-strings He is one of our people on the steering council And as well like a parser expert for for the entire team Works from Bloomberg. And finally we have Magda Gomez who is a mentee of of Pablo and contributed like large changes to f-strings for 312 And you know, we hope she's gonna keep contributing and we want to know more about how it How it kind of feels when you're a new contributor to core parts of the language So, yeah, well, we have those six people with us and we're gonna be talking to them about Interesting and less interesting things on CPython. So I highly encourage you To ask your questions Last time we did this last year people were really shy At first so they were asking like me to start with something and I didn't really have great questions So then they saw okay your questions kind of sucked we can do better and they started asking theirs But by the time they were like really into it like our time was up So I encourage you to just start early. It doesn't matter if the question is great, right? Like, you know, like there's no penalty for asking something that is either too controversial No, that's actually cool or too boring that there's you know, we don't judge we we want questions So, yeah, I see already the first person being brave enough to stand and ask so yeah, let's go guys you go Hey, thank you for your work. I was So again coming from a place of ignorance I think faster is better than slower right if we get to the same results and Some people complain that Python is not fast enough My understanding is developers are much faster with Python, right? So that's nice So why are we making Python again? It's good, but why are we making it faster? Is it for those that complain or because it's also nice for us. What's the rationale for the for all of this work? Personally, it's because it's something I've always wanted to do and I'm now in the rather privileged position of being paid to do it but I think there is a Mean if you're using Python, you don't you don't want to have to make that trade-off You want the nicer language and still have it go fast. There's no reason why it shouldn't go faster All right kind of the the typical Making Python faster process that we expect users to go through is to write the code first nice and quickly Get it correct and then find out where it's too slow so that they can go through and make those bits faster And so anything we do that kind of just makes it automatically faster reduces that amount of work You don't have to identify as many hot spots if we can clean them up before they ever get to you All right, we have another question the type hints syntax in Python is rather Accidental or like a series of accidents Is the long-term future of that going to be More cohesive or a better experience and how all I will say about it first of all I will say I don't use the type ending stuff. I'm kind of I Just don't use it. My observation from the outside is that it is something that we are making up as we go along So there's a certain amount of looking ahead and there's a certain amount of lack of looking ahead It's like we need to solve this problem right now. We're gonna here's the thing that solves the problem Okay, we saw the problem now. We can go on to the next problem. That's on fire. So there has been a lot of of Continuous evolution on the ground of the types system in Python my theory which is completely unfounded by the way is that one of these days someone will say hang on there's a much better way to do this and there will be a new type system that replaces the old one and People will say oh now we have two in which one should use and she'll use the new one going forward and so on Python's approach to type ending allows that because it's a completely generic like it's it's just a The annotation is just a place to put a value when you put anything you want there And we could have completely two separate systems and have even have them saved by side in the same File so I'm just waiting for it to happen really Okay, if I can add to this even though I'm not part technically of this panel Well, I was part of the team that worked on pep 484 initially and one of the kind of constraints We had there was to make typing optional enough that it does not require any changes to the interpreter. They did appear later on when typing You know ended up being popular enough that it was you know deemed worthy of keeping it But at first we really didn't know like what's gonna happen like if our people are gonna revolt like you know Is it not gonna get any use so we didn't know so which is why we have certain? awkward pieces of syntax it is improving slowly We're probably never gonna get like triangle brackets for Generics like we're stuck with the square brackets because it's what the language allowed us to do and now it's baked into the interpreter because we allow built-in collections to use Generics as well But certain things like you know that our quality of life are being improved only It's not really well kind of funded by You know users of those features so it is slow progress right like it's a it's a gigantic feature to add to an existing language So the plug progress there is slow Some of the weirdness of the typing system is because the typing system in Python like the actual Things that you can do in it are weird like people say cool like typing should tell me when I put You know numeric flag instead of a bull but you know bulls are in so you know kind of certain things The type hinting cannot help you with Like I put a string but it said there should be a sequence of strings But hey surprise like a string is a sequence of strings And so on and so on so certain constraints come from what Python actually implements, you know and we have to be Implementing the same type system I think I want to add something also from kind of the string comes in both of you, which is that it's I will say that You think about it Almost all features of the language have been accidental in the sense that you don't come with like a spec of what is Python You know that's the secret thing that we keep in a special safe somewhere and it's the you know We open it every year We kind of have ideas and when we made them and then we throw it to the wall and they tell us what is good And what is bad and we try to make the next thing right We could not need dot format if we thought that f strings was a good thing from the start right like now We have two ways and the reason we have two ways It's not that we think that there should be two ways is that we used to have one and now we have a better one and Like type annotations here is no different It's just that as everything else became something like organically and when we decide to incorporate new things to the typing language Especially when we are judging this thing from the string council We always are thinking about like is this coherent with the rest of the language And obviously that answer will change depending on who you ask right like maybe you don't like them And then you think they suck like on the other hand You may really like the match statement and these people that think that that is not coherent with the rest of the language But here we are right so so We take care of those things So if your worry is that if there is someone thinking about that this thing feels natural Yes, but on the other hand like Fit things feel natural once you start using them quite a lot And you know, that's that's one thing that you need to have in mind, but yeah, I feel you like We get that especially for people that are just Into starting to type in language from a you know Beginners point of view or they come from a different language and they have different expectations It may feel a bit weird, but in that regard I want to just echo what the cook has said about That Python is already a weird language if you think about Alright, cool. Can I jump in with a quick war story from being a release manager? Let's go Okay, we've got an hour to fill you guys. So these things are gonna crop up So I was the release manager for 3 4 and 3 5 as Wukash mentioned in Python 3 4 We added what was called tulip and then became async IO and back then we had a bdfl and bdfl contacted me and said Async IO is evolving rapidly and we need to keep evolving it and we're still figuring it out as we go along So async IO by declaration of the bdfl is not subject to the beta feature freeze They were changing async IO up until literally the day that I cut 3 4 dot 0 release You know the final release And actually I remember just a constant stream of changes coming from Victor's center changing this And it was like okay. Well the bdfl said so and that's what happens 3 5 We added the typing module and the bdfl contacted me and said the typing module is changing very rapidly and So it is immune to the the feature freeze and it's going to change right up until the very end And I remember a flurry of check-ins from Victor's dinner. So my My caution to you if you're considering being a release manager is watch out for Victor's dinner All right next question so Regarding the gillectomy thing. I was curious how the efforts of removing the gill are going right now Maybe the steering council wants to take this on the council says nothing Stay tuned I would I would say that for again, I have only been paying so much attention to Sam gross's work I think Sam has done a phenomenal job and and I'll put it this way I don't think it's gonna get any better like the galette my galectomy project I was able to push it so far and I kind of gave up It was like I couldn't get reference counting to be faster efficient enough And so I was like the next step is to go to Tracing garbage collection, which I really didn't want to face Since I did the galectomy project There was a published paper on this biased reference counting, which is what Sam is using And Sam said oh you didn't even know about this it didn't exist when you did the galectomy work So he gave me an out but Sam's work is so much better than the galectomy ever was so My my statement on it is that Don't if you look at if you look at Sam's no-go work And you say this isn't good enough Then nothing will ever be good enough Sam deal Sam's no-go patch is as good as we're gonna get and so either we're gonna merge it and we're going to boldly stride into a Multi-core future with see with CPI fund or we're not going to adopt it in which case CPI fund is going to have a single thread Until the end of time is it's not going to get any better than this So if I can step in I think Sam took it as far as one person can reasonably take it and Now it's the time where everybody else is Taking a look at it and filling in all the details where it needs to be consistent with the rest of the language and backwards compatible and and all of the little things that you know you have to do for for a change of this magnitude and That'll take some time and maybe we'll find that it's That's something something blocks the effort. I don't think I've seen anything. That's that's actually blocking There's just a lot of work on little details and on integration And we'll see how that goes. I mean for completing something on the string Kind of side of things like the problem that we are having with these changes that we have never faced a change this big And with these many consequences so the sting council. I mean, it's not the first one, right? I mean my third Year and I think we start the five years ago six years ago So we have successfully like rule over many things But this is the first time we need to say something that impacts so much, right? Literally this could be if we do it wrong, it could be Python 3 to 4 right the same way Python 2 to 3 so if we don't want that and There was as Peter is saying there is many things that are not just the technical problem of like removing the gill and making it fast enough There is things like okay Let's say we have like a no-gill version of Python and a gill version of Python So right now the no-gill version of Python is binary incompatible with the gill version of Python, right? And that has a problem because like either we break a VA 3 and it makes that all the packages that were Compiled long ago that are compatible with future versions of Python don't work anymore And some of them are not maintained which means that nobody's going to make them compatible with you know Python again And there are more than you think by the way or we make Both a VA not a VA compatible Which means that if you want to use no gill and you're using let's say noon pie You need the noon pie developers to you know release also a will for No gill noon pie and that the same thing for every of your Dependencies which means that they need to not only opt-in into the no-gill world because obviously they may need to even rewrite the whole thing because You know C extensions are not prepared for parallelism because they have global estate and they were relying on a big lock That were making them very happy So so that's a lot of work just just to be clear well potentially a lot of work I mean be from zero to infinite Which is complicated because you don't know like in Python 2 to Python 3 It kind of you know the deal because it's like that by a thousand cats right every single thing is like the fine You know what you need to do you know string bite whatever but there is like a lot of them here It's an unknown number of them You don't know where they are and they will bite you because this is multi-threaded backs right so we don't they don't like it So in that case it's very hard because it means that nobody can test the thing So we cannot find backs and then we don't know how to you know switch it and make it to the full so there is all these little like things to consider and The the first thing we we did in the sting council That's already done is decide how we're going to kind of approach this and right now. We are literally this past week We are well, I cannot say officially where we are but like we are close to reach some decision over this So stay tuned but but you need to understand that If we screw up as a team and as the team council and the core developer team I'm not only saying the team council it could hurt the community a lot like a lot And it could make a lot of work and you know a lot of engineering hours dedicated So it's never to be Python 2 to 3 again But it could be like a lot of pain for a lot of people if we do it incorrectly Including the core development team because this patch is huge and like it makes right imagine that you are a contributor to see Python Right, so the previous world is like okay The kind of the worst thing that you may have to find the first month or two months if you are contributed to the core It's maybe a you know a leak in a reference camp, right? But now you have multi-threaded bugs and like you need to understand locks and like all these delay reference counting And you need to understand this and it's a lot to ask for a core developer to imagine for a new contributor Right, it raises the bar quite a lot So it makes that less people can come to contribute to see Python because it's much harder Which means that it's going to be more difficult to maintain and those things also matter You're not just the raw speed of having it So we are taking all those things into account and this is just to give you a hint of why we are taking so long on like Giving you some answer to this is just that it's just not the technical challenge It's the community challenge is the compatibility challenge is like what are we going to do with this like how we're going to even merge this This is like how many lines of code is this like 15,000 or something like that Brutal like imagine that you're at work and that your intern gives you a 15,000 lines of like code like what did you do? Thanks So and this is sequel so it's not even Python code mind like multi-threaded sequel like how like it's just like How do we even review this? Like you need people pay full time to review this right so it's a different problem So we need to figure out all these small things But we are close to reach some some conclusions so stay tuned maybe in the next weeks or month but The reason we are taking so long to take a decision here is that it's a very hard program for everyone Those those big patches are the easiest to review actually you just see like this over a thousand lines So see I is green looks good to me stamp it It's it's the opposite of bike shedding. This is the nuclear reactor I wanted black so I just want to reiterate and emphasize what Peter said about all the other bits and pieces It's not everyone sort of thinks that reference counting is because we kept the gill because we couldn't remove it because of reference counting We've come to rely on it in many ways and lots of other things in the code And because of that there's lots of extra work to do to to make things genuinely run in parallel So some of the stuff so so Sam has effectively solved the issue with the reference counting But there's still all the other stuff to do and that's a lot of work And I have no idea how much work it is for some NumPy and other things that really Python's really Python without the need supporting as well so So speaking of of new contributions, right like you know, we do we still have the girls So stuff is easy right like and just contributing to Python. It's not a big problem, right? Like you might I prove that true you Submitted like a really big change to have strings now they're part of the grammar. So what was your experience there? My experience could give you into see Python. You mean. Yeah Oh, very nice So, oh, let me give you like a quick summary Over all the code was Easy to follow when I had questions Pablo helped me a lot and When I made a pull request everyone doing the reviews and everyone who you know in a way commented it was also very nice Very welcoming. So, you know, overall the experience was very nice and I learned a lot I had the chance to work on something different that I usually do on my day-to-day work. So Yeah, I mean, I would totally recommend the experience Oh, no, but uh The thing changed right you changed the tokenizer So it broke some libraries and people came and they were upset and you know, like how did you react to that? Well, I expected people to be upset because I mean it was a huge change and When something changes, there's always someone that doesn't like the change. So yeah, I expected that and It's still people who was upset about this was very nice and all the discussions that I saw Were very polite. So yeah, I would say it was So great experience. So any tips for anybody who did not contribute to Python yet, but would like to start? um, I would say A gap an issue and ask for help and Do it All right, cool We have more questions from the audience. Hi I was going to use this point that people here have actually quite a lot of skin in the game I was inspired by conversations between sessions here Talking about rust and rust is eating the world rust is even sneaking to the python world And I was going to ask you do you think that uh rust is going to eat our lunch and we're gonna stop having your python We're gonna be your rust at some point So, uh, I would like for that to be possible I would like For I would like to enable people to replace their sea extensions with rust extensions Uh as comfortably and as possible and then we'll see I mean let the best language win So let's say you have a folder with songs Like you need to find like a bunch of songs for like some artists that you like And maybe like rename these songs and like, you know, I'd like name of the The artist and maybe the album are a bunch of things you need to organize things So if you are a really good rust developer, you're going to reach for rust and cargo or whatever and then done It's super fast. It took like zero zero zero zero one nanoseconds, right? But if you are any other person, you're probably going to do this in python like half of the time and do it So it's not like There's no lunch to be like stolen here And even if rust like takes out a lot of the things that python is doing right now That's a good thing, right? At the end of the day It's not that's about if you use python or not Is that if you're happy with what you're doing and you feel efficient and then you do the task, right? And and I think in particular that even if rust it's it's a lot of the You know realm of what is now is done in python There is still many cases when you still want python, right? So that's fine. We don't we won't disappear We still will have euro python But you know if we have euro python here and euro rest over there and we can like give hacks to each other That's a even nice thing, right? So it's not a competition so If I can interject Uh, I have like two thoughts about this which are incompatible with one another So we just take it for what they are So first of all, I'm happy to see rust there because you know Kind of it removes a big class of security vulnerabilities through to You know kind of buffer overflows and so on So great to have a better systems language and python was always a glue language So if we are already glue for fortran and cc plus plus we can also be glue for rust So that's great. What is not great in my personal experience? So i'm not talking as any employee of any particular organization. Just me ukash is When I was Learning python it was always immensely useful for me to be able to click through You know the imports that i'm doing and to see how a particular library is implemented and I learned a ton about python Just looking at the libraries i'm using and how they do stuff And if you click through and you just find that this is some binary blob that was produced in a different programming language Like you lose the ability to gain this experience. So At the same time i'm happy that we're actually you know kind of embracing rust and i'm happy to see all the performance improvements that it gives as well But I see that there's you know Something lost when we're writing our own tooling and libraries in a different language because it's going to be harder for people to gain So you know accidental knowledge from just exploring what they're already using One thing about rust unlike go or java or anything else is is not garbage collected, which means It operates with c python very well Um, and i'm pretty sure for every python Packages is rewritten and rust two new c two new python packages will be written. So I don't think the python is going anywhere just yet Now see the question I thought the person was going to ask was like start with the observation that the linux kernel is starting to uh include some rust code like the the very beginnings of it But they think it's going to like grow and grow and grow and grow C python is written in c is there any possibility of starting to rewrite bits of c python and rust I was ready for that question because the answer is oh absolutely not We're uh Our tooling already like every every place you can build c python You can also build c plus plus programs and we wouldn't touch c plus plus with a 10 foot pole I don't think that uh c python is ever going to like We would replace c python, but we would not like Rebuild c python with another language internally like piecemeal over time like that's just not going to happen Yeah, and and rust python is already under development not by us Um, and I think if we were to I mean we'd have to change the name it would become crust python if it's No one wants to use that cross python That reminds me actually, um, so uh as alluded to mark panton shannon has been interested in making python go faster for a long time um his uh phd pieces Was on a project called hot pie, which is kind of a precursor to the faster c python Work that he's been doing It's it's a different approach, but it's not completely dissimilar And if you go to hot pie.org you'll see his logo, which is a little handmade Pie a little a little ceramic pie. I think I've seen it's like this big or something like that Crust I'm done Pablo you wanted to say something or no? Okay, cool. Next question Hi, I have a question about something that paulo I think mentioned earlier About python code for matters So I think people see them as kind of marmite. Some people love them Some people don't so I would appreciate it if you can maybe discuss like python code for matters in style pros and cons Oh, I like to break them on every release, right gukesh Yeah, this is the second time I break black. The first time was when we introduced a new pack parser because uh So so black initially was built on the old old parser And then Right a python version and then we introduced this new shiny pack parser and it was too powerful for black And they kind of like wrap around it and now we're introducing these super cool f-strings and now we break them again Um, I don't know. I I I mean this is kind of a joke But I like them like, you know, I like to not have to think about How I stay my code is just fine. But that's just my opinion Tomorrow is doing council members. Yes You know, I think the variety and code format is that are out there is a really valuable thing for the python community It it's It lets any development team Choose amongst themselves how they want to be developing and it doesn't it so it kind of says, you know any However, your team works Gets you can use python you can integrate python into whatever it is you're doing and it doesn't kind of say, you know You do it our way or or go away Which which I think is the impression you can get from some languages that are a bit more strict on here Is the one true formatting? If you don't like it go, I mean, you know tabs and spaces both work There's no problem there Variety of formatting and I've seen a variety of formatting I've seen teams who have been working together since the 80s in c pick up python And write it in a style that I've never seen before But it looks like the sea that people was writing in the 80s and they were really happy with it And they were happy that they could do it They could configure their format is to do it automatically And were very comfortable with that and I think that's a real strength for Python as a language and as a community that we don't exclude people By You know mandatory here is the format you will follow it so As a contributor to fedora which is a large collection of projects from all over the world Where we try to fix all the projects with every new python release and have to send patches to each individual project I don't like code for matters at all Please if you use the code for matter Make it run automatically on the pull request and make me not worry about it make it do its thing I don't care But if you if your ci tells me You're missing a space here install this thing and run it on your computer whoever wrote that I don't even know That I mean you know that's driving away contributors that you know have to touch lots of different projects That's my point and that's also a more fundamental Gap in kind of python workflow tools right now is we're starting to get there with like package Installation or package compiling where you can self describe In your pyproject.toml how your project should be built what package you need to do it What the commands are what the apis are we don't have things like that for projects to say Create the environment using this tool and format it using this tool Some other high level things do like a lot of ideas will have configuration files to provide that information But we don't really have anything truly python To that tells you that And again, you know, I also end up going through a lot of projects to make them work and yeah We dig in to see i configuration go Oh, they're using pytest and that extension and and any you know, it's just it's archaeology to figure out What workflow projects are using so I think that's a current gap that we have which also includes formatters Yeah, and there's kind of a a gap between us core developers and Kind of the packaging world. We don't really touch that And I guess that's what this would fall into packaging Some of us live in both some of us live in both, but there's like two different Organizations, let's say and I don't think that's very good so speaking as Just for myself as a cranky old developer I've never been a big fan of code for matters because I think code is fundamentally expression and this is sort of like Forcing everybody's code to like you have to follow this format And anything else is wrong and that's sort of circumscribing the ability of the developer to express themselves I admit like it back in the 90s I was working professionally in c and c++ and it was sort of a rite of passage You had to learn to read other people's code Which could be formatted in all sorts of crazy ways and I have seen some seriously Deranged formatting c++ even 90s era c++ was already pretty unreadable And it was just something that you had to get good at doing. So at this point speaking personally I can read just about anything and I don't care much about the formatting one way or another And I prefer I more or less follow pep8. It's not a problem But like I don't follow it strictly and I would sort of resent having my Code strictly pep8ified or having it be a requirement. So I don't use them on my projects. I think the the policy back in the day was Um If you if you own a file then you get to format it how you see fit And if somebody comes in to like make a check in or something they have to more or less like understand What your formatting is and and more or less try and ape it and obey it And and if you don't like formatting you can fix it whatever But like there were a lot of teams back in the 90s that didn't have a centralized coding format that was required And we survived and it we've reached today, you know, we're still mostly alive Yeah, all that is fine. But what if I want to indent with two spaces or one space? That's google google indents with two spaces Jesus Christ All right google has been using python so long like google's style code for python predates pep8. I believe and they use two spaces because You only have 80 columns right right Well, like linus has the same number of columns and he famously said like, you know, like if you have to indent so deeply that You know 80 is not enough for you then, you know, kind of you should just restructure your code. So Yeah, we'll find a bigger monitor Yeah, there's a lot of finger wagging and people who don't format the way that you think they should format their code I don't uh, so to be clear in 2018. I did not have this Ideas like hey, we should have more tools Right, you know, there was already a tool that I in fact contributed to in fact It was a google project called the app and we tried to adopt it at facebook at the time And we really tried I was on a team that was you know, really working on this and we could not because the Very intelligent way it's written and is trying to Optimize uh your dissatisfaction It's literally just optimizing a number that is assigned to how ugly is this line and you know If the number is the lowest that means okay of all the possible formatings That is the least ugly so great and you have a lot of controls in your config file that you can change behaviors So, you know, kind of some of the lines look great But some are less so and then people say, okay, we need to change this one formatting So you would change the config file and then you would fix this one weird line But other lines that used to be okay now are Unpredictably differently formatted and we played with this for a long long long time and At some point I just gave up and said like you know How hard would it be to just do something that just looks like json more or less and you know kind of and go from there And long story short when we you know kind of Published this internally it was okay. You know people adopted it and when we uh, you know kind of said like hey There's a new project externally It started gaining quite a bit of adoption So maybe it filled a niche even though the app existed and exists to this day It is still a tool So if you really need to configure a two space Formatters and single quotes and whatever else like you fully can like, you know, it's still a project So yeah, like all the formatters, I guess, you know, kind of it's true. There's many to choose from Some more popular than others Pi pi stats will tell you which one is the most popular You know a lot of steeper bids you from mentioning the most popular code formatter for python. It's it's blue, right? All right, let's get to another question One more observation by the way the first time I looked at code format is back in the 90s They were not as reliable as they are today So if 99.9 of the time the code formatter Changes the formatting of your code, but the code itself is unchanged and still produces the same output That's not good enough and that was kind of my experience like every so often you'd run it on it and now you're Well, this was a c compiler. It wouldn't compile anymore and you'd be like, oh my god What happened and it was the code formatter So that was a guarantee of black like from day one that like, you know, it does change the Look of the code but like it seems like an important feature and I don't know how we survive the 90s sometimes Ati Over the past Discussion petter talked about the formatters and looking out at the formatting community and Talked about f-strings and how it affected linters and we talked about the gill and how it affects Packages outside of c python itself and there's an ongoing discussion about the c api So I'm wondering how you guys feel about interaction with the external community what c python core developers can do to To make that interaction more smooth or or more effectual if you have any ideas Well, you could break everybody's code by merging the no go patch So I think we should keep existing code working so Not make backwards incompatible changes that should be the default position And if you're not sure if something is an incompatible change or not, then it's probably incompatible because somebody's reliant On the other hand, we also want some new features and we also want some new speed ups But whenever we need to do some kind of incompatible change then we should Announce it properly See how much code is affected Look how people are using the feature that is going away and try to help them Understand why it's being done and how to how to upgrade their code And also for anything new that's added make it very clear What is supported and what what we want to keep for the long term and what is an implementation detail that can change in the future And that is something that we've not been doing at least in the capi That well and I hope we'll change that I mean, I want to also highlight that we like there is a bit of a Difference on how people perceive this because I understand that it's very frustrating that you are I mean Going with what peter is saying that you go to 312 now and then you try your package and it just breaks right And you a lot of people are saying all, you know Cpi and core developers don't care about backwards compatibility and things like that But I want to highlight that the story is not that simple like we absolutely care and to highlight this I'm going to explain you a little story that is real and it just happened one month ago With involves some people in this panel actually So so so if you go to the cpython Like source code and then you go to the include directory and include directly you go to private Or cpython private or something like that and then you go to I think it's long object.c There is this big comment that says This function is only added here for cpython. Please don't use it Right and and that function is basically it's not a function. It's a structure Is the definition of the integer object in python, right? And it has one feel I think it's obval or something like that, right? So it is this big comment is the internal directory, but it's exposed So technically if you include the python.h header and then you use it it's there and it works And in 312 it may not work because we change it to make it a bit faster So integers are now a bit faster Although operation with integers because regions and then It does, you know, nobody should use it because it says it's prior But but it turns out that some projects were using that And we have a big discussion Because now what is this really private or not because even if we have this comment, you know, it was public You could use it and people may be using it So now the problem is that we have this huge legacy of like we Used to expose things without thinking and now we have all these rules But now there is disagreement between everyone is even people inside the core team of where this public and what it's not And because there's no rules we come with these rules as the cases appear, right? Because now oh, I want to change this. Oh, but it breaks these people So now, you know, it involves the team counseling and in both discussions with everyone So so my what I'm trying to say here is that we absolutely care and we treat these things seriously Um, sometimes it may still affect you because you know, we decide that yes It may break some things but like there is no other way forward But sometimes we you may not know How many times we have not go with one of these optimizations because we think that it's going to break someone, right? It's just that obviously when things work people don't reach to the internet to say wow, nice that is working, right? They only normally complain when when something is breaking But the message that I'm trying to say from the core team and the sting counsel is that we absolutely care All these cases is again very complicated But but we are thinking that we are taking care of of of this problem every single time We find one of these problems is just that you know, it's difficult to put everyone in agreement And and the reality is the the spectrum of kind of not broken to broken. It's it's not a binary. It's not useful as a binary like it's There's there's you know a zero case of nothing breaks And then there's the case of one thing breaks and it goes all the way up to infinite things break And if we didn't care we'd be up around the infinite things break every single time Um the fact that we do care so much is why we're typically down around the two to three things break And if we find out early enough because people are testing in beta then we can fix it And we never actually release with it So certainly like the biggest thing and I feel like we say this every single time we get up to do a panel If you have see extensions start compiling them as early as you can when we start putting our pre releases And if something breaks assume that it's our fault not yours and let us know If you can adapt unless you're siphon if you're siphon then it's your fault you can adapt to it Everyone else assume it's our fault let us know and we'll see what we can do to make it a smoother transition But that's absolutely the the Best way to learn what is going to break because we aren't always tracking everything and we aren't always tracking it in a way That means we can actively tell you As you know the example if something is very clearly labeled as do not use this We're going to assume we don't have to find out who's using it and tell you because we already told you not to use it So Test as soon as we put pre releases out and let us know that gives us the best chance of making sure we don't actually break All of your users on the day it comes out Yeah, I mean there's a few things we've changed somewhere I was Very confident that it wouldn't break anything and it didn't break anything and others I was a bit worried it would break something and it didn't and other things I was confident it wouldn't break anything and it did So anyone and what kind of worries me the most is where C doesn't really have any sort of clear semantics about it's just bits In memory and the worry is that there's a some assumed semantics of some extension and We have different assumed semantics and Then we break something but it hasn't it compiles fine and runs fine most of the time That's a bit of a worry In my in the deepest depths of my black heart There are some pieces of the c python api that I would like to change and just go ahead and break everybody's code and I'm sort of hoping like if we merge the no go patch i'm sort of hoping we can attack one of these which would be can you anybody guess borrowed references There's just as an example the the fundamental api you use in c if you want to get a value out of a dictionary It's called pie dick underscore get item and it's documented as returning a borrowed reference which means that Um the the dictionary has references to all of these objects inside and you say hi I want the one with this key and it says okay here you go Normally you would increment the reference count on that object because you're creating a new reference to it But this is returning a borrowed reference so it doesn't increment the reference count It just like gives you a copy of the of the pointer, but it's not incremented It's assuming we're assuming that the dictionary is going to keep it alive for you So this is a tiny dumb optimization Done a million years ago that probably made pie c python acceptably fast on 66 megahertz You know 486 or something, but this has been uh a Semantic uh flaw in the capi for a long time There are a handful of these functions that return borrowed references It has been a nightmare for anybody doing anything for example the no go patch I I believe that he just introduces a parallel set of of interfaces that return Strong references and it's like could you move to these please? And i'm kind of hoping that maybe we could kind of get there someday because of borrowed references I would just like to set fire to them so there's a bunch of bad apis like that and We mostly know about them And I think the best way to go forward is to make a parallel set of apis And then somehow tell people not to use the old ones and deprecate them and make them Amid warnings, but I think we should be very careful about actually removing them So maybe if people still need them despite all the warnings we can make them leak references We can make them slower. We can make them do all kinds of bad stuff But still keep code working Yeah, and the challenge there just to kind of finish Larry's example for why that borrowed reference from the dict is such a bad idea Is as soon as you have Another path of execution going on or even call back into python the last reference to that dictionary may be Removed at which point it goes through and clears all its references And it may deallocate the memory that that borrowed pointer is pointing at and now you have a use after free violation Which means we have to get a cbe and everyone gets forced to do an update by the security teams because you have a vulnerability in the product We we don't want to do that But it means that we can't allow deallocating the entire dictionary In parallel because you got that borrowed reference and there's just simply no way to figure that out Which means when it comes to leaking things, you know The way to make that work is we leak a reference to the entire dictionary And everything in it because you borrowed a reference which is really really bad for everything And there's there's a lot of these just have no good solution Which is why you know, we're being real careful not to break everyone who's using it But at some level the only way to actually get to a safe place Is to break it so completely that your code does not load Until you go in and fix it to use the new parallel apis and this is why we this is why we have these disputes And I suspect if you find actually you're leaving later today, aren't you? It's gonna say if you if you find you know the three of us especially maybe publo as well Then we'll happily have a big argument over all of this for you for the rest of the day We will resolve nothing But if you're if you're into the the deep and nasty details of the capi, you'll probably be highly entertained Actually that api is even worse. I mean, this is a bit out of the whole thing This is ranting out of the cpi But api is even worse because right now we have by dick get item with error Which makes you which made you think what is the problem with the older one? Which is like it's not reporting anywhere. It doesn't have any error. The new one's broken. It's the best api. It never fails So yeah, the original one has no errors No, it was yeah, it just clears it So, yeah All right, let's move on Next question, please. Hi, uh, thank you for your work first and the second one So in python zen, uh, we have this line like there should be one and preferably only one of this way to do it However, if we will consider even some basic functionality like Strings and interacting with them, we have this percent operator format format strings and also patterns Doesn't it look like we have a bit of ambiguity here and Doesn't it make sense to reduce it a bit? Which one's the obvious one? Yeah, uh, so there should be one obvious one for your use case every time Okay, there's I remember, uh Do you know who zed shaw is? I remember zed shaw complaining. He said, uh, f strings are wonderful And python should have should just have f strings It should have just shipped those and not done these other formatting things and I thought Well, we've lost the keys to the time machine We're unable to go back and change c python 1.0 to have f strings The sad fact of the matter is that like, you know percent formatting was wonderful back in the day It was so much better than a lot of other languages in the 90s And then we added a string template, uh, which I think still gets used for internationalization And then a string dot format and then f strings. I was like we What do you want us to do? We could have stopped at format and not added f strings Um f strings are the preferred way to do it now And we can tell you yes, we have these old ones and they're around for old code and you probably shouldn't use them anymore But f strings are the way forward for string formatting as an example We have the the c python, uh, standard the steiner library in python has three libraries for a command line argument processing get opt Opt parse and art parse And uh, everyone tells you oh, yeah, you should just use art parse and so like yeah But we can't remove the other ones because people still use them. So I'm not sure what to tell you apart from There is a best way to do I think all of the ones that I know of there is a oh No, this is the one you want to use and it's not hard to find out which one that is And I can't get mad at python for saying here's a much better one Here's a replacement for the crappy old thing that we used to have Here's a much better one. You should be using that now Yeah, that's the way to go Oh, ultimately the zen gets applied to an addition and to the addition as a whole So if someone if someone had come along and proposed percent formatting and string dot format at the same time Then we'd be looking at that and saying you're providing two two ways to do the same thing We should not do that. We should pick one way and make it obvious Uh, but that's when you when you have the choice to not have both of them And so the pointers are in this compatibility space where We cannot I mean if you if you want python with one obvious way to do it python 1.0 has one way to do it We we didn't force you to update What we did do is when you is when you update it, I mean we force in other ways, but not that way But when you do update we make sure that the original code doesn't work Does doesn't break does continue to work Even though you've even though you've upgraded so it's two Not even necessarily competing things and I think it's it's just a false argument to try and put them up against each other Like that. We don't we we can't apply the zen to history We can only apply it to what we're adding and developing as we go I think one of the key here is here one of the key things here is that the the obvious Like send like word there is very important because like the key here is that when you have like and this happens to all the languages like Some more than others right like c++ And if you think about it like you you cannot have like Like omni like presence and know what you need from the starting point, right? So you are going to have some strategy to keep iterating and your concern even if like what My colleagues are saying is it's true There is still one thing that we need to take into account Which is like when you add the new thing the better thing It it needs to be like fundamentally obvious that it's just the better thing, right? And this is how you add it because if you add just okay Let's have this new way to format the strings which is just Like better than the other one but just in these cases So now it's kind of weird because now as a new person that comes to the community and wants to learn python Now you need to know two of these like if there is like oh But there is this other third case that we didn't cover and now we have this third one And now you need to learn the three But obviously if you are going to teach python to someone and you want to teach them a stream formatting You're going to teach them f strings right away because it's going to be easier And then maybe maybe when you are teaching them Logging then you're going to say well there is other format like you know percentage thing or something like that Because you know logging has this problem that you probably don't want to interpolate the strings if the login Has not a level activated and all that stuff But that's kind of like already a super specialized thing and you could argue that that's a failure Maybe but like 99% f strings are normally better and they're much more powerful and much more readable And that's that's a good thing and I think it's key to know how to keep iterating over language features And in a way that you know, it makes them very easy to use for new users and for existing users to port to those things But obviously as my colleagues are saying we cannot just remove the old one So I think the obvious way here is the key thing in the sentence, right? I'm Friends with the the father of f strings is a guy named eric smith And when he was working on it, I remember him saying that he was working Very hard to make sure that f strings was the fastest way to format strings in python because he didn't want Anyone to have any excuse to use anything else? He wanted it to be the obvious best way to format your strings in c python. So as an example There's there's an old Idiom in python If you want to join together like let's say that you have 20 strings Do you want to join them together you could like say Let's say that they're they're in 20 variables a b c d e f g You could say a plus b plus c plus d plus e plus f plus g on and on and on That is actually kind of pathologically bad because the way that python computes this is it computes a plus b And allocates memory and stores it and then it computes that plus c and allocates memory and copies this stuff over and stores it And then it computes plus d which is we have allocated memory 19 times and copied this the first string over around 20 times It is much faster for you to add all of those Strings into an array and use say Empty quotes join parentheses array. So this is an idiom in python. It's like, you know, don't add strings together use The the string formatting or the the array and the join and all that sort of thing And I believe eric worked very hard so that if you want to join 20 strings together and you do it inside of an f String it actually is as fast as joining it with the empty quotes dot array thing because again if you wanted to remove excuses for Doing it other ways if you have an arbitrary number of strings, you don't know how many advance Of course, you still use the array method But if you have a fixed number of strings it is actually I believe faster to use an f string So that brings me to the recent changes in f strings So have you measured when you were making your changes like whether it's now like behaving faster or slower You know compared to the handwritten c code that was there before it's anecdotically faster like it's faster But like I'm not in a way that I could say Happily yeah, it's faster. Just that's one advantage. It's faster because Isn't there is no intermediate objects right because right now the first pass in python 3 11 yeah So in python 3 11 before our change You need to construct first a string token that has the whole string and then that is passed to the Stream person and that chunks it over but like, you know, there is at some point in time You have both things living at the same time the all the string and the new string And you need to put the chunks and that allocates memory. So right now that's done in one pass So the token actually chunks them up. So so that is the reason it's a bit faster But like the compared with the rest of the parsing and compilation times is nothing so it's like Yes, technically if you only have a program that uses f strings Yeah Is much faster, but like, you know, very unlikely that that's the case Well, there are two times that are relevant here. It sounds like you're talking about compile time versus runtime So I was kind of expecting you to be talking about compile time versus runtime because you're changing the parser and that is a Runtime is the same runtime. Okay. That's I assumed you didn't change the parser is the same and that's the key right because that's how we knew that We are not going to break anyone because what we say is that we are just changing the way we create the nodes But the nodes are the same and that's how we know that F strings are not changing anything. Well, except one thing What what is this? So there is except one thing So there is one only thing in f strings that goes all the way from the parser to the eval loop Unchanged and if you make an error, you will only know at runtime and this is the format specifier So the format specifier just is just placing the node from the parser and the compiler just puts it in in the eval loop In a local enable loop and that parsers and then executes and there was this sneaky sneaky bug Yes, so so we are using now the the C tokenizer the C tokenizer and one of the things that the C tokenizer does with So we were parsing the the like format specifier as a variable name because it has to be like a contiguous thing, right? So or one of the two there is three possible to the one that is just a variable name So we parsed as a name, but the C tokenizer over variable names does this Whatever normalization someone probably knows the letters like what is this a unicode normalization that is done NF key normalization thing. Yeah, so so it just grabs your unicode thing and normalizes So so it was normalizing format specifiers and of course Somewhere somehow there is this little town in like, you know the northern europe When there is this person who is using format specifiers with one of these unicode Non-normalize things and they appear on the backtrack and say hey You broke me and and we fix it But that's the that's the only thing that you know, it affects runtime, but the rest is just the same. It's just the same nodes So f string parsing should be faster in 312. It's faster. Yeah, it feels more snappy By the way, we almost it was three times slower So so yes, so actually we we thought like we we are not changing the whole thing in a Massive way that it should be much faster much lower. So we never check And we we did the whole thing We have the pr prepare and then I told batuhan one of the people that is doing the thing Can you measure if it's any faster? And like he came with the results and he was like, you know, four times slower or something I don't say whoops and the thing is that we were like apparently What it adds is to support Jesus Christ to support the debugging feature like the equal sign So that is just very hard because the parsers expect tokens, right? Like the parser doesn't care about your source. It's like Full and that doesn't know that it's full. It says like name But for the source it needs the for the debug like thing It needs the source it needs that if you write one plus one with three spaces It needs to respect those three spaces. So it is the source So we keep adding the source to some buffer and that was quadratic because every time we see a new character We allocate a bigger buffer and then we put the new character and that was kind of stupid So we fix that. Uh, so yes always profile your code. It kind of matters. Exactly, right? Cool So we have still three questions. We are a little over time, but now what's happening is the coffee break So maybe if we're quick about it, we can actually get all three people to to to ask their questions Okay, now we have two We're doing it. We're doing it. Yeah, okay. So let's have the question. Thank you. Hello Uh, so my question is about AI tools like chat gpt and github co-pilot I start using them recently in my development workflow and uh, and I've been pretty amazed by what they can do and I someone did I study I run recently that python is really a great language for that like if like they They gave a coding task to chat gpt, I think and Compared different languages and python was the one that got some percent, uh, correct with the testing suite and Mostly correct with the other testing suite they'd given They didn't give a chapter anyway So this to say that python is really a good language for this kind of tool and My question is have you been experimenting with these tools? And do you expect that this could affect the language design? And if so, how? Thank you. Am I getting the looks? I I've I've not done much of my own experimentation with it though. I have used it a bit Obviously they work at microsoft So a lot of these tools are things that I've had access to for a little while to to play with absolutely like python is a great language for it and You know, I just feel like the nature of those tools Doesn't lend they don't lend themselves as well to to grammar and punctuation or to punctuation as much as kind of natural language Which is I think why python Seems to fit in a bit more cleanly, but it does do very well on a lot of cases I would be very surprised if We bias language design to make it work better for those cases. I think it's really a situation where We're talking about developer tooling which tends to be considered a level higher I mean python in general has not been biased for the sake of developer tooling. We haven't Apart from kind of type hints are about the only sort of concession we've made for hey I I want to know exactly what members of this variable that I've said nothing about. Ah, it's like I've typed def f With you know parameter a now tell me what members they has every other language particularly those that You know really try and work with developer tooling will make you do extra typing before you can even start using that So that they can tell you things about that variable python doesn't make you do that and as I say type hints are the only sort of concession We've made that way And so I I'd be very surprised if if we do Start making kind of more concessions for a developer tool, which I We're certainly is how I view a lot of the AI stuff. It's an interaction model I'll probably talk about I'll I'll probably get on a rant in my talk tomorrow about this It's not it's not in there, but I probably will So so I think it's just a level above like we're not designing Python being good with it is it's is an accident I think well, maybe or maybe this python developer is building the models And so they test on their own code and they tweak until it works well I don't know, but it's it was certainly not intentional from a language point of view uh, but again We see how the world changes like 10 years ago. We wouldn't have entertained code completions Like type hints for the sake of making code completions better In 10 years time, maybe we will entertain certain design features for the sake of AI generated code I think like there is one aspect that I may consider which is that um, I was reading this paper that were Mentioning that a lot of people are actually using these AI tools not to just generate the code and whatever But to explain errors, especially beginners And they they post like based on code that doesn't work or they maybe even the error message and like can you explain what is wrong And the paper was concluding that because at the end of the day the the quality of information Is just depending on the data that you feed they were concluding that if language Like if the errors that compilers and interpreters are showing are unique per kind of error instead of reusing the same text This will help these tools like identify better these errors for beginners So for instance right now that we are adding all these new error messages for the parser This makes me think that if two situations are almost the same in the past I may have used the same error, but now I may tweak it slightly. So, you know, like is this is different So if the tool picks it up it can teach the person a bit more about what is going wrong because it costs me nothing And it's just like nothing I wouldn't call this like specializing for AI, but it's clearly having this this in mind Like a normal person reading is not going to notice or notice the difference But like a person may be feeding this thing to the AI may but yes, I don't think there is any other way when this is relevant Cool last question of the day great great stuff you're doing and this is back to the Multiple ways of doing the same thing in python. I understand all the reasons for it. I'm very familiar with it But have you considered for for new developers? Have you considered? Documenting if they go to a page where it's talking about a feature that they shouldn't be using Have a very strong language that you should not be using this Rather than so like recently I wanted to do something and I was like I can I can override get adder or I can do abc metadata or I can do descriptors within itself class Um, how do I know which one I look at the number of the pep related to each feature which one is hired? That's probably the one to use So documenting that way one way the other way would be like Would you consider adding a feature to python that would limit compatibility as in like I run python and runtime It would refuse to run certain code that is old syntax, but by default it supports all the old ones So for documenting we just talked about that on the last documentation work repeating I would adding some some icons like they have on on the mozilla network for web web apis They'll deprecate it or experimental and things like that might be neat We do have deprecation errors. So if something is clearly bad, then you do get a warning And right we don't have any solution for something being slightly worse So you shouldn't use it a new code We might There are some libraries that that are documented as do not use this library Up to parse I'll pass um a lot of url lib. I think it's marked now htdp server is is marked as but mark is unsafe Yeah, it's it's my well, which is because every you know, if it's there and it's usable Yeah, but that that that is complicated because stdOS parse is also marked as unsafe and everybody uses it Yeah, eval is unsafe and people use it and a lot of time it's not unsafe It's one of those things where we can mark. Hey if you're You know, you need to evaluate your own Situation your own context using this in because if you're going to put it up as a public endpoint on the internet You have a very different situation to if you're going to type it once at your own machine and look at the result and labeling stuff Is you know runs a serious risk of People typing stuff for the very first time in their own console seeing warnings and going. Oh, no, I'm getting it all wrong This is so hard. Why is this shouting at me all the time? I quit python I'm going to go learn something easy like javascript Or while while those warnings are actually meant for the person who's about to publish it on the internet and let the entire world You know hack their toaster So so like the warnings are for different people and there's only so much we can kind of very bluntly do Because it depends on the person using it and how they're going to be using it So we're kind of leaving a lot of this to linters. So if you Use some kind of tool that tells you the quality of your api We might want to give them better machine readable information, but we'll probably not like Give out warnings use bandit All right, cool We are 10 minutes over time. So I think you know everybody's thinking about coffee now already So yeah, like we could do another hour of this but you know, obviously the coffee break is very important So thanks for your time And feel feel free to come and find us and chat with at least me at least I don't know who else is volunteering for that but if you find us feel free to just grab us and have a chat Yeah, we're here for the rest of the conference. So if you have more questions, you can always just find us in the hallway We're always happy to talk to you