 Okay, I think this is the first time we're using this mic today so I can get a show of thumbs up if everybody can hear me. All right, perfect Okay, so first of all after you already said we'll hopefully get the most boring part of the conversation or the talk out first That's my name and if you've been to our booth, you know that this is like our adorable little logo His name is Constantine. He will make quite a few appearances First and foremost, I am like they said a Kraken engineer for working for octopus energy or Kraken technologies We build Kraken which is like a big software platform for green energy companies If you want to talk way more about that definitely come and see us at the booth And then just like trivia facts like I'm also a yoga teacher You might ask yourself like why the heck would he mention that and it's for no relevance at all other than it took me a Very long time to do so I like to brag about it as often as I can so if you are interested in yoga I know that we have like these open spaces So like if anybody wants to like do an impromptu yoga session tomorrow, maybe we can organize that we'll see but yeah Again come and talk to me about that later. That will be the end of the yoga portion of the talk So, okay, the talk is lint all the things. What is I guess we should first start off with like what's a linter? And this is I think I think I got this from Wikipedia So analyzes code to flag errors bugs stylistic errors and suspicious constructs as Oftentimes I find stuff on Wikipedia is pretty wordy and may or may not actually be true So we'll start to like kind of break that down into something that makes a little bit more sense For us, but it does do I mean this Like does make sense and it does do all these things that kind of it's there's a little more nuance involved So I like to think of it as like three things a linter can be a critic So it can complain about your code like when you do something you probably knew you shouldn't do maybe the linter can say like Hey, by the way, that's not good. Don't do that a lot of the way we use linters as well at Octopus and Kraken is as a teacher So sometimes people can do something in their code and you know that they can do it in a better way So not only can the linter be a critic, but it can also say hey by the way I think you were trying to do something like this maybe you should do this instead So for junior developers that being like very helpful and like them learning best practices and stuff and then this last one Since you can probably already tell from my accent I am from the States and I used to have the word janitor here Which I talked to my UK colleagues and I realized that that word is probably not gonna make a whole lot of sense So I changed it today to caretaker, which is like someone to come up and clean up behind you So we also do that is like linters as well Like maybe you kind of want to make a little bit of a mess and then you want something to come back and clean It up for you. It's really great to use computers for that. So linter can kind of be a caretaker as well So why okay, so now we know sort of like what a linter is for and what kind of things it can do for us But like why do we actually use them and for me the first and foremost reason is because we're humans and humans make mistakes As you all know if you've written any line of code. There's a pretty good chance. You've written at least one bug Or often we write bugs that just happens because software is very complicated And because the discipline itself is actually pretty young in comparison to like other disciplines So we're all still kind of figuring this out together But since we're humans we make a lot of mistakes Another way that we can use like a reason to use linters is we can codify these best practices So for like some of the people on our team that have been developing software for years and years and years You kind of get the sense of like what's a good thing to do or what's not a good thing to do And that typically comes from like doing the wrong thing a bunch of different times So like me specifically I've done the wrong thing a whole bunch of times That's why I have like a set of best practices and those come from mostly just doing it wrong But instead of just like putting that in a documentation or something like that We can codify those using these linters and start again like taking a linters like a teacher We can help to like junior developers realize that these best practices exist and how to use them Another reason is consistency So I always tell people on my team if I open up a file and I can point to the person that wrote that code That's like not a good thing because you want all your code base to look very similar And so that it reduces the cognitive load when you're like doing PRs and stuff like that and In the Python like world we don't have to worry about this quite as much because like as everyone knows It's why space delimited so we don't have to worry about like to have that conversation of like where do we put curly braces or any like That little stuff we have like our own arguments. We don't have to worry about the curly brace argument We deal with like the hanging like parentheses or something like that is kind of the equivalent but yeah, so consistency and then Going on with consistency. It also makes for faster code reviews So at octopus we have like hundreds of developers and we're working really really fast So we want to be able to like review code as quickly as possible and not have all those conversations about like oh Actually, you should like put it a blank line here or you should do all this like other like a little like Knit next stuff that doesn't really matter as much So we can like strip away as many of those things as possible And so the code reviews are really focused more on like the actual functionality and the changes that are being made as opposed to like little like things and Finally linters are really great because criticism I find at least I don't know about everyone else But criticism from a computer is much easier to take than from another person. So if I push a like a bunch of code and Circle CI or whatever CI system you're using complains to you That's much easier than like your boss coming over to your desk and complaining at you about that code I at least for me I think it's easier So we can codify a lot of these best processes and let the linter complain or let the build break it turns out like to get developers to Code in practices that you want if you fell the build like they actually start to follow those practices way more than if you have to Code at their desk all the time But you might ask okay, those are all pretty good reasons, but like really really why For us it's because of this Like I said, we have 300 plus developers all around the world nine different countries And as you like played our game yesterday, you would know that we have about 2.1 million lines of code inside of cracking So it's a lot of people working on something all at the same time So we can't just rely on just code reviews because someone comes on the team We try to get them to do a PR like in their first week or something They don't really know any of the conventions and then like there's lots and lots of PR is merged every day So we need to automate that practice So just like a tiny bit of background if you haven't been to our booth Yet and you want to know a little bit more about cracking It is like a giant Django like monolith application Supports millions of customers worldwide and then I think I need to update this because someone said it was 14 But I can't count to 14 I can only get to 9 so I'm not entirely sure whether I'm wrong or someone else told me wrong But yeah, we're adding more and more countries all the time and like I said like 300 plus developers all working in one repo And then yeah like I mentioned before it's all for like I updated this slide because it recently was like I think I did this talk a couple of months ago And it was only like 1.5 million or something it is 2.1 million as of yesterday when we ran this So if you can imagine yourself like being a developer that has worked for maybe like two years at a company of like five people And then you get thrown into Kraken which is very much like the actual Kraken It has all the all eight tentacles It is massive being able to work on a code base of like two million lines with 300 developers is a pretty difficult position to be in So this is what it can feel like sometimes if you when you first come to Kraken When you start working on it it can be pretty difficult So Constantine sometimes has a fever sometimes you get a bump in your head or sometimes your brain is just completely on fire because it's just so much stuff to put in there So the one thing I want to mention because I know we're at a Python conference So of course everyone knows like I'm going to talk about Python linter's But one thing to note if like this is your first time coming to the like linter world or hearing about them This isn't just a Python concept this exists in a bunch of other ecosystems as well Into the non-Python portion of the talk Okay, so here are the come of the tools that we're going to talk about briefly We're going to give you like a little overview of like how we use these different like linter's and formatter's and stuff in Kraken And then I'll give you a little overview about like how we actually use them So first up is black. I know the creator is here earlier. I talked about the booth And it is known as like the uncompromising code formatter I think that name probably comes from when black first came out. It had almost no configuration Basically on purpose So if you've ever been at a company and you start a project and you have like 10 software developers pretty much the first thing You do is you all get in a room and you complain about style And you just like argue and argue and argue with each other and you spend a couple of days And then everyone's mildly unhappy, but you kind of settle on something. Well, so black skips that step I like to think that Black takes a lot of the configuration out. Everybody is still mildly unhappy, but then you don't have to have that argument You just like start with blacks configuration go for it But I think black is probably easier to understand if you see what it does So this is probably way too much code for a single slide, but we'll go with it anyway The code itself is not too important. So if you look at this like maybe you write code that looks like this I mean, this is valid Python. It does work. I do have this pointer thing. So this is pretty neat I'm like, you know, this looks a little weird like I probably wouldn't wrap the code like that I probably wouldn't put a bunch of space like outside parentheses and stuff or quotes like that But what you can do is you can throw this code into black and you end up with something that looks exactly like this So notice we have a lot of white space Here to make everything a little easier to read And everything is like wrapped nicely and it's just I don't know I feel it's a little more comfortable to work in and I always tell people on my team too like sometimes people will write code like this and I tell them like, you know white space is pretty basically free these days Like let's just like let the code breathe a little bit so that you can read it and kind of break it up into little more logical Concepts or look little chunks. So that's what black does you feed it something that looks like this and it gives you something It looks like this my personal favorite thing about black or maybe my least favorite thing about black I'm not sure but is that I really hate typing like double quotes to me It's like an extra keystroke drives me insane. So when we first switched or when we first first started using black I was like, oh it uses like double quotes. That's very annoying But what I found is like I can just set auto save in my editor to throw it through black And so I can write all the single quotes I want and it turns out looking like this and everybody's code looks the same So what black gives you is a lot of like this consistency and again You get to avoid that whole argument about what is the style look like? Just like a couple of like interesting little things that we do in Kraken since we have so much code and black can be kind of intense to run We I think 99 might be the default in the black configuration now I can't remember exactly but that's something that we do extend also like I was pretty like hardcore supporter of like I sit in Vim all day So I'm pretty hardcore supporter of like 79 column links But once you start adding like static typing and stuff like that type definitions It's like really hard to do anything in 80 anymore. So 99 is kind of what we use and then We exclude like we have some front-end stuff in the Kraken mono repo We don't like auto format those because it's mostly like JavaScript code We're not gonna like spend the time black looking for Python files in there And then we also don't auto format like our migration files If you open the file up and saved it and maybe had your editor set up that way that would work But we don't bother like auto formatting those is any either because those are like auto generated code So another couple like interesting things I find about black is like the one of the ways we use it is Like I mentioned before I put the state like I put an auto save on my editor and then it just auto formats the code But we don't use that and then like our CI pipeline or like in our pre-commit hooks or anything like that Which we'll talk about later. We use the dash dash check option So all that means is it tells black like if it would have reformatted any code to complain and like return Like failure to the shell There's a couple of reasons the reason like that we do that But like I think the most important probably is that like if you're running black and something like in a pre-commit hook The diff that you see is the diff that actually gets committed enough We weren't using the check option then if we maybe ran black after that you might end up getting something checked in It wasn't exactly what you see and then the other real reason is so that like formatting the code is still like the developers Responsibility, but you can just set black up to auto save in your editor Another thing that I haven't like I think I don't know that any other formatter There's like quite of other like few other formatters. I think there's even one called blue It's like a fork of black with like a few configuration options tweaked But something that like is very cool about black is like I first when I came across it I was like I'm gonna give it my code and then it's gonna change my code and then commit it Like that's very weird like I spent a lot of time trying to make this code work And then something's just gonna change it for me. That seems kind of unsafe So black actually runs by default I think with this dash dash safe option and what that does is Black actually looks at the AST the abstract abstract syntax tree And then it reformats your code and looks at the AST again and make sure that those are logically equivalent So it's kind of guaranteeing like the run of the safe option Which again? I think is the default that it makes sure that it doesn't actually logically or like change your code semantically It's only just like the way it looks and I don't know if any other code formatters do that And I think that's probably a very very difficult thing to do. So I want to like point that out So like using black in safe mode is a great way to go if you want Like I think really the only big benefit if you turn this off is that black does auto format it quicker Because it doesn't have to do this AST like comparison stuff So that's black so you if you're like me you type a bunch of single quotes you hit You know colon W it and your vim editor or in your pie charm or whatever you want Auto formats your code and then you get double quotes and all this like nice like nicely formatted code And then everybody's code in the file looks the same So the next up is I sort so what I sorts claim to fame is like I think it's kind of clever It's I sort your imports so you don't have to it's kind of like a nice little name So what does I sorts do? It's another formatter or kind of like Yeah, like a formatter for your code, but only deals with imports So if you're working on a project that's as large as Kraken It's pretty common to have like lots and lots of imports in a file So it gets very difficult to navigate that group of imports like if you want to try to find a particular import or Oftentimes like maybe you have duplicate imports and you don't notice them because the import section is so large Like you just add a new one so what I sort does is again kind of like black It's very useful to see if you just kind of like see what it does So like you throw in code like this and Then you get something out like this So again, there's like a lot of code for a slide the code is not super important But one thing to like note is that like there's some blank lines in there Like I said white space is free. We should all use it more So it's broken in but the little section so like the first section is the like built-in stuff And then the second section is like the third-party stuff and then the last section is like the stuff That's actually in Kraken so this is gonna make it a whole lot easier to navigate those imports It's like you want to add a new import. You're like, oh, I'm adding like a new import from the standard library What I tend to do is just like add it at the bottom And then have I sort do like autosave stuff when I say the editor and it just moves everything around it Does it for me and then it I think it also removes like a duplicate imports So I don't know if about you but like if I'm given code that looks like this And I save it and it looks like this. I kind of feel like this. It's pretty great Like I get pretty happy. I don't have to think too much about it Everybody's another really good benefit is that everybody's imports are in the same order So it's alphabetically sorting them as well So like diffs you don't really get diffs because someone moved an import which is very annoying and again It contributes to like faster code reviews and stuff like that because you're not having to look at a diff of like Import lines just moving around So I'm a few of the configuration options that we use and I sort I sort is unlike black And then it has like tons and tons of configuration options plenty of which that I don't even know or understand But we turn some of these on if you're like interested You can tell it what's like a known third party or what's a known first party because it doesn't necessarily know about your imports Other than like what's in the standard library and what isn't so that way you can get those like breakups in the the chunks Yeah, and then we allow trailing commas and stuff like that Similar some of the options that we use I sort has the same kind of thing is like Fail if it would have sorted your imports except for it like in black. It's dash dash check here It's check only so what that does is you feed your code into I sort and if you do check only it will fail and return Failure on the command line to tell you that it would have sorted your imports or not So same kind of thing as we'll see later We put it in our pre-commit hooks and then we can fail if like you didn't sort your imports So again, it's another thing that like you put in your editor you hit save you really don't ever think about it again So now we've got like our code mostly formatted and it's consistent And then we've got our imports mostly for all way formatted and consistent Like now we start to get into more like probably more traditional lenders and not necessarily code formatters So flake 8 it's not as quite as clever of what they're like tagline is But your tool for style guide enforcement But I kind of think that this is like really selling flake 8 a little bit short for just like style guide enforcement It turns out flake 8 can do like a whole lot more than that I like it just the other kind of sell themselves short a little bit But again, let's just see an example of what flake 8 can do so that we can see and better understand what we're looking at so again, this is like really tough to see but here's a bunch of code and Normally if I would like we're gonna do this in a smaller setting I would ask someone to like yell out But this code doesn't actually work and the weird thing is it's like there's this if statement here That says like okay if you're on a weekday do this if you're on a week and do something else So like basically this function runs sometimes and then this function runs sometimes So down deeper like this function sometimes gets called and sometimes it doesn't so Often like or everybody knows like or potentially everybody knows in the room, but Python's interpreted so You don't necessarily hit an error until you actually execute that line which makes it very difficult to write really good code That's why I like a lot of the like old C developers like me when they don't lose their compiler And they move to like a language like Python you're a little bit worried like you don't have types And you don't have a compiler like how am I supposed to actually code reliably? This is kind of the Python answer or one of the answers to that so again Maybe you've already seen it before but this code doesn't run because of this So instead of the word to do's I type to do Very easy to miss we have we all have very complicated lives what's going on like it That's a very easy thing to miss But if we throw that code through flake 8 it has a whole bunch of like configuration and like default stuff to tell us About like maybe a typo and a variable name or all sorts of different things that are just built into flake 8 Which is pretty great So that gives us like a little bit of like what I would call static analysis on the code So it's it's not like a compiler that's complaining about types or something like that But it's at least like a little bit better and trying to like catch like more obvious errors and then just you know Maybe your website will be down a little less But wait actually like I said before flake 8 actually sells itself short in my opinion about the like just a style guy Or a code format or kind of thing and then just like another cute thing a Constantine to pause and just like wait for everybody To like soak it in if you need to shed it here because it's so cute. That's fine, too So what else can flake 8 do it's actually fully customizable using Python So you might ask like what does that actually mean like flake 8 is written in Python? Like what do you mean by customizable? It means that you can write your own flake 8 linters or your own like plugins. I forget exactly what the terminology flake 8 uses But that means that like if we have a coding convention like it for example a crack and we have quite a few We can code those into flake 8 and put them in linters You know set up our setup config file and that sort of thing and we can actually complain at the code if you do something So here's one that I kind of like simplified from cracking actually Since we mostly develop like a Django application. We don't allow you to ever like commit code with the print statement Just because it's not super that use like it's not really all that useful Like if you're shelling in and like looking at the logs of the web server like you might be already in the weeds Right, so we're just going to not allow you to do print statements. So it is kind of funny like when you first Start working at octopus or kraken you might commit some code with a print statement to build completely fails Luckily, you can never merge to master like that So you're usually the only one that sees it but it's kind of funny to have a linter come back to you and say like hey I failed because you have a print statement So that's kind of the idea This is like the talk is not quite long enough to go through like the details of how you actually make your own flake 8 Linters and stuff like that if you're super interested in me come and talk to the people at our booth We write a lot of these and like for various different reasons But the main idea is that like in this scenario we want to just allow disallow code to be committed with the print statement So we would you know implement this the thing about flake 8 is it gives you the AST So that's how you can kind of tell like the AST is like a breakdown like an abstract syntax tree of the code So it's kind of this like structure that you can navigate broken up into expressions and like comparison and like operators and stuff like that So flake 8 does all that parsing and then kind of just gives you the AST And then the files and like the file name and the lines and stuff So we can just write some like simple reg X's to say like hey this line actually had print We could do more complicated stuff if we were using the actual AST in this scenario We're just doing like a string matching in the lines So like flake 8 allows you to do a whole lot of complicated stuff This is like one of the more simple things like you could have written this with just like looking through every file And like writing this reg X so it's not like the best example of like flake 8 complexity But it's an easy one to kind of grasp and get the idea So yeah, and then also just like another little thing. We also don't allow the pretty print Just another thing that doesn't really make any sense in a web app usually So I went through the code base and just to give you like an idea of like stuff that we have in Kraken We don't allow unit tests that use the database So we have linters to disallow that sort of thing because unit tests should be completely separate from that stuff You shouldn't be talking to the database. That's like an integration or functional test or some other category of tests We also have conventions that all of our salary tasks need to be in a task stop py file Another pretty neat one is that like consistent naming of date time fields. So we have like hundreds and hundreds of models so we always name things like Start at or end at or start from effective from or effective at and stuff like that You have to have the underscore add and underscore from so when you open up a model It has all these like auditing fields on it or it has other stuff like that. They're all consistently named So if you add a new model and it has a date time field and it doesn't can like work with this restrictions Then the build or like you will not be able to merge it Another one that it's like super super useful like We talk a lot about like code coverage in Python community and people are like oh my code is like 99% covered But it doesn't really tell you all that much because like what if all your tests don't have a cert statements Probably all gonna pass. So you have a whole lot of coverage and absolutely no testing So it's pretty easy to accidentally write it This is again like speaking from experience It's pretty easy to write a test and forget to add the assert statement And you basically just wrote a test that always passes So we have like linters that like detect whether or not you wrote a comparison and say like hey by the way I think that comparison should probably be an assertion and Then we do some like progressive type endings. So cracking such a large code base been around for quite like five or six years We are slowly adding typing to it So we have like linters that say when you add a new function you have to add a return type So we do flake eight on just the diff of that file So that like when you open up a file and say it has like a thousand lines in it and we just added a new flake eight rule It says you need to have return types on all your functions You don't have to be the person that goes through and like as the return types to like 15 functions or something in there Instead we can just do the like flake eight can take input from standard in and then put it back to standard out and stuff Like that so that we can see That we only want to add like return types to the function that you actually added in this commit as opposed to like globally So we have tons and tons of flake eight linters very very cool If you're super interested in that like definitely come and talk to me afterwards And then just in the interest of time will keep moving forward. So my pie is like a static type checker My pie is like a talk all on its own But it's kind of like us trying in the Python community to get like a compiler and to have like type checking and stuff like that so like some interesting things that we happen to have in our configuration is like Doing like strict equality. So like is versus equals equals and stuff like that or like finding unused Like ignore statements. You can have those like I think it's like pound no qa or something like that depending on which one you're using So like a lot of these are like kind of minimal things that my pie can do What my pie really can do is like find out that you're passing different types of different functions and they're not them not not matching Also import linter is something that was actually developed by developer David Seddon at octopus and it allows you to define and enforce rules for your imports Which means like we have a very layered architecture in Kraken So we don't want like a bunch of circular imports so like we like our dependencies kind of flow down and you can Define these layers and import linter was like a config file saying like this layer should never import this other layer So in a code base that biggest Kraken this can really get you out of like import hell very easily Yeah, so I encourage you to check out import linter How do we use it? So there's like local and then there's CI. Here's how we do all of our stuff So locally you can do it with git hooks. These are built into a git and they're easy to create and install And they have lots of them like called like pre-commit post commit post merge stuff like that Warning though when I say pre-commit, I mean like pre-commit and git not the pre-commit project And people always ask me like well, why don't you just use the pre-commit project? It's mostly because a lot of this stuff was added before the pre-commit project was like mature enough So I don't have a great answer for that other than like time So, okay, those are all the linters we have finally like Luke get to the point like how do we actually own these things? So like I said before we can do it locally and on CI So there's two steps to do it locally. We need to figure out what files to lent and then actually lent them So a pre-commit hook in the git terms is usually just like a shell script or you can like call whatever you want In our case, it's usually just a bash script So we do some like a little bit of git magic to get us like a diff and then get like the name Like the name status which gives you like an a and m or d next to the file and we pull that out So now we know which files that we need to lent so we know like we only want to lent files that were added or modified Not add files that were deleted. That's not super useful So once we get the list of files to lent we can throw them through all of our different linters with different Configuration options that we have we've already talked about before so throw it through flake 8 See if black is going to complain and see if I saw it's going to complain if it doesn't great You might notice that my pie and import linter were not in that list There's a good reason for that because my pie kraken is very very large and my pie does a lot of very complicated stuff So it takes a long time to run on kraken So as a pre-commit hook when you make a commit and you're sitting there about ready to make a commit We don't want you to have to sit there for minutes while my pie tries to figure out what the types are and Similar thing for import linter. It's like looking at all your imports So it's doing a lot of work. So both of these are just very slow to run as a pre-commit hook You can run them outside of the pre-commit hook, and then they also run on ci Downsides of git hooks is people have to have to install them You can usually sim link it in your environment So it's not actually very hard to install But that's one extra step also get includes this like escape hatch dash dash no verify that just like skips the entire Pre-commit hook so if you're like one of those people it says like sets up an alias with no verify and you just do this all the Time not so great. So again, you want everybody to be using them Like if you have all these linters and no one's using them It's pretty useless and then also to write git hooks usually you have to know bash Which is just like not a great time if you're like me you're like looking up how to do four loops all the time And bash even though you've written bash for a very long time So the best way to really enforce this stuff is to integrate it with ci The main reasons is guaranteed to run and then it also runs in a predictable environment with like all the same configuration and all the same versions and stuff like that So I think I have one more minute to go Thanks everybody for coming if you are interested I think we'll wait and take questions at the booth if you haven't we are going to Give away a big plushie Constantine if everyone hasn't seen that so we're going to go and do that But if you have any questions about all this like Linting or have any ideas or any experience with it definitely come and talk to me afterwards and Thanks everybody for coming