 First up, yeah, I thought we'll get the party pooper out of the way Mikhail Svetlik The title is Golang versus Python and why we are losing Don't don't don't shoot me Yeah, I'm good All right, so some yeah, I had to get it off my heart like I said in my previous talk I come from the system admin background operations and What I see in my field is that we used a lot of Python in the past and nowadays We switch more and more of our stuff to go And the reason for it is I don't think it's performance like yes go is awesome for those quick API micro services that you can just Do it very quickly and they're they perform much better than just stock Python But I think one more like contributing factor to it is how easy is it to set up your Python project on somebody's Laptop so if you think about you create some awesome CLI to in Python now your user needs to install it He needs to go to this world of pain of peep virtual lens Python dependencies do I have Python free install Python to install all that good stuff? And Go makes it very easy the tool set is just awesome You just do go go run go install you create a binary you can cross compile it in just one command and you can give your user Just an executable and you're done So this is like I don't have really a solution. I guess I tried few solution in the past I tried using pine store, which if anyone knows Or pi to exit so basically they create kind of like a one binary for your Python with the whole Python included inside It's not trivial you need to work around with it and Basically you kind of like you become an expert in this to use it and you don't you can't really cross compile So you can't use Linux and create by installer bundle for your windows You need to like use virtual machines all that good stuff the other solution is just to rely on your like distribution in a distribution I guess packaging and create create RPMs create devs create Something for Mac and something for Windows Windows don't have Python installed by default. So the game is over there So I guess I don't know. I guess it's kind of like What do you maybe maybe somebody have a better idea for me better solution what we can with yes Go Okay, cool. So so please so please come to me after this Yes Yes That's that's a good point actually container. So not even containers one one other option is now stuff like flat flat packs and in Linux there's like Snaps. Yeah, like obviously in Linux, you know, like have five different options to do the same thing And now they will compete for like 20 years and nobody will win So so yeah, so we have another option. Yeah So I guess the bottom line is I Think like if anyone here got any connections to the like Python software foundation like This is like a real problem I think that Python is losing customers and losing users because of this like packaging problem this cross compatibility problem of your like how hard is it to install your your awesome Projects on somebody else computers. So I Guess think about it. And if you have any any any good ideas what to do. Let me know and definitely will catch up Cheers. I'm done Perfect timing Perfect timing Could Ben Denham, please come down to get mic'd up next up we've got John Graves Talking about Zappa serverless AWS Lambda And he's put a URL in his talk title bit.ly Slash Zappa 17 Hamish. Where are you Hamish camel? So there's a talk to this afternoon on this whole topic. So I'm really apologetic that I Might be stealing a little bit of the thunder, but a lightning talk called Zappa. I mean, how good could that be? So quickly how many people know what AWS is? How many people know what AWS Lambda does and how many people are familiar with Zappa? Okay, so there's a little bit of room for me to help you out with a really cool simple tool that Gives you this no permanent infrastructure serverless is about having systems that are there on demand You don't have to maintain any of the infrastructure and it really is this easy pip install zappa You've got a folder and you're gonna zappa in it that gets all the permissions and S3 buckets set up on Amazon for you zappa deploy you're done You've got a Lambda function deployed on AWS and this is what it's gonna do for you a request comes in It gets turned through the virtual tape library. I didn't know what VTail was until I saw this slide last night a Server on the fly is created. It serves up your your whiskey Request and then it dies. It's destroyed and this all happens in 30 milliseconds Your python logic can be executed implemented and done and then you can forget about any of the maintenance You can fire your DevOps team is the is the claim so I Actually had a demo of this on my other laptop, but I'd like to just ask who would use a service like this if All right, and who has and what have you done with it? Is it great because it does so little or okay, so the the discussions yesterday we were having was Constellation of these things is Step functions and microservices. So I think that's what the the subject of This afternoon's talk may be if you want to get into using this approach to avoiding Docker avoiding other kinds of deployments if I could just take up the last little bit of my time to promote my Step-in talk for for Ned not being here. I am gonna show you something cool that you can do with your bare hands computationally in the seed new things and act talk and you can Learn a little bit about the superpowers of big data. So thank you very much Yes, thank you. All right, Samuel Bishop if you could come down to get my top next up. We've got Ben Denham giving us tips for Python web scraping you got 10 seconds for those slides and go Thanks very much So I had to do some web scraping with Python this year. So I thought I'd just share some practical tips I learnt some we learnt the hard way So the motivation I needed to collect data from about 13,000 different PHP modules from the Drupal CMS Each module required scraping several web pages And a source code tab all I needed to do some static analysis on and the scraping all up to about 60 hours So the tips I'm showing you here really saved me a lot of that time First of all, if you don't already know about it beautiful soup is how you can do really easy HTML passing in Python It allows you to kind of do CSS jQuery style selection of elements and scrape out data I didn't use the scrapey framework, but that also looks quite good That's built on top of beautiful soup and gives you some extra stuff. So maybe have a look at that Async it's really easy to do multiple scrapings at the same time And that is literally the only change you'll need to make your code probably If you want to find out more about how to do really simple Python async look at grants talk from last year That's got some really simple practical examples But of course if you are going to scrape multiple web pages at the same time you should check the rate limits for whatever service you're going to be hammering For me, I didn't end up using this because the service I was looking at said yeah, just please keep it on one thread Also, you'll want to make your web scraper restartable You probably want to do something like scrape 10 records Check that some of them are okay or check that they're all okay And that you haven't missed any data or haven't had any errors Then scrape 100 records and do the same process and incrementally look at more and more data So redirecting all your output to a CSV file isn't really going to work because either a you're going to be rewriting your entire CSV or If you're appending, how do you remove the records that you're trying to reproduce that were failed before? So a really easy solution to this I found was just using the SQLite 3 library that comes built into Python Allows you to have a single file SQL database And you can run SQL queries on Not only does this allow you to Easily find and delete subsets of the data that you want to redo but also makes searching and aggregating your results quite easy Finally next next finally you'll want to log absolutely everything if you can think about it You should log it you'll probably want to do some kind of try accept to catch every single exception because if your program is running overnight you don't want it to Fail an hour after you went to bed to wake up in the morning and nothing's been scraped So you do want it to keep going even in the case of failure, but if you're catching exception You should log that out somewhere and capture it And if a value can't be scraped for some reason you'll also want to log why it can't be scraped That's going to be saving you a lot of time. Also if you're going to be What was this? if you want to take the logs a collection of logs from your Front from your scraper and pipe them out to some file You can use the t-command to get that output into the console as it's going But you'll also want to use just standard out flush to make sure you keep all of the So you can see so python continually Outputs things as they happen and not doesn't buffer out your output So that you only see things in chunks every ten minutes Finally, this is a really nice little tip if you need to leave the house And you can't see your computer anymore, and you want to be able to know when something goes wrong or something goes right You can set up a free slack account for yourself Use the python slack client to post messages there install the slack client on your phone and you'll get free Mobile notifications coming from your python script. That's quite helpful as well And with that happy scraping. Thanks very much Thanks, Ben if Grant Pat and Simpson go over here next up We've got Samuel Bishop for those of you forgotten yesterday morning The title of his talk is failing is awesome just ignore the slides. They're not mine So yes failing is awesome. We all know this It's the foundation for success that we actually use our entire lives right through to the underlying principle behind many things We actually Aspire to I'm sure most of us aren't completely following TDD if we are trying but it's red green refactor We're supposed to fail first but Failing in public is not usually that fun But it's still just as useful and successful in the long term How many people in this room have had a talk at any conference about how big or small How well did your first time go? We all know the nerves and the experience can be quite surprising But it's interesting for anyone who hasn't done a talk to take some of the feedback I've gotten most of the people who saw the talk fail may be thinking that it was a disaster Especially considering it happened twice But it wasn't a disaster because I know that it was a good topic because almost everyone who gave me feedback Said that it was actually something that we interested in hearing about which means that it was something that while challenging Was wanted and so failing to deliver what was wanted isn't a failure to create something good It's a failure to deliver it Which is still something that can be improved upon which is the other half of failing Improving everyone likes to get better at doing things I've never met a human being who doesn't like getting better at doing things And so the failure is an opportunity to get much better at this I went from knowing nothing about C++ and Unreal Engine to knowing enough that I'm patching Chromium and looking at working on database clients It's and it's been an amazing two months of learning focused around something that Didn't fail it, you know a good way it failed in a way that left me fairly exposed in a public way, but I learned a lot Enough that I'm actually really excited to go and do things with what I learned so much so that I'm hoping to actually repeat the whole presentation successfully at some point because It all the work is being done. Just the fact that it didn't work the time it needed to work Doesn't mean that it was a complete disaster and it doesn't mean that I can't take good things from it so In summary, I guess anyone who has not yet done a presentation Stand up and try and do that and don't be afraid of the failure Do I riff in if you could come down? Next up is Grant Patton Simpson Okay, I'm gonna pronounce it asterisk comma Where you should use it and why? Okay Okay, basically readability counts which looks a bit ironic under the circumstances but the It's basically all that a story about an asterisk and actually a comma Yep, that's all right. Yep. That's all right. All right. Here we go Here we go This is really Python. We have a function. It's a simple function just plots x and y and a few things and now we use the function We just feed in the arguments perfectly readable without keywords in this case, and it's nice and lightweight. I quite like that Okay Then how about these What's what's happening here? We've got one two. That's okay But gray and black which way around is it is one foreground is one background is one something else altogether We can't tell and even worse we have a bullion on the second one What does false refer to what has been set to false? It's not so readable anymore Okay, so we want to change the function signature by adding the show grid parameter. We're making a library So there it is show grid is now false. Is it a fault? What will happen to our existing code? Which uses this function? Well something a bit mysterious to be honest We have the same arguments as before but what they do is different. So we'll be wondering why the pop out is not now showing So it's safer in my view to always use keywords for bullions unless it's blinding the obvious what it means Which is unusual. So here for example show pop up equals false We can now be sure whether there'll be a pop up or not assuming. There's not a bug so we digression on asterisk args and asterisk asterisk keyword keyword arguments quags Quags, okay Okay, so what they do as you may already know or probably already know is they conveniently gather up the arguments into variables so there we've got positional arguments and keyword arguments and If it was a real demonstration, you'd actually see can you read the gray at all? Yeah, okay. Okay, so that's what it does it gathers them up. So here's how it works I've got another example of the same function and this time. I actually Have it intermixed. So what happens is the asterisk only gathers up What hasn't already been explicitly included? So if we have a look at the results here Run that again We get x and y out there, but we get the remaining positional arguments only And and so on now introducing our hero asterisk comma So you'll notice that I've put this I think this is Python 3 only So if how many people are using Python 2 at the moment mainly? Sad 3 Python 3 Yeah Do you guys using 3 fuel smug half the time? Yeah, it's good isn't it much better. Okay, all right So what happens is the asterisk gathers up all the positional arguments and the implication is you can't have any more And so you only allow keyword arguments. So what happens here? Let's test it. So we have one two. Well, that simple case still works Doesn't require keywords for the x and y because our asterisk came after them But it will require keywords for all the other arguments. So for example if I do plot one two and gray I mean, what's gray? Was it the background the foreground? That's bad Anyway, it gives the error message that plot takes two positional arguments But three were given which is not entirely helpful, but it stops them in their tracks. They have to fix it as a client Coder so now we give the function what it actually wants and Mark a color equals red. We now know what red refers to and we get success And this is useful because we're forcing client coders of our library to write readable code And you don't you're not forcing them to use keywords for everything like x and y I think it's much smarter to allow the lightweight code But as a library developer, I can change the position of keyword arguments or stick new ones in and so on Later without breaking client code So that's really useful Okay, that's it Benoît Shabor Next up Duan Griffin will be generating text from ebnf with antler It's from line it's a fat file system Don't worry never mind about the slides So first off seeing as I don't have slides to show you who has heard of ebnf extended back is now a form Okay, so a few of you it's something we typically learn in computer science So if you haven't done a CS degree, then there's no reason you would learn it It's a way to formally specify a grammar for a language typically a programming language so you use it for specifying the the formal specification of a programming languages and you can take Something that's written in ebnf and you can use it to generate a lexer and a parser for reading code Written in those languages So there was an old tool called yak and lex which we used to use a lot back in the day And now there's a newer tool called antler Which I use and a lot of other people use for writing parses And it's very nice because you can write your language specification in a high level And you don't have to worry about Programming, you know if the next character is in this then we go and do that I was going to show you what that looked like, but I can't and so if you you just have to take my word for it That it's very nice and readable And a lot of languages come with an ebnf description including python specify the the structure of the language So if you can generate a parser and a lexer from an ebnf description It seems to me that it should be possible to take that Ebnf and to generate text that conforms to that grammar So I'm using antler. I can take an antler grammar file, which is effectively ebnf and some other antler stuff stuck in there I can generate the parser and the lexer from that and I can take them and use them to generate text hopefully I'd like to do this to Generate text for test cases. So I'm writing a parser and I want to generate weird test cases that I wouldn't come up with and basically fuzz my my parser so I'm writing a utility to do that. I kind of got it working a couple of days ago. So hence. I'm talking about it now I think it's going to be quite useful. I Need to extend it to generate more interesting test cases maybe Choosing which branch to take when when following the grammar rules based on some metric That's more interesting than random choice. So that's something we need to I need to look at and improve Potentially maybe trying to get something that's Not semantically valid, but semantically more interesting than just randomness by possibly allowing hooks for users to Make a smarter choice for things like identifier rules, which are high-level rules, which you may want to Not just completely blindly choose items for But yeah with all that hopefully we'll have something which will generate test case input And we can use that for stressing and then fuzz testing Parsers, so I think that will be useful for me Maybe it will be useful for some of you or some other people When the code is a bit tidier and a bit more bug free. Hopefully I'll put it out there And anybody who's interested come and talk to me about it later. Thank you Clinton Two more talks to go By the way someone wrote on the whiteboard talk request if a sync IO why twisted who's that Is that what you want to hear I See we're like the last one to put something on the whiteboard, so Yeah, so next up they'll be interesting to pack into five minutes Continuous integration and continuous deployment with GitLab by Benoit Chabot So quick question who uses CI CD at work who is using CI CD for their own small website Three, okay nearly nobody so Normally with this talk you'll be able to do it with for your small website So continuous integration and continuous delivery with GitLab. GitLab is like GitHub for those that doesn't know it it also includes Travis or Jenkins and Docker hub all in one Website Git is the starting point a commit will trigger a pipeline First stage will build the Docker container second stage will test third stage will tag the Docker container and fourth stage will release Web UI looks like this With all your stages when you have a failure you have a big red red cross if it's all good. It's all green So GitLab use Pipeline in YAML. So you just need to define a YAML file, which is a dot Dot GitLab dot CI dot YAML in your folder and it will start to run a pipeline for you You list your stages you define your variables with environment variables. You can get keep secrets on the GitLab side You script the stages and then you enjoy all your pipeline So for the build you have to log in Docker you'll rebuild an image And you push your image. It's super simple. It's like 15 lines of code to just create your own build with Docker For your tests, it's even Easier you create a SH script in your Docker container in my case It's running coverage manage dot py tests you fail if one test fail you expose your coverage reports With the artifacts inside GitLab That's how it will look for a pull of requests you can see that there is a coverage Percentage so you can even check with a regex. What is your coverage percentage 84% not too bad? Then the thing is if your tests are all okay What you want to do is really your image So your pool your latest image you'll tag it and then you'll push it to the internal repository You want to deploy a very simple you have a script on your web server on a subdomain that would accept the curl command that will then trigger a SH script on your small server and would Release your entire stack. So in my case with Docker, it's using Docker compose behind the scene Yeah, we had a talk yesterday. It's very close to what we saw yesterday and The curl command is just running docker compose Restarts How much is it going to cost you nothing the private git repo is free with GitLab? You have one gig of Docker images for free. You have a thousand minutes per month of build pipeline for free You can also install your own Like git lab runner on your own servers if you want to and then you have an unlimited amount of build pipelines and The interior code base is around 80 lines of YAML and 20 lines of bash scripts to be able to release all of this So now you have no excuse, you know what to do next weekend. Thank you, but no other Last lightning talk for the day will be by Clinton Roy who wants us all to come to Australia for Picon and for Linux dot com for a you That's pretty much the talk Hands up everyone who's from New Zealand Right, I did not put my hand up because I'm from Australia if you haven't worked it out yet So you should You should return the favor by coming to some of the conferences that I'm going to be helping That I'm going to be attending and helping running back in Australia The first is in a very few short weeks Linux confey you it's going to be at the end of January in Sydney There are two days of mini cons I'm helping to put on one of those mini cons and then it's three days of low-level Linux stuff so kernel and Low-level user library stuff It's all based around free software. I think this year is going to be like my 17th LCA that I've attended and I'm on part of the team that's putting in a bid to run it in a slightly nicer location down there Gold Coast in 2020 oops Later on in the year Picon Australia I'm not involved in running this one at all. I've been involved in running the last four in Australia And I am giving myself a break So we have three days of your regular conference with talks and tutorials And then we have two days of coding sprints where everyone who's come to the conference Gets together in a room and works on whatever project. They're interested in Both of these things were in Sydney I Can't I can't help that But I can move some of the conferences to the Gold Coast eventually So Thank you all for helping to put on this lovely conference this weekend And I hope you'll all join me at some point in Australia in the future. Thank you. So before