 Bon midi, Montréal-Piton. Good afternoon or lunchtime, Montreal-Python. We're just about to start in about two minutes. So you have some time to go get your lunch, go get some refreshments, and join us again. Don't forget, join us on Slack if you haven't seen the invite. It's in the text description, but I'm going to show it to you right here. It's mtlpy.org slash fr slash slack in, and we're in the meeting channel. So let's start in a minute. So hurry up, go get your breakfast, your dinner, and a refreshment, and join us on Slack. And then we'll start very soon. So, good afternoon, Montréal-Python. Welcome to this very special midi edition. So we have some European guests today. So we'll change a little bit of our... Welcome to this special noon edition. We've got some European guests today, and that's why we are changing our time. Just a few announcements, and then we're going to get started with some technical content. If I can get my slides to advance, obviously. Excellent. So first, join us on Slack. The URL is here. We are in the meeting channel. And when you're on Slack or on the YouTube comments, don't forget, we have a code of conduct at Montréal-Python, which is very simple. It basically says, be excellent between one and other. So join us on Slack. The URL is here. We are in the meeting channel. And we have a code of conduct at Montréal-Python. So it's very simple. And it says that we want everyone to be excellent at each other. If there's something that you see that doesn't seem like it's in line with the code of conduct, that doesn't seem excellent to you, find me on Slack or Duke, who is Duke on Slack, and we're going to resolve that right away. Our schedule today, if I can get my slides to advance, is we're going to have two technical presentations. First, right after I'm done rambling here, we're going to have Wukash who's going to tell us about optimizing a piece of Python code with Cython. And then Martin is going to give us a little bit of a recap of the translation sprint that we just did for the official Python documentation. And then Basilis is going to tell us about using OpenAPI to build a large-scale REST API. And then we're going to open our usual virtual happy hour. And I don't know if you'll be able to make it, because I know that the schedule is a little bit different than usual. But we're going to have some drinks and have a chat and be as technical or as un-technical as we want. So before I finish making these announcements, Wukash is going to tell us about optimizing with Cython. And Martin is going to give us a summary of the translation sprint that we just did for the official Python documentation. And then Basilis is going to tell us about using OpenAPI to build a large-scale REST API. And then we're going to return to our virtual 5.7, which is going to be a one-to-two, on GTC. So, another thing. We're going to have another week of exercise, next Tuesday at 8 p.m. We're going to have Montreal-Piton 87, Italic Retifusion 28. Join us on Facebook, LinkedIn, if you want more news. And then we have the calendar of events here where you have the URL. And other things. We're going to have an evening of practice. Again, next Tuesday, which is come and join us. And if you're starting with Python, we're going to run through some easy exercises. We're going to have a couple of different rooms for people with different skill sets. And basically, we're going to run through the exercises that you do them, ask us questions, or we do them. And you look at us doing it and you ask questions. It should be in French. It's going to be mostly in French. You can ask your questions in English. And if you want to see that happen in English, then we just need a couple more volunteers. So tell us in Slack if you'd like to volunteer. And we're going to be happy to add an English part to that evening of practice. Join us on Facebook, on Slack. All the URLs are in the YouTube video description. We have the calendar of events for the rest of the year here. And that being said, I just need to thank our partner, Evji and I, who's doing excellent consulting work mostly in the Montreal area for any Python development. And the Eddie... I cannot pronounce this name, but this excellent photographer who took this picture of that beautiful snake. Okay, that's enough rambling for me. And now I'm going to introduce Wukash Lange. Wukash is a Python core developer since a long time. He's going to tell us exactly how long in just a moment he is the release manager of Python 3.8 and 3.9. He is the author of Black. He also is a contributor to many tools in the Python typing ecosystem. And he's going to tell us about optimizing Python code with Cython. Wukash, the stage is yours. Hi, cool. Good to be here. Using the opportunity that our time zones don't match, like I allowed myself to fetch some orange juice. Yeah, it's 6 p.m. right here. I guess it's all legal. Today we're going to talk about optimizing stuff with Cython, maybe in a little offhand way. So starting with essentially introductions, like my name is Wukash Lange, most of how you can find me can be found through my website. Both the Polish character and without it should work. If not, let me know. Today our plan is to essentially talk about performance. We're going to first answer some big questions that most people have when they think like, hey, like Python, like can it be slow? Can it be fast? You know, like why? And then we're going to get to like how you can make Python fast actually today. And one of those ways is by using Cython. So we're going to introduce some basics of this usage with a particular example that I prepared and then finish with some practical tips as somebody who's using a Cython very often, you know, like I wish I known them a little sooner than actually I ended up. So starting with big questions, right? The biggest question is, hey, like, can we change Python to be faster so we don't have to do anything extra? It's just automatically fast. For example, see Python is interpreted, right? That is weird if you're used to see or rust, you know, like maybe we can do something else or maybe we should add a jet, you know, like that's weird if you're used to JavaScript and many other languages these days or maybe we should just get rid of reference counting and the gil, like that's weird if you are, you know, used to Java and would like to see this technology be faster without any action from the user. Well, the problem is, like, all of those are actually pretty hard to fix and the reason for them is as follows. Like, see Python is interpreted, but would it still be Python if you couldn't use dynamic features of it, like decorators, like get adder, like mocking, like conditional imports? Like, would it still feel like Python if you didn't have an interactive console? So the REPL, see Python also doesn't have a JIT, but JIT slows down startup in a very popular use case for Python or command line tools. It's structs and sets specific, so the implementation will get much more, well, hard to maintain and it isn't actually faster per se. Like, to make the JIT faster, you need to employ optimizations on the generated machine code. So it's essentially a long project and maybe we'll see progress here with Anthony Shaw's pigeon or the Cinder JIT open sourced by Instagram, but so far we don't have this built-in in to see Python. And see Python uses ref counting and the GIL. So what about that? Well, the GIL and reference counting provide implicit correctness guarantees. Most people don't think about them that way, but if your high-level code works today, it would probably stop once we would remove reference counting and the GIL. The memory structure, the memory life cycle of your objects would be now different and without the GIL, some guarantees that you have in presence of threads would no longer be held. So it's not even that easy as just removing the GIL. Larry Hastings did that and what he found out is that, well, it's very hard to also ensure that the resulting fine-grained locks scale with the available cores without hurting single-threaded performance, which is arguably the most popular use case for Python right now. So all of those are pretty much, you know, kind of theoretical, maybe aspirational, maybe we'll see them in the future, but how do we make Python code fast today? Well, the most obvious way is to find a particular place in your code base that is slow and just replace it with C. Hot code, right? Hot code is code that is either executed very, very often or does a particular computation very many times or does have a computation that is in any other way expensive. So you replace Python code with C. How? By writing a C extension to Python's C API so that now your program can import that C extension as a module just as if it was a regular Python piece of code. Well, but it is not actually that easy, right? Like if you want to have a C extension, you need to know C. Many operations that are super simple in Python can be actually tricky and potentially dangerous in C, like using dynamic data structures, performing operations on strings, dealing with numeric overflows, passing by reference and so on. You know, like this causes endless security view, view, narrow abilities, even in code by seasoned programmers. So, you know, like not something to be taken lightly. And you need to know C. Python C API. Like if you are a Python user, that doesn't necessarily mean you know how to create a Python module in C. It is actually an API that through C requires quite a bit of border plate. So, to actually make a module that can be imported, there's quite a bit of code that you need to prepare in an O. And even to create a Python class, right? In C, you will have to write much more code than you would think because it is not so high level as Python. So, then dealing with Python from C, which you will have to do because, you know, like at some point you're going to get Python objects from Python code, exposes you to explicit reference counting via InCref and Decref. And those are tricky because if you forget one or the other, you might either crash your interpreter or just cause a memory leak. So, now you have to think about those and like very carefully place them in pairs, which sometimes is very tricky if the life cycle of the objects is not trivial. So, that is not even the worst problem. The worst problem is that all of those things change, like both the compilers change and you have many of them across different platforms, but also the Python C API changes. There are no, like, radical changes these days, but there is definitely evolution. So, the fact that you know how to do something today might not necessarily mean like that particular way will be the best way to do it tomorrow. So, this is all problematic and then you cannot even reuse any of your C code and Python code. You'll have to essentially have double implementations. And this is essentially what C Python does itself. Like, there are many modules in the standard library that have both C implementation, which is the performant one, and a Python implementation, which is used for, I don't know, comparison of correctness and used by alternative implementations of Python-like PyPy. So, fortunately, all of those issues that you're seeing here are solved by Cyton. So, what is Cyton? In 2002, Greg Ewing released Pyrex, a Python-like domain specific language for creating C extensions, because obviously, even then, like almost 20 years back, it was, you know, clear that writing C for C extensions is not optimal. So, the Pyrex tool transpiled Pyx files to C files, and then you could compile using a C compiler and import from Python as regular Python modules. So, this was fantastic because it lets you still use your Python mode of thinking when creating a C extension. The code looked like Python for the most part. It was not exactly the same, but it handled things like InCref and Decref correctly every time automatically. You didn't have to think about it. You didn't even have to see those. That just happened for you. And dealt with boilerplate for creating modules and classes. You just saw the high-level code. So, the Pyrex language also added C-compatible type declarations. So, you could add them to function arguments and to variables so that the resulting C extension could use specialized code for faster computation. Mostly by omitting unnecessary layers of Python dynamicism and using native data types instead. Like, five years later, the maintainers of Sage, a mathematics system in Python released a fork of Pyrex called Scython. Its main goal was actually higher-level compatibility with Python so that it wasn't a Python-like language, but it really was a super set of Python. But on top of that, the goal of Sage team was also to provide a much wider array of optimizations and made Scython able to interact with C++ code, making it a great choice for wrapping C++ libraries. Now, Scython. To use Scython, you write some Python code. Then, you sprinkle it with C-type declarations. You run the Scython compiler to generate .c files, which then you compile them to a valid Python C extension. That is essentially the process of which you're using this. Most of the time, you're going to have a build system that automates most of those for you, so you're going to have those commands by hand, but that's the exact process in which it happens. So how can Scython compile all Python? It's a super set of Python, so how does it turn it to C? Well, when you run a regular Python code in the vanilla C Python interpreter, your modules first get compiled to PYC files holding Python interpreter byte code. C Python then takes the instruction by instruction inside its evaluation loop. Byte code instructions get translated on the fly into C calls. Scython does the same thing only at compile time. So it chooses the same C calls that the C Python interpreter would make and hard codes them in the produced C file, removing the eval loop overhead. So how can it be fast? It gives a modest speed increase, but the real advantage comes from doing less and you do less by telling the Scython compiler as much as possible about the low level details of your function. So the more types you specify, the faster it goes because it can create more optimized code and now we're going to actually see this in practice. Let me switch here. We have a prompt. Let me change to the directory I was supposed to be in. There's a project right here. Let's activate a virtual n. Okay, cool. And now that we have this set up, Python 395 all is good. Let's look at the project. So this is actually an audio synthesizer in Python. We're going to look at this maybe a little later, but for now let's actually create a pretty new file. Let's call it sd and implement standard deviation in straight Python. To not embarrass myself in front of you by just making typing fumbles, instead let me just paste the code and let's just talk about it. So this is actually pretty idiomatic Python for what a standard deviation is. It is not super fast, but it's also not terribly slow. We're using some of things in Python that make it pretty fast, like the sum built in and built in mathematical functions. So we don't really loop over things like in straight Python. We tell Python to do it for us in C. So if we actually run this, we can now kind of see and weigh how fast that is. To run it, maybe let's just create a test file for it. Test first. This is going to be a branch marketing test, but bear with me. For now, the only thing that we really need is from our fm to import that sd and create some test case. I prepared a thing here which essentially uses the array because we want to have some low level types. We're going to be using time it as well just to have some notion of how fast it goes. And by just running that, we should be able to see whether what we just produced is fast or not. So we can just run it by saying pytest-s is only so that we can actually see the result. So we have our function, Python standard deviation, returns in a second and a half. If we do it another time, it will probably be a little faster, but not by much. Kind of a cool. So now we might actually do the same in Scython. If we take this exact same code and just copy it, copy paste, cool. Let me rename it. To say sd2 but pyx. So now it's Scython. We're going to be able now to compile it. But we cannot just run our test because there is nothing that is compiled. So if we here added a test that just says import sd2 as well and here just copied the exact same thing only saying just take the function from sd2. Well, I bet this will actually not work, right? Because if we try to run it, like one of the tests is going to say, like, hey, this is not a Python module. Well, you need to build your pyx file into a C file that then is going to be compiled into an SO file. So how do we do this? Well, I'm using the poetry build system for us. So here the project Toml describes how a project in poetry looks like. Most of those things you probably know. However, one of the things that you might not is that in the build system right here we have to specify Scython as a thing that we're going to be using and we're going to have to specify a custom build script for us. This actually lives on GitHub. I'm going to give you the link after the talk. But here just to show you how it looks like, it's a bunch of setup tools sadness. So unfortunately, poetry at this point doesn't support this automatically. So in essence, what is happening is we have to find our pyx files and actually tell our setup tools to compile all of those pyx files using Scythonize right here. And then we copy the files over so that we can run tests. So having all of the setup we can actually run poetry build to build actually like a distribution. So for example, a wheel. But for us to actually test stuff right now what we want is to just poetry install. We just want those things to actually happen like locally, right? So having now pytest.s we can run it and we're going to see both results. Oh, yes site function. So Python function py standard deviation. It is already a little bit faster not by a long shot but it's already a little faster. Oh, it's only running the second one because I named them the same. Let me just look back at my okay, cool. Yes, so now we know that Scython works. It is actually compatible with Python. All of this is good, but it's not really that much faster than than regular Python. So what do we have to do to make it faster? Well, as I said, we we're going to have to go a little lower level. And to do this, let's re-create our function in a little less idiomatic Python. So if we did the same function in a C what is like Python style C, it would probably look something like this. Let me just paste it for you right below. Yeah, so like now we don't use any generator expressions. We don't use any nice like some built-ins we essentially do everything by hand. It's still pretty readable Python code and sometimes we write code like this, but it is not what you would call like Pythonic, right? Funnily enough, if we look at the C file that was produced from IPYX right now, we can actually see already of that one function, it's a couple of thousand lines of code. It's huge, right? One thing is that actually we can see the parts that generate a particular line of code. They go back and forth creating, you know, particular lines and what they do. For example, somewhere here we should have this particular line where we have the generator expression that is inside and you know, like there's a bunch of C code. It has to have those ugly names because the namespace in C is flat, so otherwise it would conflict with C code. So it doesn't look very nice, but it's good that we have it because we can look into it when stuff isn't fast and you expect it to be fast and see why it is still producing too much code. So that is obviously like a huge, huge produced library like at like 6,000 lines. So let's try to actually shrink it down. So to do this here, we already have our site on function and we expect it to be faster, right? Like let's try to actually test it right now. So we are going to do this as the Psy and we want Psy, standard deviation. Again it is sad, but you have to remember to compile. So let's just call poetry install always before Pytest. Okay, it's Psythonized as the file again and let's see what happens now. Oh, actually the Psython function is the slowest of the bunch. Like this is not what we came here for. Like what happened? Well, this is the Python compatible Psython. We don't have any optimizations here yet and this is less idiomatic than what we had here. Like this is all it goes to show that Python is in fact like pretty well optimized for what it is. Like it is super dynamic but it does a very good job at, you know, like providing building blocks that are fast enough for the common case. To fix our slowness here what you need to do is essentially tell Psython what that I is, what that N is, what that M is and let's just do it right now. So, for example, a built-in type in Python is PysysT which is essentially an integer. So we can say that this I is going to be an integer and the same with our N which is here this obviously is not an integer. So we're going to have to say it's a double same here, like that is also a seed of double and by this essentially, you know, kind of we use a little bit of type decorations and expect miracles to happen. Let's see if that's true. So running this now ok, shows us that there is some improvement compared to before it was 73 now it's like 54 but it's still like not even comparatively good to the pure Python version. So something is amiss and that's something is declaring what our argument is. Before it just thought it is some pi object right and see that it actually produces the same kind of code that we would have in Python. So now saying that actually it is an array like we can already have some optimization right there and what we can also do now with our superpowers is to say let's import a better square square root function right from see and use this one instead of this one and now like with just a little bit of code like it is probably a little verbose taste but let's see if that's faster already. Ok it's not really why not well we have to look at what it is doing and the most expensive parts of that our function is going through the array like right here and right here and indexing this array if we don't know what's in it is inherently expensive. So what we can do now with our superpowers is to essentially tell it like we know that this array holds shorts which are like 16 bit integers right and using this and actually painting it's right here so we are not using the Python level array but a C level array see it's a pointer a scary pointer let's see what happens right now well see this is what we wanted like now it's crazy fast right like and if we actually look at the produced code like you're gonna see that our raw yes piece of data right here is actually pretty pretty low level we're not at this line yet now we are so look we are literally taking this field from a C struct and using it in T2 later on which allows us to essentially just do a very simple for loop in C instead of unpacking by objects all the time that made it faster so yeah that was my short example for today I wish more time like for you to just actually go through much more kind of cooler and bigger examples but some pro tips for you so Symem is an extension it's actually like a library but it's meant to be used with Scython that also lets you do the same as we have for InCrefDecref that you don't have to think about when should I actually decrease the reference but for allocations right allocs and freeze is another pair from hell in C you have to remember to always allocate and free but only free once you know so Symem does this for you you only have to create a pool and allocate your thing they're gonna allocate automatically you know like just like for reference counting in Python you can now do the same for C code and it produces actually performance C code which is pretty cool and another thing that I wanted to say is the Sython documentation is actually very good so whenever you actually don't quite know how you would do something in Sython like you'll probably find your answer there but unfortunately the truth is like if you want to for example deal with C++ in Sython you're gonna have to know some C++ so I started from this perspective of somebody who only wants to optimize a small bit of code but then you grow and grow and kind of learn maybe C from a little bit of a safer space like I know actual organizations who write Sython instead of C so even though they could just be writing like C in C they write it in Sython because it's safer it's just as fast so yeah like that was essentially what I have now the only thing that I maybe still have a minute or two to show you before taking questions is why is this weird repo called FM well it's on a frequency modulating synthesizer like written in pure Python so like you actually have a full synthesizer that has phase modulators and those actually do mathematical operations like counting essentially like math that produces real-time audio from user input so yeah like if you allow me to actually show you a bit of this oh yeah for example here's the envelope generator for it and this is essentially a class that can be very easily also translated into C right and by this what we can do is we can achieve much greater much greater polyphony so we can do more things at once we can play more sounds at the same time without breaking Python without breaking a sweat so like I don't really have time to go through translating that class right now I can only show you how the end result of it looks like so let me go and do just that right now PYX that's from a more mature version of this repo but in any case you can see that there are functions like saturation that is all covered in C-Defs just so that it's faster our envelope that I showed you the Python version of it now we unfortunately cannot make it data class anymore so we have to create our init by hand and do a bunch of things like in a different notation those are no longer Python type annotations those are C type declarations but they're still pretty readable you know like I don't know about you but like I don't really find myself like really horrified by what I'm saying it's still pretty readable and like for example like this main logic of what it means to advance an envelope that is still readable just as regular Python like you don't really have those declarations clouding the algorithms like yes there are declarations they're all the way up here but when you get to the logic all of this actually looks almost the same the only difference is that it's 80 times faster than a pure Python version and that without NumPy without any other like domain specific thing like this is totally generic this is totally generic which allows us to use it for anything you can use it to interact with other C libraries you can use it for you know kind of doing algorithms to essentially like you know whatever you need for example HDB a database also written in Python managed to be super performant thanks to Scython and its optimizations and making all the places that count profiled and replaced with pyx files instead of py so that FM synthesizer to just leave you with something concrete like if you really wanted to hear it which if you were not going to have survived with me for a while like let me just play you a song Black Narcissist I guess it's topical so that you can hear what the thing is about like you know is it actually that interesting is it not like let's hear it yeah the amazing thing is like that thing that you're hearing is produced by real-time Python it's like you know something that I always thought it is impossible to achieve but yeah it is so if you if you'd like to see more about that particular thing like I gave a talk just on that topic at PyConUS this year so as soon as their videos are up you're going to be able to see how actually you produce the synthesizer like in the meantime I'm very happy to take questions you can find me on all sorts of places online thanks for your attention thank you Wukash I'm going to introduce Duke who's going to monitor our questions right now thank you Wukash for your presentation and for the music pretty cool yeah I have a small question from myself who who don't know where to start I know that you already show the documentation of Cyton but is there the link for the tutorial for someone who want to start yes there is a very basic tutorial right here which essentially gets through all of those things that you need to do it still talks about setup UI and you know like a bunch of things that are not relevant to today's Python but you know it is very good because it kind of shows you a bunch of things like in very increasing detail so like they for example go with the Fibonacci function and also you know can start optimizing and then seeing like you know like how much better it goes so I essentially recommend it but the best way that I actually learned because like you know the tutorial is good like you know to have a feel with it is to find existing Python code on github and look at that you know for example I want to deal with some C library like just look at for example HDB it is open source you can look how they interact with say Postgres and what not because it is already there it is probably easier to just see how somebody did it than try to figure it out with an empty file ok thank you let me take a look if we have another question ok with another question here so I guess I will go with my personal question again but I will continue with that I mean and do you have like I know that you already showed that we can synthesize music and all other things but in the live live example do you use like for example connected with Postgres or something like that let me just find something for you for example what I have for HDB is like actually big codebase that uses it for low level networking right so like that is different from what we have here like let me let me just find a typical file so I can show you some example yeah ok cool oh for example a connection to Postgres how does that sound ok so that is from the HDB codebase it is a pyx file mostly actually looks just a regular python but it has a bunch of like you know defines that are C defines and it has a bunch of C depth which are you know like typed objects mostly looks the same there are only some key places in which you need to actually have types and that is for stuff like let me just find some of those for example a class that is a C class so that is not a class that you're going to be able to access from it is a class that we need to interact with C and to use with the rest of Cython like there's a lot of Cython files in the HDB codebase like one particular example here might be for example like here like we reach out to objects that are C objects like directly right like there as you can see it is it doesn't actually look very scary in practice when you use it in real production code because you don't you know kind of optimize everything to the last second right like we kind of did that right in our example right here just to show you that hey like it can actually go from like you know a second and a half to like point zero point you know like two two hundreds of a second right but in real code you only optimize what is on the hot path right so for example parsing parsing obviously is very important right like so here you're going to have more C definitions it is more important to optimize that but the rest of the code again looks like Python because you only need those definitions to make it fast the rest you only have to deal with logic okay cool okay thank you okay I have another I have a question just come in we might have to move on I'm so sorry guys because we have so many interesting things to talk about today but Fort hopefully Wukasz is going to be able to join us for the happy hour after this talk or if not maybe you can answer some of these questions on Slack or we'll figure out a way for all of these questions to find answers thank you so much Wukasz and thank you and our next presentation is going to be Martin who's going to tell us about that translation sprint that we did last two weeks for the last two weeks the word is yours Martin thank you thank you so yes I'm going to give you a little overview of the translation sprint in French of the official Python documentation which was organized by Montréal-Piton it lasted from 16 to 31 May so what do we translate in fact we translate the standard library we choose each one a module multiple modules and we translate them and why do we do that it's to have a doc in French of course it's also to give back to the community because personally I use often the community tools and it makes me feel good to be able to give back a little for all the work that was done so that's cool and also to learn the modules that we translate and also a module that interests you in particular and you really want to know it's a really cool way to learn it because by translating you're guaranteed to have read it at least once and I hope you understood and translate in French so it's really cool as I learned I had a module that I didn't even know that I translated called Context Vars so it was really interesting and also to learn I didn't know what the PO files were I didn't know how it worked today 18-20 July now the structure of the files and the tools like PoEdit to edit the PO files for example and PoRap, PoSpell, PaPo these are tools to check the PO files that we write so there's automation and also if you're new in Git or GitHub it's interesting to know how to develop in these environments but that's a good way to learn so let's think about the results so we managed to translate several modules so we made some that were almost already finished so Get Text, Horizontal, Context Libre and Cincaio we finished them we also made some modules to translate and we also made some that were almost not translated at all and we almost completed them they were as well completed DocTest, PoEdit Context Vars, what's new which translated to 95% IP Address we translated it to 62% and Platform now translated to 100% so the total of translations in French we translated from 54% to 59.79% 5% it can seem not too much but there's so much Doc that it's a big chunk anyway so that's it and the people who were in the sprint I won't present them all but they are there I would like to thank Edith and Yannick who showed us how it worked and that's it we showed them how to translate DocTest in French it's not bad at all it's very short but that's it so I'll go back to the ball thank you Martin Martin, if people are fascinated by this beautiful translation work how can you start to simplify? we have a channel in Slac in the Slac de Montréal-Piton so I didn't follow the exact name of the channel but it's so that would be a good place to start asking questions there's also there's a guide for that so we can send the link later in the comments yes, we'll do that that would be the starting point great I think I might have missed if you announced it but in October we'll do another print of the translation because there will be the version 3.10 that will be released somewhere in October and we would like it to be released with the beautiful documentation in French from day one that will be announced somewhere in the summer excellent thank you very much Martin ok, our next presenter next presenter will be Vasylis I hope I said that's right I practiced it a few times excellent and he's the lead developer at Mr. Io and he's going to tell us about using OpenAPI to build a large-scale REST API Vasylis I don't see your slides so you can just share your screen and then I will add you and pass you the control ok alright Bonsoir on Real, ComonSava hi, thank you a lot for having me here today so let me quickly introduce myself my name is Vasylis and I'm a DevOps and backend engineer at Mr. Io today I will talk about how MIST builds its own RESTful API and CLI for multi-clad management and we'll focus on the development workflow behind it so here's our agenda for today I will first talk about what is MIST to give you a better context on what we do then we'll see our previous development workflow and we'll compare it against our new one and we'll see why we switched to our demo where we will add the new feature to MIST and we'll see this in action after that we'll see the benefits that we gained from this change and at the end we'll see what are the next steps for this workflow so let me answer the first question so what is MIST? MIST is an open source multi-clad management platform it supports over 20 infrastructure providers including public clouds like Azure, AWS and Google Cloud it also supports private clouds like OpenStack and VMware it also integrates with containers, hypervisors and even bare metals MIST connects with all these overnative APIs and on the other side exposes a multi-cloud API so MIST's RESTful API is written in Python by at least 95% so you can think of MIST as a single pane of glass where you can easily manage your infrastructure so when you are dealing with multi-cloud infrastructure here many questions arise here are some frequent ones what resource do I have and where are they? how can I control my team members access to these resources or can I automate some self-service workflows and at the end how can I optimize usage and use costs? These are all questions that MIST helps you answer so MIST consists of many components and microservices that all work together to help our users manage their multi-cloud infrastructure so it's quite obvious by now that MIST integrates with a lot of providers and offers a lot of functionality for its users but all this comes at a cost so every time that we need to update our API integration with the provider or every time that we need to add a new feature we have to touch a lot of MIST components so for example if we want to add a new feature we would first have to implement it into our RESTful API then we would have to update to add this functionality in our UI then we would have to write tests for this new functionality then we would have to update the docs the CLI and the SDKs so this overhead that we had every time that we needed to add a new feature even a small one it really slowed us down and for this reason some of our components like the docs the CLI and the SDKs were usually a bit out of date so when we felt that our API needed the breath of fresh air we took our time to engineer a new development workflow with it so here's a high level overview of this workflow so when we want to update an API endpoint or edit an existing one we just create a new branch with the change that we want and we push it in a new branch into our OpenAppSpec repo so by the time we push this new branch this in turn triggers some GitLab biplans which utilize code generation tools and generate code and docs which after the generation is finished are pushed to the relevant components so let's take a look at this workflow in more detail let's say that we want to add a specific feature to MIST let's say that we want to be able to remove block storage volumes with MIST what we do then first we describe the necessary endpoints in the OpenAppSpec 3.0 so what is the OpenAppSpec it's essentially a language agnostic way to describe a restful API so it's essentially a collection of YAML or JSON files and we also plan to upgrade the OpenAppSpec to version 3.1 when the tooling catches up so after we have described our new endpoints we now can push them to our OpenAppSpec repo into a new branch so then this new branch will trigger the biplans as we said earlier it will generate the codes and docs and will push it to the relevant components which in our case are the API the docs, the CLI and the SDKs so the next step of this workflow is to implement the controllers and add the business logic we have with these biplans we have generated the server but of course the business logic is missing so after we have done that now it's time to review all our requests and merge them to our code base so for this workflow we use several tools we use the OpenAppSpec to describe our restful API we extensively use Datatocs AppIgen tools to generate our server code the SDKs and the docs AppIgen tools is essentially a Python wrapper on top of OpenAppEgenerator and does that job so for our Python server code we use Connection which is a Python framework which maps endpoints to controllers we also use the OpenApp CLI generator to generate our own CLI and we also all the biplans that we have mentioned are done with GitLab and at the end our interactive docs are entered with the help of the AppI doc component so now we'll go on to the demo in this demo we'll add the future we talked about earlier we will be able we want to be able to delete block storage volumes with mist so as we said earlier the first thing that we have to do is to go to our OpenAppSpec repo so here on our left we see several files these are our AppI representation in the OpenAppSpec so this is all thanks to AppIgen tools which allows us to split our AppI representation into multiple files so now since we want to add a new feature for volumes we'll go to the relevant file which is the volumes one so here we want to add the new endpoint to make things much simpler I will just paste it I have already in my clipboard collapse this I'm sorry yeah just had a small problem over here so now it's fixed so now we have described our delete endpoint it's a very simple endpoint we have to state a description an operation ID which is a single parameter which is required in path parameter which is this over here and we also declare our responses which are simple raster responses and after that we add some configuration options for our security for the security of this endpoint okay so now we have created the description of our endpoint it's now time to create a new branch it will push the changes let's call this branch delete volume we'll commit the code okay we can now push this code one more step okay so now that we have pushed this new branch it will trigger several GitLab pipelines so I may have to log in I think yeah there's a different time okay let me go again to the link right we see that we have a merge request for our Mistape spec so we can go to the new branch and we'll see that it has one pipeline running so we see that this pipeline is responsible for generating the code and if we see above we can see that it has some Git tips, it has generated the code that we need now we have access to the docs in this link these are interactive docs for some reason I think it takes a bit of time to load them okay because I have zoomed in too much okay so now we can code the volumes we'll see that the endpoint that we have created is right here and I have to mention that these docs are interactive now as you can see in this URL over here we have also generated our CLI and we can take this URL we'll go to the missed commutation that we have running right now and we will download it and we'll place it in our computer okay so now we have generated the CLI the docs without having to write anything so if we try our CLI we have one volume called Python Montreal so let's try to delete this volume okay we'll see that essentially it does nothing because we haven't implemented the controller yet it's only the service tab so I will remove this line and we'll do our magic here and we'll remove the comments the commented out code and now if I start the API code and see that we still have our volume and if we try to delete it right now let's wait it now got deleted we no longer have any volumes so that was the demo I will now switch over to the slides I'm sorry okay let's continue from here oh no have a demo some demo slides in case something went wrong okay so now the benefits of this your workflow are quite obvious first we don't have to worry about writing repetitive boilerplate code the open appy spec tooling takes care of it and of course it's now way easier to manage multiple components at once since we don't have to write a lot of code we only have to write code that does the actual work the service tabs, the SDKs the CLI and the docs are handled by the open appy spec tooling and of course by having our app representation in a simply repo it makes it easier to reason about which endpoints we have and how they work and of course it also makes collaboration easier when you want to add a new feature to our RESTful API so now we have several next steps that we want to do for this development workflow at first we want to generate both Python 2.0 SDKs in our pipelines which we almost have ready and then our next big step is to upgrade from open appy spec 3.0 to 3.1 so the open appy spec 3.1 uses the latest JSON schema and this is important for us because we have already developed a UI component called UI forms so UI forms parses this JSON schema and it generates forms based on it so the goal here is that we have already done some work to describe our API in the open appy spec and by doing some small adjustment to our open appy spec we will be able with the same spec to generate UI forms without having to write any custom UI code for these forms which is very beneficial for us because we don't have to spend a lot of time writing UI and yeah that was my presentation for today for more information about MIST you can always check out our website and you can always download MIST community edition from our github page and of course you can always reach us out on social media like LinkedIn and Twitter and of course this is a work in progress so any comments that you may have or any suggestions are very welcome so I'm now ready to answer any questions that you may have thank you Vasilis and I will pass it on to Duke to moderate the questions yeah thank you Vasilis it's pretty cool and I see that it's a bit fast but I see that you combine it with digital notion if I am correct I guess it's going to be compatible with the other cloud yeah we support a lot of both private and public clouds so yeah you can use Google Cloud, AWS, Azure most of the big public clouds and smaller ones like the digital ocean and line out okay yeah thank you let me see if I have a question from Slack or from YouTube channel oops and for now how many members is working in the team we are not that many to be honest we are less than 10 how many less than 10 okay less than 10 so the project is pretty new but I mean when you have a new project open source I hope everybody will find it interesting and enjoy the communities you know I don't see any more questions I think we are going to after hour are you still with us yeah okay yeah excellent thank you Vasiles thank you Duke and as Duke mentioned it's the time for the virtual happy hour and for those who don't know what that is it's pretty simple we just all join a big video chat and then grab our beverage of choice and then we have friendly conversations so as Duke just mentioned it's the time of our virtual 5-7 which is going to be 1-2 so we are going to join a video conference together so we can share our stories of pitting and having a refreshing refreshment together before we do that I just want to mention two small things it's going to be very quick thank you to FGNR our partner who is hosting our video conference platform and don't forget to fill our satisfaction survey the link is in the description of the video on youtube there are just three small minutes you are going to fill those six questions which help us a lot to know if we should always make meetings tell them if you like to know our European friends who join us we are going to open the virtual happy hour thank you FGNR to host our video conferencing chat and don't forget the satisfaction survey the link is in the youtube description it's going to take you three minutes to answer six easy questions that's going to help us to know if we should always do our meetings over lunch if you like having our European guests and things like that thank you so much everyone and oh I forgot to mention the link of that happy hour virtual happy hour so it is in the youtube description but it is right here as well by mtl-meat.fgnr.ca slash mp86 we'll see you there and then we'll see each other again next June I forgot the date but it's in the description bye everyone