 So, good morning everyone. So, a little bit about me. I am currently working in Red Hat in the Fedora engineering team as Fedora cloud engineer. I am also an ambassador and contributor to Fedora project for almost over 10 years now. I am one of the core developers of C Python that is the Python programming language and I try to write blog posts at kusheldast.in. So, that is my formal work and whenever I get free time I try to become Superman. You can see this. I used to practice at home. Nowadays I am being able to fly outside also, but this talk is not about that part, but this talk is more about what I do at work that is Fedora cloud. But even before I go into that side I want to, I have few questions actually for my audience. How many of you are actually using Python in your day to day or somewhere in your work part? Can you please raise the hand? Okay. So, around 50 percent I guess. So, okay. Good. So, what we do at when I mean, what when I say Fedora cloud. So, it is actually not any public cloud running somewhere, but it is more about the image which runs on different cloud infrastructure. That is you can run this image on OpenStack or OpenNebula or CloudStack or we also generate the AMIs for AWS. So, if you look at from that perspective, the major two customers are obviously right now AWS and OpenStack. And for OpenStack, what we create is normal standard QCao2 images which can be used in most of the other cloud infrastructure. And we also put the same image into others places like DigitalOcean where you can boot up a Fedora. So, as you know, Fedora project is a community project. So, it is a group of community people along with engineers and people from Red Hat who work on this project. And part of the project is actually testing whatever we release, correct? So, we don't want to give you any broken software, any broken opening system which you cannot use. So, and most of our test cases in general are actual best commands. So, let's say I actually started working full-time on Fedora cloud in 2014. So, part of my job was to make sure that whatever we release is good. So, work along with the QA team who are like we have a great QA team, but they are always overloaded with work. So, what we do is we ask volunteers to download the image or boot up an instance in AWS and log in manually using SSH and then execute some commands and check the output. I have this actually. So, this is one of the commands. So, one of our goal is or one of the test case what we say is no service should fail at the time of boot up. Like whatever the default things are should always start properly. So, the idea is you log in, you execute the command and then you check the output like nothing should have been failed. And this is just one example. So, there are things like AC Linux would be enforcing. We want that to be working properly. Like different packages must have been installed, things like that. But this I'm trying to tell you what we do as test cases. And this is how all the images get tested manually by people. And that's the exact point where the failure part comes in. So, the biggest problem we got in was about not tested enough. Because end of the day, it is human who are testing the images. There was no automated system at that moment. And like I think it was Fedora 22 around that time just before the release date, we made some kind of mistake which made the image unusable in certain ways. And the release engineering team was really, really angry because it happened just the night before. So, they had to work really hard to make sure that actual release goes out is working. And in another case, after a few months, we also have a project called Fedora Atomic project. So, Fedora Atomic is now the primary output of the cloud group, which is an operating system meant for containers, a bare minimal operating system with atomic states. So, that's meant for containers. But in between, we did a, I think we did a nightly release, which never had Docker. So, and you cannot install a package inside Atomic. It has to come directly from the, as a part of the operating system, but that was our mistake in a nightly build. So, we tried in various ways to fix this. And me being a very lazy guy, that's true. I actually, even though if you see me jumping around, I'm a pretty lazy guy in front of a computer. So, my point was, it was difficult for me to every time go boot up a, try to upload an image to OpenStack or try to download the image locally and then go through various steps to make sure that a cloud image can boot on my laptop without having an actual cloud and then test it. That was my condition. And if we look into the next question about the projects. So, the way I was trying to solve my problem, I think many of you do the same thing. If I just asked right now, how many of you have any kind of pet project lying around on your laptop or in a corner of your server, which you use to help yourself in a certain part of your job? Do you have any such pet projects? Yes, no? So, we all know how this project starts, correct? It starts the open source way, scratch your own itch. So, you have a problem, you solve yourself. That's how most of the project starts. But then the next stage happens. I'm skipping. This is one such example. So, it can happen that you will find someone on internet that who is trying to solve the same problem in a wrong way. Or it might happen that you will find one of your friend or co-worker has the exact same problem and they need some kind of solution. So, what do you do? You generally help them out, correct? We all of us do. And we go like, hey, I wrote this best script or I wrote this Python or anything. I wrote this application. Why don't we use it? And we try to give them this simple project. So, what was before that? What was your baby? Suddenly, there are no two people using it. And instead of two people, it can become 100 people. It can be even bigger. So, that project can sometimes become something which is there in all of our work laptop or all of our server's places. But at the same time, think after six months, think after one year, what happens? How many of you actually remember what you wrote as a program script or email after six months down the line? I mean, or one year? Do you remember everything you did a year ago? Yes, no? Anyone? I also don't. So, and then, like as system administrator or DevOps, when we manage various these systems, and as I think most of us will agree that all of us have some scripts which helps us to automate some part of our work. How do you think what will happen like when your company or job will require some new guy to come in and actually fill in your soup? Will they be able to understand what you're doing in that script and the stuff? But at the same time, our main part or work is to make sure that we don't get into those failures at the same time. And for that, we are actually we want something which is there in all our computers. In my case, it was all our cloud systems, the operating system which will work on those cloud. And there are mainly those two main parts. One is base, the standard base. Not all the commands may be available to any particular installation, let's say default installation. And the other thing was Python. So I would actually tell you my talks may primary agenda today is actually to convince you to convert those base scripts into something more readable usable Python scripts. So yes, I'm telling you what is my rest of my talk is about. That's it. So this is the actually end result of my whole project, whatever I was doing. So it's a very small Python tool, like around 1000 lines of Python code, I guess in total, which can boot up a cloud image on a laptop, or it can actually now do vagrant images also, or it can boot up something in the AWS and execute a series of test cases, which can be which are actually basically any Linux command. And at the end of the thing, it will report generate a report and it will destroy that instance. So we use this right now in our production Fedora environment, as a part of the Fedora QA process, that we test every each and every build we do, every each and every cloud build. So I'm giving you this link right here, because if you want to check it out, it's there is completely open source licensed code, and you can actually install and execute it on your laptop. It will not destroy your laptop. But coming back to my problem, like we had those bass grapes and obviously, as you know, there are not many people who can come up in the beginning and help us with those tests. So now this is actually a line from a poem called Zen of Python. How many of you know about it? Zen of Python. So for the people who never saw this before, actually going to, it's not used enough. So in any Python, if you start Python interpreter, and just type these, actually import these in this case. So this is the Zen of Python. This is one way of saying what we love about Python, what we want to do in Python as a programming language. And you will find there are like thousands of explanations of this particular poem on internet. So one of the major part was readability counts. So think like this. For my work, what I had to do was parse some sort of configuration file. In my case, we started with JSON, and then parse it to a place where we can actually use it inside our code or script. And all of you, at least most of you are here bass experts. If nothing else, you know how to work around with the cell. And you can also very easily imagine how easy that is to do it in bass. And at the same time, how easy it is for a newcomer to come and actually understand what you do. I know it can be done very easily for many people, not for me though. But how easy it will become for the next fresh out of the college student or a CIS admin who never worked that much. So this is one of the example configuration what I have for my project. So in this case, what I say is it's with a name, it's a VM and the image which one to boot up and some RAM and the username. So I actually had a JSON parsing example from GitHub, but I don't want to show it here because it's too large, like how to parse that thing over. Instead, I'm just going to show what we do in Python. So we already, in the part of the standard library, we have a JSON module. So it provides two different, is this, thanks. So that provides two different methods. One is to load from a directly from file, or the other option is to actually load it from a string, like if you already read that information. And that will provide you a very nice, in our case it was a dictionary of the data we want to read. So the next thing, so we just had a tool which can solve our problem for us, but that's not the end of the thing. As I mentioned, one of our biggest problem was we never had enough people. In most cases, it's not a computer, human become the major issue. So it was same with us that we never had enough people to do the testing for us, other than the automatism pass or write new test cases. So what we did next was that we moved out from only those simple commands, but we moved into something you may think a little bit complex. We used something called Python unit test cases. So that command becomes something like this. If you're coming from any programming language background, you can actually understand most of this code. And what we basically check is that no units, loaded units, should fail. That's why zero loaded units listed as failed. But what is the benefit? What is the benefit coming out of that bad script into a Python unit test case like this? So for answer, I have a question to you again. How many of you actually have code running on your production which can do release blocking? This is a release blocking thing for us, by the way, which is written by second year college students in India? Anyone? Or you can think about if you don't have college students that way, you can think about interns. Do you have interns writing release blocking code? We do. Not interns, but volunteers, contributors, like one kid is from Bangalore, Farhan, one girl, Trishna sees from second year again, computer science student from Calcutta. So they wrote most of these test cases now, like over the last one year. And the benefit is that they already know Python as a programming language, which they learned, which they use in other places also. So what they figured out, like, oh, we can easily jump in and start contributing to this. And they can already understand, like, what is the effect out of it, that they can block such a big project like Fedora to be released. So before that, there was almost no one else coming in with new test cases. And almost in the last one year, I never had to write a single one, because we just had to document like, hey, we do you want to test this? And someone just came up, I want to write this, give me a day. So by nightfall, we already had the test case written, then we discussed about it. And all these test cases, by the way, again, are on GitHub with open source license. So later on, if you want to check it out, one of the major feature of the language, what we call is batteries included. That means the standard language, the standard language comes with a standard library. Python standard library has way too many different modules for you the batteries. So it has modules to do SMTP to send out emails. It has modules to, like, say, if you do not have enough networking related commands installed, it can give you a socket player, which can you can use. It has a simple HTTP server module, which I can use to create an HTTP server out of a couple of lines of code. I just can name all of them. So there are many. One easy way to look what you have. This is the standard Python 3.x, I compiled it locally. So it will show only the things which are there in the standard library, I hope. Like you can see FTP leave, stats module, URL leave, unit test cases, mock objects. So there are, is this visible? So there are various modules available right now for you in Python standard library, which is again there in all of your servers, which are running. So another thing, the language using a complete programming language rather than only basketting will help you out is about errors. So this is again a line from our poem that errors should never pass silently. So Python has a programming language provides you different various ways to look into the errors in a granular level. So I'm going to open up a code from the same project. This is again part of the, like I skipped few things. But again, this is a part of the code coming from the tuner. I guess, yeah, it works. So we can see like we can have exact error case conditions. And most of the Python modules, most of the Python projects you'll work with, they will have this kind of granularity, where you can see you can go get into the exact error statement. Even though many times it happens that the developers chose not to go through these details and they prefer to just say file or directory not found. That happens too. So again, this is one slide I actually added. I completely missed about it. But yesterday by Joe told me like, hey, we should talk about that part also debugging. You know how easy or difficult it is to debug in various programming languages. So if you want to full blown debugger any part of your code, you can just add the actually this one line, which is basically two statements import the Python debugger and just start the trace. And then you can go through the whole program. Most of the times I personally just use print statements. But if you want, you can actually go through your whole project. Again, a little bit different thing, Python decorators. This is a feature of the language which provides us many useful solutions. So one of the things we have about our test cases is that our test cases can become today from a getting test that is that means that the same test can block a release from to a certain extent where it may not block a release. It still will fail, but it may not block a release or a test case can become something only for let's say AWS instances instead of all clouds. So we have or that's only one that's not only one single test case, it can become more than one. So think about a condition where you have to move suddenly like 100 test cases from one state to another state. Do you want to add if else loops or if else statements to all of the source code? So this is just one such example. You remember I said that we had missing Docker few days back. So this is where we are testing if the Docker is installed or not. But again, this is not supposed to run on a bare minimal base image, the cloud base image we have because that's for everyone. Not everyone will want Docker on the image. So this has to run only on atomic images. So what we do is we add something on the top of the function with the address sign, where we say that if it is atomic, then you go ahead or skip it. So I'm going to a little bit more details about this part of the code base coming in as functions as first class objects. So in Python, because Python has a dynamic language, you can pass around functions just like a normal data. So another maybe stupid example. So I just have a function which will check if the number is greater than 10 and divisible by two. And if only that, then it will increment or multiply it with two double or else it will return minus one. And we have a function called map, which takes input. The first input is a function. And the next is a list of inputs which will apply to the function. And at the end you can see it did only for the values which are two in that case. And this is how we write a decorator. So this is the example of a very basic decorator. So where we are taking a function as an argument and returning another function as a result, the last line return wrapper. So what happens that if I go next, we have a normal standard function def hello. And just on top of it, I wrote the name of our decorator, adrid my underscore decorator. So even though we call hello, it's still automatically added those extra code, the extra layer on top of the function call. And it did, even after my function exists, that after the call. So you can add these features to, say, add some kind of special error checks to, let's say, 10 of your source code functions. Another major usage of Python in the system level is about being a glue language. It's a great glue language. That means you don't have to reimplement or write actual programs. But instead of that, what we do is, you use existing applications, existing scripts and code, and use Python to glue them all together. That's where one of the major feature for Python on system comes in. And that's where most of the operating system use it, use it as a glue language. I'm going to give an example again in the same problem. So I wrote to me, I thought, oh, we are done like we can automatically now test, we have people who are coming in contributing new test cases. So we had one final call. It's an IRC meeting actually on one of the Fedora channels. We go for discussion. And two weeks before production, the release engineering suddenly mentioned that, oh, we, by the way, you did something wrong in one way, that we also do Vagrant, and we have to test that too. We cannot just say that, oh, we'll test only half of it, we'll not test the rest of it. So as the part of the Fedora Cloud Group, we release Vagrant boxes, both for Fedora base image and the atomic image. So, yeah, it was just exactly two weeks before going into production. We are in stage environment, we just have to move it to production. And so our answer was like, okay, because we knew at that moment that we don't have to develop a new Vagrant. What we have to do is just use the Vagrant and test it out. So what we had code something like using this particular function call, subprocess.popen. This helps us to call any other random command within our opening system. If you remember before, there was a system call. It's basically a wrapper on top of this, but we wrote and our Vagrant code, the code base which actually does Vagrant, it does something like this. You may laugh at it. So it adds the image as a box and we already write down the Vagrant file and then you can see up the Vagrant. It actually calls the Vagrant up command with the provider because we test both on Livert and on VirtualBox. And again, we check the output to see if everything is up and running, if not destroyed. So if you remember yesterday's morning talk about crash-only software, so our tool actually assumes, I never actually never knew about crash-only software before, but in general our tool assumes that we crashed at the last time. So it tries to clean up everything and then come into a state to do so. And that happens a lot of time with Vagrant because it works really great over VirtualBox. But with Livert, sometimes, sometimes if it crashes for some other reason, it may give you different outcome. But again, using Python and using the calling the system commands, just like this, helped us to get the Vagrant feature up by next day morning. So it only took us a couple of hours to add the whole Vagrant plugged into the system. And we as you can see, we never wrote anything super new. We just all the steps one by one, the way you generally do in a basket also. But using a proper programming language, that gives us the option like, you know, the new people again, the same thing anyone can understand. And everyone even like in our community went through the point, they read the source code and they said like, okay, we can put this on production because if anything happens almost all of us can go and fix it. So yeah, this is the failure I was talking about. I actually have a blog post about it. So Livert, the storage was failing because we forcefully removed the image file. So again, that's our mistake as a programmer, but we did that at the first. So we had to clean that up. As I mentioned, from the community, what they're looking from us is that happy users, correct, happy cloud users. So they don't want us only testing for like one particular cloud. So today they may want us to test for two cloud providers. Tomorrow they want us for the third one or the fourth one or the fifth one. It can grow. So I'm going to talk a little bit about a project which we use inside our code base. That is called libcloud. It's Apache mod. It's a project from Apache. And it's already can give you options, same code style options for 30 different providers. That means you can fire up cloud instances over 30 different cloud providers, including AWS, OpenStack, Eucalyptus and many more. And when I actually used it, I used it, I just went ahead with this because I knew there are enough better documentation and I can use it with both OpenStack and AWS. I never even noticed that it provides 30 different providers. So now I know that when in future we'll be able to automatically test on any other cloud platform, we're going to use the same project and we don't have to increase our technical depth. This is again another point about C library usage. So Python as a standard, as a programming language standard, we make sure that our users should be able to use already written C applications, C libraries. So we have various ways. You can use something like CFI, CFFI or you can write your own extension by raw C by hand. And that's one of the major reason like our C extension code is is huge. There are so many different projects for which we have a Python extension written using C. And again, many people actually says Python is slow, correct? Are there anyone here who will say that? Yes, no? In some cases, yes, correct? So this again, this becomes easy way if you feel like, oh, I don't should not use Python, but instead of that, you can use that other library which is written in C or C++ and get it working with your Python code. So from the programmer's point of view, you will still be using Python, but you are using a library which is written in C or C plus plus. Now, the best part of the language, the language comes with a great community. All of you almost, most of you are here from Bangalore and how many of you heard about Bankpipers? Okay, only very few. For the rest of you, Bankpipers is, I think, one of the oldest and the most active Python community in India. We have monthly meetings here in Bangalore and some special workshops also. And it's exactly same in the global level. So if you go to Haas Python channel on IRC and ask any question, it's almost certain that you will get an answer. You can talk to almost most of the upstream developers over the mailing list and places. We have a core mentorship list where if you want to contribute to the language, you can go and do that. That way, you will increase your own scope into the whole language area, how it is improving for the future. Even all the discussion, what happens about how the features will change of the language that is there in public. And we have something called sprints on in the Python, Python US. There, the last four days of the conference, we all sit together in few rooms and we write new part of the language and different other projects. So if you, even if you are a complete newbie, you can come to us in the Python sprints or even one of the bankpipers meetup and you can start doing Python from there. Whatever I learned about Python or about programming in general, I give the full credit to community all the time. And for Python, I mostly learned a lot of things from JS, who is sitting there silently. So and that happened thanks to the community only. So at the end, I'm going to talk about two things, two particular Python projects, which are actually of great help for sysadmins. One of them is fabric. Anyone here used fabric before? Oh, so a lot of you. I just copy pasted one example from the documentation. Like this is not a difficult function, correct? Yes, no, anything? Okay, someone said yes, I guess there. Okay, I'm sorry. But yeah, so you have to learn at least this difficult thing. So and using the fab command, you can now after this, you can execute it over thousands of different hosts at the same time. So so if you have to do something over a lot of servers, a lot of different places, you can actually reduce to eat to a Python code, something like this and then execute and do some particular job. Let it be a git pull or deploy something else. Just thinking if I go more examples, no. So the other Python project, how many of you here already know what is this? So somebody said that, say it loud, shout it out. Ansible, correct? Another great example of a Python project, which is being used all over the world now to manage our systems. And I don't know if you saw it or not, but you can actually extend Ansible by writing your own Python modules. The source code is available. There are lots of great docs also. So if you want to use Ansible something super fancy, you can do that. Actually part of our project, my project that to me, particular case was being able to use Ansible so that we can set up complex systems, something like Kubernetes. As a part of the federal project, we actually publish our infrastructure details also. So we have a Fedora Ansible repo. I'll give you the link later on over Twitter to my slides. So the repo part is actually clickable. So you can go or you can search Fedora Ansible repo. So we publish all our roles, playbooks, every details on internet so that you can actually reuse parts of them. Documentation, how many of you think documentation is important? So I try to talk about documentation at least once in most of my talks because I find it like one of the most critical thing of our work. So we use something called RST restructure text to document most of the Python projects. So if you go to any of those docs in readdadocs.org, all of them are written in RST. And the presentation you are actually looking at right now. I also wrote it using RST. There's a Python 3 tool called hovercraft. So I can actually and this is the same way to write doc string in every Python function. So when you write some Python code, you also want to mention correct what it does. Or when you do help on up some Python code, it shows you those doc strings and you can write the same restructure text there. So that's the end of my presentation. And I think I'll open for some questions now. I should have at least eight, seven, eight minutes. That's my Twitter handle. If you want to have and want to know anything about Fedora or how we are doing in the cloud, or if you want to come and volunteer or you want to use the same technologies of your place, you can come and ask us over the Fedora booth. We'll just point you to the right link. Hello. Yeah. Yeah. Not really a question. I just want to stress a very important thing that we have to do is we all are involved in operations in some capacity or other. The fact that we we are an open source community, I'm talking about Fedora, we are an open source community and we have our entire Ansible modules, playbooks, everything is open. And that says a lot about what we are trying to achieve here in terms of community relations that you can actually browse our entire infrastructure code. We have that sort of faith in community and that sort of, you know, inviting to please help us, please, you know, help us in contribution. Yeah. I mean, he is also part of the Fedora infrastructure community. So yeah, other than the passwords and few other things, I mean, few other variables, everything else is open there for you. So you can take that and start using it in your production system. And yes, we use that thing only. I mean, right now, I think all of our production and staging environment is on Ansible and all of the things are available in the repo. So you can trust our like code to be used in your production also. Like any other question, anything related to Python or any open source communities or stuff? Or like if you want to start and nothing really about that. So it's about the fabric. OK, so I teach Python and then when I'm teaching Python to develop, I can tell everyone to go and use fabric. But I always get this trouble. OK, so I teach Python 3 and when I go to fabric, fabric is not a ruler for Python 3. So you end up do everything else in Python 3 and just for fabric, you have to go back to Python 2. Yeah, so. So what we try to say that if you're someone is learning Python right now, you should learn from Python 3 itself because that's the future of the language and moving from Python 3 to 2 is very easy compared to if you know only Python 2 and if you want to move forward to Python 3. So yes, there are some projects which are still on Python 2 only and sometimes it's difficult. But the basic of the language is most of the time is same. But the test cases which I showed those are all Python 3, by the way, because from a default Fedora installation only has Python 3, not Python 2. The other thing is what in slides I saw you said Python 2.8. What's that? Oh, you got the job. Yeah, I know. I was meant to be caught. I just put it up just to see if people are actually noticing what I am doing on the screen. So yes, there will be no Python 2.8. Thanks for noticing that. So Python 2.7 is the last major release of the Python 2.x. We'll be doing bug fixing. But all the major releases, all the new features will be always on Python 2. 3.x series that is 3.6 upcoming. So there will be no Python 2.8. Thanks for noticing that. Any other question? By the way, if you want a Python resource, like there are so many, we have great tutorial documentation. I also have an open book on the internet called Python for you and me. You can just Google about it. It's again on restutter text on the readerdocs.org. Yes, please. Yeah. Yeah, I can see you now. Yeah, so question on Ansible. So is this a like a computer for Chef and Puppet, you know, for deploying on cloud or is it more on your own infrastructure? It's, you can say it's almost similar thing and it's not only for cloud, you can use it anywhere in the same way. And it's written again on Python and has its own features. So I'm not an Ansible expert to, you know, tell you which is better or why it is better. But yeah, it is exactly the same. Yeah, they are in the same space. Yes. So you can just type Ansible and these days, like too many people are using it. So you'll find a solution. Anything else? Yes, back. Yeah, I'd like to add to the Ansible Chef thing. I don't think Ansible is like a direct computer to Chef or Puppet. We use both of them together. You know, you can use Chef and Puppet for your config management and you can use Ansible for, you know, having like run these parallel commands, have some sort of a dynamic inventory for your network devices and stuff, right? So they can coexist together, provided you decide the use cases for both of them. Yes, and Ansible right now can actually do network configuration also. So it's like in Fedora, we use Ansible for everything, like in case. So depending on your use case, of course. Any other question? Hello. Hello. Done. Oh, yeah, okay. Can you throw some light on the project atomic and how different it is from CoreOS where people think like you know, every component of the kernel is a Docker container. Rather than there are two approaches. Where you have a microkernel or you go with containers. Everything is. Yeah, so I have someone better than me here to answer. So can we get the mic to hear? So here's from the project atomic team. Yeah, hello. So I'm out of the right. Yeah, you can stand up actually. OK, so project atomic is in the same space as CoreOS, obviously. But there's a lot of difference in terms of if you look granularity of the project. So project atomic actually is a framework. It is not operating system. But using the framework, we can create the atomic host from the existing packages of Fedora, CentOS and Red Hat Interpen Linux or any RPM based operating systems. Right. So what is the benefit? The benefit is the trust chain we have already that we actually build a trust chain RPM. So RPMs are basically secure, signed through GPEG and all the stuff. So we create an operating system for containers out of all these RPMs. And the way the name suggests atomic operating systems is basically set less. So when you start the operating system, the whole operating system stays in one state. When you upgrade it, it goes to another state. And if this stuff doesn't work for you, we can revert back to the previous state. It's like it for operating systems. So the idea is when you run containers, right. So the operating system is like the catcher. You should not worry about the operating system much. The attention should be on inside for the application to the containers. So how do we create an operating system that will reduce the work of operations guys or sisters? I mean, so that they spend less time maintaining the operating systems. So I mean, there's the basic thing. There's so many other things around. But they're competitors. Yeah. Any other questions or else I can go down? Yeah, no, we're going to wrap the questions for now. Okay. So I'll be there in the federal booth. So if you have any questions throughout the day, come and meet me there. Thank you.