 Okay. Good morning. Welcome. Thank you for coming. Please receive to engineers Terry Carras and Doug Halman. Thank you. Good morning. So we're here to talk a little bit about two overlapping Python communities, the OpenStack community and the broader Python community. My name is Doug Halman. Yeah. And I came to OpenStack from the Python community. So I've been doing Python programming since the mid-90s at some point. I'm working now for Red Hat, specifically on OpenStack. I'm on the OpenStack technical committee, which is the governing body for the contributors. And I'm a fellow for the Python Software Foundation as well. Okay. So I... It's working. And my name is Terry Carras. I work for the OpenStack Foundation. I'm the chair of the OpenStack technical committee. I'm also a Python Software Foundation fellow. And you can find me on Twitter on Freenote. And so this... I come from the... from an OpenStack perspective going to the Python community. And Doug is taking the other way around coming from the Python community and reaching out to OpenStack. And I want to start with a short intro on what OpenStack is. Who knows what OpenStack is? Lucky. I don't know what OpenStack is. So for the few of you that don't know what OpenStack is, it's an open-source infrastructure software that you can deploy on top of a secure data center to make it programmable infrastructure. So it's usually deployed by decent-sized companies that happen to own a data center and want to provide resources for their internal development efforts or other types of companies that want to run a public cloud, basically, and offer resources for rent to the wider public. It's not exactly something you would deploy in your garage on a Sunday. And I think that's where the corporate image comes from because the users are large organizations and not really small things that you would deploy as an individual. And this translated into a negative corporate image for the project itself, which is a bit unfair because as we'll see, it's actually one of the most community-led projects out there. In particular, we have a fairly independent technical governance. We elect our project technical leads. We also elect our oversight board. The technical committee is the technical oversight board for the OpenStack project. And the election is based on one contributor equal to one vote. It's not one contribution. It's not one company. It's really if you contribute to OpenStack, you get one vote to elect the leadership of the project that will decide where the project goes and if something gets done or not done. We also have a set of principles which we call the four opens. The first one is pretty obvious, open source. Everyone does it those days. But the others are slightly less obvious. A lot of people do open source. A lot of people do also open development. By open development, we mean that everything that's happening in OpenStack development is done publicly in the open on publicly accessible websites. You don't have to be... You don't have to sign up. You don't have to be an existing contributor to see how the development is done. We do all our code reviews, all our specs, all our RC channels, all our mailing lists are fully public and open. And we also run only open source tools in our project infrastructure. We don't rely on GitHub. We have even the configuration for this project infrastructure is also open source and committed in our repositories, in our Git repositories. So everything is transparent. We also do open design. The design of OpenStack is not done by a small group in a corner and then the rest of the community is there to just do extra work. Everything from base design, new project is done within the community. There is no secret group that does things in a corner. And finally, we also do open community. By that we mean that anyone can raise two leadership positions in OpenStack. You don't have to be a sponsor of the OpenStack Foundation or coming from a large company that's happened to be there at the beginning to be an important member of our community. So you can get elected to project technical leadership positions. You can also participate to the project as equals without any attention paid to who actually pays for your work or where you're coming from. And we're pretty adamant that we need to have a legal playing field. It means that anyone coming in the project should have equal chances to contribute to the project. So we reject projects that would benefit one company or be where contributors coming from one organization would have a specific advantage. For example, if you work on... if a vendor wants to propose a project in OpenStack that requires a specific hardware like a Cisco switch that costs tens of thousands of dollars, then that project could not be an OpenStack project because basically people that have access to that hardware have an advantage against someone that comes from nowhere. We don't have appointed positions. We don't have dictator for life. We believe that change in leadership are good and that's where that Open Community thing comes from. And finally, to close this introduction I wanted to raise a bit about the Big Ten thing. We used to define OpenStack technically and so when a new sub-project would be proposed we would try to assess if that fits our mental picture of what we think OpenStack is. And that was pretty useless exercise. We ended up having those discussions. That's this project. Is it OpenStack? Is it something that uses OpenStack? Is it something... And that was not really useful. And at that point we realized that what binds us all together is those principles, the way we do development, the fact that we are a single community more than a very defined scope on what exactly OpenStack is and where it stops. And so we moved from defining if a project is OpenStack to judging the community that was producing it. Are they following our principles? Are they coming from... Are they part of the OpenStack community? And if they are part of the OpenStack community then they are an OpenStack project. The reason why we should say this is OpenStack is not OpenStack. So we moved from a very product oriented definition of what an OpenStack project is to a very community centric definition of what an OpenStack project is. And that has served us quite well until now. So the meat of the matter what Python brings to OpenStack first. So I was there at the beginning when we picked languages and I think there are reasons six years ago where we picked Python originally for OpenStack. The major reason was that it was pretty important for us at the beginning that we would get packaged into distributions. I had the past experience. I worked for Canonical before and we packaged Java software for our cloud offering back then. And Java is really poorly supported in distros while Python was extremely well supported still extremely well supported. It was the major language for development within Canonical at the time. And so picking Python gave us a step up with distribution integration. We didn't have to fight against the distros that they would support us. It's also extremely easy to read which means there are not so many ways you can write stuff. And so there is no endless bike shedding on exactly the format of your patches, et cetera, et cetera. It's easy to learn. My 10-year-old daughter is learning Python right now and she is quite successful at it. So it's really easy. It's easy to patch because whenever you run Python code you have the source code. It's not as if when you run a Java stack you end up with Jaws and you have to download the source code separately but anyone running OpenStack can actually look at the code that is running, see potentially a mistake about somewhere and propose a patch. It's really easy. And finally the last thing was ops familiarity. The target user, direct users for OpenStack are the operators that would deploy it. And so picking a language that was something that they were familiar with was really helping us as well. I'm fine. Thank you. Their operators are using Python internally already and so that also gave us a step up with reaching out to our users and closing the feedback group was a lot easier. Thanks to that. So Teri mentioned that we chose Python six years ago. We've had discussions recently about changing but we have found a lot of reasons why Python is still successful for us. As an integration language, it's very common to, especially in our project, to need to interface with lots of different devices or other services. Most of the OpenStack services are actually built on top of other tools that have already existed or written in other languages and Python makes it really easy to communicate with all of those things. Part of the reason for that is the large ecosystem of libraries that's available for Python to use. So we have a new source index and that just continues to grow every day. So we find that we don't have to write all of the source ourselves in order to build things. It continues to be easy to read. So six years in, we have a lot of old code and it's very important for new contributors to be able to go and look at that old code and then six years in, we continue to have a lot of new contributors joining our project. And so having a language that is still easy to learn continues to be an important reason to use it. And we'll talk a little bit more about that in a bit. And then the community itself, the inclusiveness gives us good examples for subgroups. So PyLadies is a great example. We have Ladies of OpenStack folks that go to conferences and join meetups and that sort of thing, which are all very welcoming to brand new members. The example that the Python community sets for welcoming new members is really important to us. So we've talked a little bit about why we think Python is great for OpenStack and I wanted to talk also a little bit about why OpenStack I think is a good thing for the Python community. There are three main areas for that, funding contributors and quality. A lot of us don't like talking about money. As we saw this morning in the keynote though, it is important. We're all trying to make a living doing this and OpenStack gives us a good opportunity to be paid to do open source work. Often the company's funding network have goals in mind but there is still quite a bit of freedom in what you can do and the way the community itself is organized lets you take advantage of the fact that there's a demand for that. OpenStack companies when they're looking for new employees they're typically looking for one of three different key areas, either some previous experience doing cloud or infrastructure automation of some sort with DevOps tools or existing cloud systems. They also look for specific technologies like maybe you're a storage expert or you have real expertise in networking or virtualization or they look for Python expertise because Python is a key component of OpenStack. So this is some job information. This is trends for job listings containing keywords Python, OpenStack the combination of the two storage networking and virtualization and this is US data because that's what I had available but OpenStack is an international project the trends to work in a similar way and this is showing growth in those job listings over time things like storage networking and virtualization and actually Python by itself are all those lines along the bottom that are relatively flat, some of them even go down a little bit over time but if you look for OpenStack jobs that's the orange line in the middle that is trending up and if you look for OpenStack and Python together that's the green line at the top so there's definitely an increased demand and that's 2100% growth over that time so that growth in demand has the expected impact on salaries OpenStack salaries are 17% higher than standard Python salaries just standard Python jobs the salaries are trending up as well that I believe has a lot to do with the demand from the increased use of Python within the OpenStack community other than salary there are a lot of other benefits to being an OpenStack contributor you are valued as a member of the community for your contributions that you give in your name by the employer that's providing funding for you your influence is based on your reputation and so that means that you have job mobility so we have a lot of members of the community have moved jobs while staying within the same place in the community but they're working for a different employer I myself am on my third OpenStack job right now so it lets you find a company where you feel like a good size and a good role for you if you are interested in product based companies there are lots of those if you're interested in service based companies there are lots of those large enterprises, startups you name it there's an opportunity there and it lets you find a fit for you both within the community and within an employer that influx of Python developers, folks who are joining OpenStack to work on OpenStack means that they're also learning Python they're also joining the Python community so we have Tiri mentioned Java earlier we have a lot of folks who are joining our community coming from more enterprise situations where they are used to programming in Java they're learning Python which means there's a lot of opportunities to sell training to those people they are also many of them for the first time learning about open source applications then they don't know about open source communities so they're learning about that and getting excited about their jobs again and joining the Python community coming to conferences like this one going to meetups in their local or regional area joining the Python software foundation and then contributing in other ways so outside of OpenStack contributing to other Python based projects so another way OpenStack contributes to Python is quality and by quality I don't mean the quality of OpenStack software I mean quality insurance that you can get for Python libraries out of OpenStack using them so this is a number and I don't expect you to guess what that means it's actually the number of test jobs that OpenStack runs on a given typical work day that means we run a test for three, four or five seconds and those tests actually take around one hour to run and they deploy OpenStack software, they run functional tests et cetera et cetera so what that means we are using new Python libraries as they go out we automatically integrate them by the time they reach our PyPy mirror then we'll start using them that means if Python library new release breaks everyone it will break us really really fast and so there are lots of times where we detect problems before anyone else the other interest in having that kind of number of tests is that we also catch corner cases and risk conditions if something is working 99 but 99% of the time we will hit it like several times a day and so it will break us too so we can contribute bug reports especially on new releases that break us really really fast and that's not just talking about it I have a few examples here PyMemcache 130 if you're in the room it's okay that was released on a Friday July 24 last year it moved exceptions around in an incompatible way so if you were catching exceptions then you get screwed the bug was spotted on Sunday and then we reported that and the new release was released on the same day that's a classic release regression and we caught it really fast QZ2.1 had deadlock when acquiring a lock that was already acquired which is kind of a corner case but should have been in their functional test suite the bug was reported the next day the day after the release and they fixed it just the day after and finally NETADDR0716 failed on validating a previously valid MAC address corner case again and we reported it the next day and we got it fixed so the amount of testing we're running within OpenStack helps getting regressions and we did it really fast on Python libraries because we were using a lot of them so Terry gave a couple of different examples of dependencies that we have we actually have over 300 dependencies in our global requirements list for the different sub projects of OpenStack to use and so part of what we want to do not just reporting bugs but we also want to contribute code to those code bases as well so we directly create libraries that we hope that other people will find reusable as well so these are just a few examples of recent releases that we've had different libraries from command line frameworks to testing libraries to packaging tools if you saw the previous talk about Haskell where he was talking a lot about making packaging more declarative the PBR library is something you might want to check out but I'm going to stop and set up tools from declarative packaging so in addition to our own libraries we've built some tools that you don't necessarily need to be a Python programmer to find value in so Jenkins Job Builder is probably the most popular one of these it lets you use EML files and templating syntax to deploy copies of jobs with slightly different parameters so it's very common to see that set up with a commit hook so that when you check in a new YAML file your Jenkins server automatically updates itself which makes managing your test systems really easy and there's a bunch of other testing related tools and release notes tools and things like that that are usable no matter what programming language you're writing your code in and then we do contribute upstream as much as possible so I did just in preparation for this doc I did a really quick survey of our community and I had 20 or 30 different packages that were suggested as things that people contribute to mostly there was evidence for each of these it ranged from bug reports to bug fixes to new features to documentation all sorts of things like that and the range of all of those tools is pretty broad given the number of dependencies that we have on occasion we've even taken over libraries where the original maintainer said that they were no longer interested or able to support it but we still needed it as a dependency until we could migrate on to either a newer tool or replace it entirely in some other way and so these are a couple of examples that I was able to find where we'd done that each of these tools is a key component in one part of OpenStack or most of them are actually used in multiple components and so it was important to us that they continued to receive bug fixes and in a couple of cases security fixes were important so we've talked about things that we're already doing to work together we also wanted to highlight a few things that we should do to improve the symbiosis between the two projects and Teri's going to start by talking a little bit about things the Python community can do so so one of the things you can adopt from OpenStack is to do actually some testing before you commit in OpenStack we can't really commit anything that doesn't pass the test that's a good pattern a lot of people do it but some others don't because they have commit rights they can just push things and yeah you will test around release time but then you get around to do it framework where you can only commit things that pass tests and then whenever you have a regression you just add the missing test so that you don't get caught by the same issue next time I think it's a really great pattern there are lots of regressions in libraries we produce libraries we also regress quite often but that really helps getting them under control the next tip is to have a library author you should really use semantic versioning so the idea is to signal the type of change that's been happening since the last release if you only fix bugs you just increase the patch level if you change behavior or add a new feature but did not break backward compatibility you just increase minor and if you break backward compatibility that happens then you increase major it is simple but a lot of people don't follow that and it really helps the users of your libraries assess the risk that they are exposing themselves to by upgrading and the amount of care that they need to apply before they use your release sometimes it's counter-intuitive because you will increase the major version number for a security bug fix because the only way for security issue is to break backward compatibility so you end up with a big bump but from a downstream user perspective of your libraries it's really helpful to know that if you upgrade to that version of the library something might get broken versus there is no chance something gets broken because it's only a patch patch bump next one once it's out it's out if you put a broken release out to delete it from PyPy and pretend it never existed it may look like the easy solution and to a very embarrassing situation it's it happens to everyone and we we do quite a lot of them as well monthly but you really can't recall the message once it's out it's out the cat is out of the bag someone on the internet will have done noted it some PyPy mirror or cache will have a copy of it and so if you remove it and pretend it never existed then you've made a embarrassing situation embarrassing and confusing and so that's not really rewriting history is not giving you anything you should just roll forward really is a new version that fix it and in your project documentation history just tag that version as unusable and that's fine it's the only way to really make progress in a constructive way and so last piece of advice don't release on a Friday and by Friday don't really mean Friday just don't release before you go into an extended vacation or last hour of the day and go to sleep because for the previous due to the previous points you will release regressions that happens to everyone and so you kind of need to be around to fix it and get early feedback from people so wait until Monday or do it in the middle of the week when you still have a few days ahead of you rather than do it as the last thing before you disappear but there's also plenty of things up and so I can do it to make part and better right so the first of those is contribute even more upstream than we do now both in terms of creating new things that are more usable than we have but also contributing to all of those 300 plus dependent disease that we have and not just fixing bugs but also writing new features so we frequently write little wrappers around libraries to do extra things that we feel like we need to do a lot of this could really be contributed to the library itself to make either wrapper APIs that are part of the native library or just extend the existing API in some way to make the library more accessible and to give those things to everyone we've become a little bit better about creating reusable components from OpenStack a lot of them are really only reusable within other OpenStack projects because of the nature of some of their dependencies they sort of expect to be running inside of an OpenStack framework in some way but I had a list of six or so libraries that we've created recently and there are actually a few others that we have that aren't well documented yet that I could have mentioned on that slide so we need to do a better job of finding places where our code is not so special and snowflakey that we can't give it to other people and then highlight the work that we've already done so I mentioned that we have some things that are not well documented we should fix that work on the documentation we do have a bunch of documentation for our libraries up on OpenStack.org so if you are interested in taking a look at what we have all of the documentation is hosted there with our other documentation and then we should also be doing some work to share the tools and experience that we have by coming to conferences and speaking more often so frequently it's pretty common to have folks come and talk about OpenStack itself but there are a lot of aspects of the work and there's not specific to OpenStack and experience is working on large open source projects at scale deploying things at scale automation, things like that that we could talk about using OpenStack as an example but not requiring that you care about OpenStack in particular to make use of that stuff so if you are interested in learning a little bit more about OpenStack there are a couple of other members of the community who are presenting this week and if you want to add it we actually have the OpenStack Summit every six months and it's going global everywhere it comes to Europe every two years and this year in October it's in Barcelona and Spain so if you're local want to join it's a great experience to be around and it's actually free to attend if you contribute one patch to OpenStack so it might be interesting doing that and if you have any trouble like or anything we also have a travel support program to send people that don't necessarily are paid to work on OpenStack and are interested in learning more so that they could attend the events so if you're interested you can also join us in Barcelona because it won't get in Spain until at least two or four years from now so take the opportunity if you can and that concludes the talk I guess thank you for joining us we have plenty of time for questions Thank you to the organisers for providing those awesome orange towels that match our slide theme A question? Hi When do you think OpenStack is going to move over to Python 3? Great question we have been working on that for a couple of years actually I've been involved in OpenStack about four years now and we've been working on it at least that long the process has been a little slow so the first phase well, first of all I should explain that it is given the size of the contributor base and the amount of code we try not to do things big bang so we try not to say we're going to move everything to Python 3 right now so we phased it into a couple of different steps the first phase was to make sure that we were only using dependencies upstream that supported Python 3 and I think Eventlet was one that it took a little while to get a Python 3 version that was stable for us there were a couple of others that that was sort of the big one where we knew we really didn't even need to bother with anything else until that was ready and then the next phase was to make sure all of our internal code our internal libraries that we manage were ported we have most of the projects running unit tests under Python 3 now so they've run under Python 2 and Python 3 and we're at the step where we can start running our functional tests and so projects are starting to set up their deployments for the continuous integration tests to run under Python 3 so one thing we do is that once you get the unit test to pass with Python 3 we run tests for Python 2 and Python 3 and we really reintroduce a regression on that and so we support both at some point we'll probably only support Python 3 and go forward but the idea is to support both first and like Doc said we already have libraries and I think like 80% of the is Victor here I think we're down to Nova and Swift yeah and that's the work is coordinated by one person Victor I don't have a question Hi Doree Hi Do so there's been a lot of discussion lately in the community about adopting Go because it's faster for some stuff how is the community facing this problem in terms of feeding back the limitations of performance kind of code to the community in particular with the async kind of code it's a bit of a complex discussion to have because on one hand you want to give developers the tool that they need to do their job on the others like I said on the slide about the big tent we basically define OpenStack as being a single community and so it's really difficult to fragment it around language lines even if Go is not the worst language we could work with coming from Python it still will create a connection between the part of the community that only does Python and the part of the community that only does Go and people that do both and so it's a bit of a struggle to try to like enable developers to do it without ruining our community the way we've been looking into that is every time someone has a need for Go in OpenStack it's usually that they have some kind of a specific problem that we kind of need to look into on a recent one people assume that Go will be faster for async programming and very fast network outputs and the recent progress on AI, HTTP and other libraries in Python shows that it's not necessarily true and sometimes it's education sometimes people just want to jump to the new Chinese thing that everyone else is using so we need to kind of debunk that and if people still have a reasonable use case for Go we need to find a way for them to use Go they could also develop a Go library in a corner and call it from Python code they could mean there are other ways to allow developers to use Go without fragmenting the community and currently we're basically looking into how we can unblock developers without without ruining the community and as far as upstream we're going to work Larry Hastings is giving a talk about the Gill Act to me that he's working on to remove the Gill and I think that was one of the big performance issues that one particular project was seeing he's giving a talk on that later this week so I'm looking forward to that and we do have a couple of members of our community who are Python core developers and they're well aware of Victor Stinner especially is well aware of the async issues and we're going to go back upstream that way but we're sort of at the point where we have a particular project where it is very clear that there are low level issues dealing with hardware devices for their performance and as Terry said there may be architectural changes to that particular application that are needed in order to split it up but everything that we do to push back upstream as far as issues and performance I've filed a couple of bugs myself I know some other members of the community have too Okay thank you Last question for you Along the same line of the last question you briefly mentioned that you were thinking about replacing Python somewhere in the past what were the languages you were thinking about as contestants to we're not thinking about replacing it wholesale there are a couple of projects that have run into particular performance issues at very very low level dealing with storage devices and those projects are experimenting with writing things in go I'm not sure whether they evaluated any other languages We also had a number of projects that wanted to have a project written in Java but it was more because they had loads of corporate developers that were Java developers and they wanted to reuse them for open stack without training them to a new language so there was no technical reason it was more like a business reason so the two things at one point there was Erlang at the very beginning people wanted to write a message queue and they said that Erlang would have a very specific use case that they wanted to use a very specific language for and our role as the technical community for open stacks not necessarily to give direction for every project to follow it's more to make sure that we still have this coherent community and follow those principles that we've set and so we the language is a thorny issue because of this potential for fragmentation versus the need for technical development and that is the big thing right now people want to rewrite things in Go so we need to have them pose and look at the problem really deeply before we can actually say well Go is the only solution for you, Python can actually be made to work for you probably and you should look into every specific issue in detail before you switch. And we do use plenty of tools and it's not that Python, everything in open stack is written in Python but most of the application stack is written in Python. Okay, thank you for this interesting conference and thank you very much to you for coming. Thank you. Have a nice day.