 My son is going to be talking about it. Yeah, there is a snake in the bird house. Nice title. So it's all yours. Remember, for everyone, you can ask questions at the end. And you can type the questions in the Q&A option in soon. Go. Thank you. Good luck. Awesome. Thank you very much. Hello, everyone. My name is Mason Egger. And today I will be presented to you. There's a snake in the birdhouse building a Python culture at Verbo. So quick introduction. I am currently a developer advocate at DigitalOcean. But before this, I was a site reliability engineer at Verbo, which used to be called HomeAway, which is kind of where the theme of the title of this talk comes from is HomeAway's old logo used to be a birdhouse. So that was where it came from. And then they changed the name and ruined the title of my talk. But there's nothing I can do about that. Still kind of works. When I first started at Verbo, there was no official support for Python in the Verbo ecosystem. I'm going to go into this a little bit more, but that's really kind of the starting point for it. And today I'm going to kind of tell you my experience of how I led the effort to build a Python culture at Verbo. This talk is a little bit different than some of my other talks before, because this one's really, really, I would say really anecdotal. I'm just going to basically tell you the story of what I faced when I got there, how I convinced my manager to allow me to start working on Python and start supporting Python in the ecosystem, and some of the lessons that I learned and some things that I found that worked when I was trying to convince people to use Python, some things that I found that don't necessarily work so well. So that's what this talk is going to be about. So who is this talk for? I always do one of these. This talk is for anyone who uses Python at their current job, but you use Python at your current job, or you're unable to maybe use it in production or use it with more people, or maybe it's not quite supported. You want to build a stronger Python community at your current job, and you want to learn how to introduce or evangelize any technology, not just Python. So a lot of these tips and tricks that I'm going to give you are they work for Python, but in reality, this is kind of how to evangelize and teach things on a broader scale, not just necessarily Python. Hopefully, you'll be able to learn from my successes and my mistakes to bring Python or any technology that you wish to use into your current work environment. So a little bit of background on the way that the systems work at Verbo. So engineers at Verbo deployed applications to an internal PAS that my team built. So if you wanted to deploy an application, you basically would create a Docker file with your application inside of it, follow a list of instructions that we gave, such as tell us where your logs are, tell us what port your health check is going to be going across, whether or not you want live track it, what environment you are, just a whole list of things. And they would deploy it out to this PAS. That was not Kubernetes based, by the way, because it was built before Kubernetes was even a thing. And that was basically what served all of the VRBO website, the Verbo website, and now has actually been migrated into the Expedia ecosystem. There were, at the time that I started, there were project templates that existed for the supported languages of the company. Verbo had initially started as a Ruby on Rails shop, but by the time I got there, almost all of the Ruby on Rails was pretty much gone. There was a little bit of lingering remnants, but none of it inside of our ecosystem, inside of the ecosystem that I maintained. The supported languages that we had were Java and Node. And what this essentially meant was, is that you could go into a, basically an application generator, so say you had a new idea and you wanted to create something. We had a whole fancy system where you would go in, answer a few questions, and a project would automatically be generated for you, and then it would be ready to deploy into this ecosystem, and would even be deployed automatically into the test ecosystem, so you could kind of see it working. Very similar to, say, how a one-click works in DigitalOcean or sample applications work in other PAS systems. So at that time, Java and then Node.js were the only two that were supported in this way. You could go what we called off-road with your applications, which meant in reality, the specs for the platform were very vague, as long as you had a health check and told us where to find your logs, all these things, the language didn't really matter, so you could build a Python app if you wanted to. And in fact, there was, I think, a handful, not a lot, but there were a handful of Python apps when I started, but there was no official support. So if your application started misbehaving, if something started breaking and you couldn't figure out why it was doing this, then you would be kind of on your own. Like we might be able to help you, but in reality, there was no official team that supported this. The Java and Node archetypes had their own team for this. So you could go off-road, but you really didn't want to. So that led me to kind of start the question, hey, let's start using Python. Verbo at that time was starting to get really into the machine learning and AI kind of things for some of the chat bots and things that we were building, but the workflows for that were not on the new system and the ones that were very suboptimal. So we decided to start figuring out, hey, what would it take for us to do that? And that leads to essentially a discovery meeting. So if you are trying to get a language set up at your company and you have it really, you're just starting it, you're gonna need to have a discovery meeting with key stakeholders. So you're gonna want to meet with all of the people who you think might be useful or who might be interested in consuming this. So for me, I met with the data science teams, I met with the machine learning teams, and I met with some of the API teams to gauge the interest because as we know, Python and Flask is a great way of building great APIs and data science and machine learning. Well, that's just kind of where everything was at least at Verbo, everything was being done in Python. So we led a discovery meeting and this after kind of pitching a couple of ideas, I come up with my first, what I call don't. And my first don't would be don't build something that nobody wants. So some of the things that I was talking about was Flask applications or Django applications databases and of those other kinds of things. And nobody really wanted that. In reality, what they wanted was they wanted a way to either run basic data science for machine learning algorithms or notebooks and stuff, or they wanted to just create simple APIs and they didn't really want like the full Django experience. So I quickly was like, okay, I evaluated what people wanted and I decided, hey, let's not build, don't build something that somebody doesn't want. So we talked with these teams, figure out what the interest level is and what you may want to actually build and what would they actually use and then it becomes valuable to them. Another great don't that is here is don't develop an isolation or avoid others' opinions. If I say I had just decided to build the Django app and what I was originally envisioning in my head and didn't get any buy-in, yes, it would have worked but nobody would have wanted it. So you definitely want to get others' opinions and make sure that there are people who are excited to use this product. If it's not really, if it doesn't excite them, then the likelihood of them trying to use it isn't any good and you'll waste a whole bunch of time trying to evangelize a language that nobody really wanted to use in the first place. So especially in these discovery meetings, it's great to just get as many opinions as you can ask everybody in your company because you need to be able to like sift through all of that and come up with a definitive plan. And the other hard thing is, is luckily when I was at Virgo, everybody was interested. But if you do a discovery meeting like this and nobody's interested, then that's sad, but it may not be the best use of your time to continue down this path. If you could ease, if it's obvious to see that nobody's ready to be interested in adopting either Python or whatever you're trying to evangelize at that point. So the result of this meeting was I decided to end up building a cookie cutter that creates a default Flask application for the platform that included built-in static analysis, unit test, project documentation was already set up, logging, metrics that reported back to Datadog, the base application itself, all of these things were built in, the CICD, all of the configuration files for the build and the deployment were already built in. So basically that's what we decided that we were gonna build off of. And what that leads to is build a complete product for them to use, use that for, build something that everybody wants to use where they don't really, if you open it up and it's the out-of-the-box experience is great and everybody's ready to use it, that's awesome and people will use it. If they have to go in, like say they want static analysis but you didn't provide it and they have to go add it in, that's more work, that's more of a barrier for entry to them, provide them with everything you think they would need but make it easy to disable the things if they don't want them. So when I started building this archetype and I started building this application that everybody could use in Verbo, I learned another good one is create a meaningful set of expectations and use cases. Don't fall into the trap of this will solve everything. That's not a really good way to present it. Python is great at many things but there are other languages that outperform in certain areas. So don't try to sell Python into situations that it may not perform the best at. So basically create a base standard application for Python and data science specific applications with many commas. That's what we decided to do. We had actually two different supported Python paths. There was the Python path of creating an API with Flask and a lot of people in the company really liked that but the data science people had much different needs. Instead of trying to make them use the standard one we just split off and then added all of the other things that they needed, many conda, some of the other I'm not a data scientist so I remember installing packages for them. That's about it. So we created use cases for people if the use case was big enough. Like if there was a big enough segment of the engineers that needed it we would create a specialized one for them hence the data science specific one. I had just mentioned this but the same thing with like when I said creating the static analysis unit test provide an out of box experience with sensible defaults. So the project whenever you run it it should work the static analysis should work with alerting on enough things. Like whenever I use pilot personally I turn off like the refactor or the convention ones because while they are useful I don't need to fail a build over them but if I have a missing import and that comes up as an error then yes I would like to fail the build over that. So that is what I consider a sensible default. I very clearly document how they can use this or turn this back on or make it more clear but with sensible defaults I just only go with what really in my mind matter. And then at the same thing as I mentioned provide everything unit test documentation was the one that people really liked because their products were already set up with Sphinx ready to go their flask gaps and it already used like the Sphinx flask web plugin so their project automatically generated documentation for them as long as they put the doc strings in. If you help people do with their documentation and you give them a place to put it they will actually do it but I've learned and if you ever seen some of my older talks if you don't people don't write docs I have a whole line of talks on documentation about that. So another really good one for you to do whenever I did when I was building the archetype is create good documentation around not only the process like how to use it but also why you would like to use it and why you chose to use things. So as you can imagine if I'm creating an entire Python ecosystem I have to think about what package manager am I going to use pip? Am I going to use pipM? Am I going to use poetry? Like what am I going to use? Am I going to use pipTools? There's so many different things so you need to create a rationale of your decisions. So not only do you document the use of the archetype how to use it, how someone getting started guide document all of the known bugs and limitations because there will be some like there's going to be gaps and things if you know it has a bug or an issue document it let people know that it's going to be a problem rationalize the why you chose technologies. I have for everything I chose between using PyTest for the unit test using pilot for the static analysis we use pipM for all dependency management and because of the we really like the lock file and stuff we used Flask. I used a couple of other different libraries inside of the base application but everything that I chose I wrote very specifically not only why did I chose it but I also wrote all of their competitors. So under Flask there was Django and Tornado and I wrote not only that these were options but why I chose not to use them because one as you can tell I'm not working there anymore but that product is still in use. So if anybody ever needs to know why why did I make this choice? They can go back and check my documentation and it explains at the time that pass Mason wrote it what he was thinking about and I also like to develop the document right about the development journey there's like a little mini blog in the documents for this project I got stuck on something for a week and learned a very valuable lesson about dependency resolution and I wrote it all out and I wrote out why I chose it and why I made these why this section of code exists what I was thinking when I was going through it because again, as you can see I no longer work for there for a verbal anymore but people still use it. So it's setting up the person who's going to inherit your code for success. Don't try to solve try don't use I made my grammar sped don't use Python to try to solve every case. Whoa, wow, I can't even don't trying to solve every Python use cases one tool. Okay, I remember what that is now. So what you'd need to do is yes, flask is great but I had someone come up to me with a request for using Django and they were going to use some database stuff and they had a really good use case for it and what I did was I made sure that whenever I created all of the sections when I created when I put it all together I made it very usable to easy to take out certain technologies and replace them it would be easy to replace a pie test with say unit test if you wanted to or replace flask with Django don't force people to use tools if they have a good use case for using a different one make it very easy make the interfaces very abstract to where you could easily remove the pieces and put in a new piece in we had someone use Django within the first couple of weeks of me releasing it and they complimented on how easy it was to just take out flask and put in Django because the way the project was structured the way that the way that it integrated with the static analysis and the way the unit test yeah they had to change a little bit of the code but it was not ingrained in so deep that it was impossible to change it and then another really big one because at the time that we were doing this we were that I was building this at Verbo we were discussing our open source strategy and our open source paths use as few internal or proprietary tools as possible one this the the closer you can stay to vanilla open source projects the easier it's going to be for you to upgrade things I have stories from my past that I could easily tell in another talk about dependency nightmares and the horrors of upgrading things that are 15 years past due and that happened because we got locked into internal tooling so don't do that it also it makes it way easier to open source later if you don't have anything internal one of the biggest boundaries you will face when you try to open source something especially from a very large company is are there any company secrets is there any company magic that is in here and if you've used nothing but internal stuff and then the code itself doesn't really deal with those secrets it's very it becomes very easy to open source these things so try to use as few internal or proprietary tools as possible it will not always be possible there were a couple of things in my archetype that were very custom to our CI CD pipeline but it was very easy to take those out by again by creating well-defined interfaces for us to be able to replace things we were able to pull out the internal proprietary tools very easily so this is a weird so at the time that I was doing this this is just part of the story we were working on the basically the version two of the internal pass for those of you that are Paz nerds or do DevOps stuff our initial version of the Paz was built using Mesos and Marathon as our container administrator and we were migrating into what into Nomad which is a hash product so Verbo's version two of the internal pass was being built at the same time that I was building the Python support and what I was able to do with this and I don't know to this day at the time this was a good decision for me this made sense for me some people may disagree I it was a toss up in the air the Python archetype was only going to support version two as someone who was on the Paz team who was actively trying to migrate people onto the new, better, more advanced system we decided to use the Python archetype as kind of a carrot and stick kind of thing where we could lead people into the new Paz by only allowing them to do it on version two now if you knew anything about the Paz it was actually relatively easy to port it over to version one but we didn't say that so as we were building it I was building concurrently with my team as they were building the Paz and I was helping with that I was also building this Python archetype we were going to, as I said, we were going to use it to entice this was really good because I could use this to take advantage of new features in the Paz and new things that we were trying to do and people got excited about it because it was things that you couldn't only do in the Paz system however, this was also bad because I became very heavily dependent on the Paz development timeline it took me six months to build this archetype in reality it should have only taken me two or three but because I constantly kept getting stuck behind the development of the Paz whenever they would have an outage or at the time that they were building this Paz they were still responsible for version one so if there were major outages on version one like all development time got lost so I got stuck behind it and also as people were testing it and they found errors in the Paz it was very difficult to distinguish an error between what was an error with my Python project and what was the error with the new platform so the reason I bring this up is because this was both a success and a failure at the same time it was a success because it did work to pull people over to the new version people actively migrated into Python they really liked it but at the same time it did leave a sour taste in a few people's mouths because the failures that were associated either with my code which I'm not gonna say my code was flawless there were times that I failed but at the same time there were also times that the Paz failed it kind of made it a push and pull so if you're, I guess with this one the lesson to be learned from this one is if you're trying to introduce Python in your ecosystem and say you're in a major DevOps change like maybe you're going from magic VM to Kubernetes style deployments and they're building out that stuff tying it to be it being released on Kubernetes could be a success for you but it could also backfire on you so just be weary that it will be difficult for end users to differentiate the difference between a failure of the Paz or a failure of your actual application that you're building so some, yes, so some testing that some things for testing that I learned find yourself an empowered group of beta testers so they will help you evangelize later so I had a couple of customer what we called customers they were not really customers they were internal people that were very excited about this application about using Python so they became my beta testers every time I released a new version they were the ones that would update their stuff and actually some of them were kind of were kind of brave enough to run this in production when it in my mind I wouldn't have they were they were that confident and they loved it that much they ran some of it in production before the internal passes even ready which led to they were some like some ancillary services that weren't terribly vital but it was really good because it helped us find bugs you will need to find people to test your stuff because after developing it for six months I knew every happy path through it and I started to become like blinder I had blinders on regarding regarding what was going on these first people or first customers will actually attempt to use it in production and it'll make your heart squeamish a little bit because you're like, oh, it's not ready but they do it and it's fun don't become upset or by the negative feedback or bug reports it's really easy for us to like sometimes take things personally when it comes to building these kind of things because we want it to succeed so badly because we're so invested in Python the negative feedback of the bug reports is more often than not a criticism of Python and not even really a criticism of you it's just that what they are trying to use how they're trying to use your product is not fitting their needs take this feedback adjust what you're doing to fit their needs and they'll love it even more so a couple of evangelism do's that I would definitely recommend doing if you want to evangelize not even the product so this is kind of what I did at Verbo is demonstrate the flexibility of Python let people know how easy it is to just write something like bang out an application or something and show it show that it's good for APIs, it's good for data science maybe it's good for, you know, log processing and stuff like that show all of the different things that I can do and people will come to love what the product what the language does and they will want to adopt it rewrite existing applications that are considered troublesome or complex in Python if you have old languages or old applications that are always going down or you're constantly having to fix them because they're just unstable or they're just overly complex try writing them in Python I have found many times that whenever I rewrite complex C or Ruby applications in Python they become less complex just because of the nature of the language so by doing this instead of what you can do say with your manager is instead of patching this for the 10 billionth time can I try just rewriting it really quick can I just get like a sprint to see if I can replace this application or at least get a proof of concept of this application in Python running, you know hopefully at the same performance level and it's easier to understand it's easier to maintain because maintainability is one of the big things especially when you're in like a DevOps or an SRE thing write new applications or features in Python if you're adding a new microservice or a new API or something pitch the idea that hey let's just try it in Python we have this, you know we've done everything we've done so far Golang or Ruby or something let's try it in Python and just see how much easier it is if we don't like it we can always go back and rewrite it but let's just give it a shot let me try it you'd be surprised how often people are willing to let people just go and try something on their muse you know managers are meant to like you know they're meant to trust us they have to trust that we know what we're doing and we know what's best for the company and I'm not saying Python is better or worse than any specific language but you know we as Pythonistas know how easy it is to maintain Python and in reality the ease of maintainability of a programming language of source code is more important than almost anything else yes performance is great and all that we can get very close to Python but like if I can't maintain it and it becomes unruly then it becomes a black box and then it's its own set of problems be the temporary solution this is kind of what I consider kind of a sneaky way of showing the powers of Python but it does work I may have done it once or twice if there's an outage or something's going wrong or you need to like you need to write a tool or a piece of code or an app really really fast just say okay I know I like I know I can write this in Python really quickly let's go ahead and just write it really really fast and you'd be surprised how often those remain in production as my ex-boss used to joke with us and I used to laugh at it but then I realized it wasn't really a joke and it kind of makes me sad there is no discernible difference between a proof of concept and production software I have a whole story about that that if you want I can tell you about it later where one of my test pieces of code made it into production I'm still upset about that some evangelism don'ts when you're trying to evangelize Python or anything don't say that language sucks you should use Python or you should use X as we as Pythonistas know people are very protective of their language they're very opinionated when they come to the language people don't write in a certain language because they don't love it like we love Python that's why we choose to write Python so if you come off confrontational from the get-go you're not going to win anybody's affection all you're going to do is put not only a negative connotation around Python you're going to put a negative connotation around the Python community which is honestly I would say some of the bigger damage or the worst damage you can do because the Python community is amazing hence all of us being here so just don't do it don't come off adversarial with this stuff it really doesn't work out don't try to use Python to solve problems that Python doesn't handle well if you need a super high-performance, multi-threaded thing that behaves at the level of C and is really really really really fast and really really high-performance amazing amount of cores maybe Python's not the best thing for that it might be but maybe it's not maybe Golang and C might work a little bit better there whenever people see things fail they tend to blame it maybe on the language itself so and it's really if you put Python in a situation that it's not equipped to run or do as better than other languages and it fails, people blame the language when in reality it's not the language's fault it's actually your fault for picking the wrong tool I don't try to screw in screws in my wall with a level it just doesn't work like you can't do it so don't do that it's a really bad way to give people a negative impression of a language right off the bat and those impressions stick don't be overbearing if someone isn't interested in using Python right now that's okay don't be like a pusher constantly trying to push it continue to do what you're doing continue to build cool tools and stuff continue to show off some of the neat little things of Python and eventually you may change their mind but if you don't that's fine too it's totally fine for not everybody to write Python the world couldn't exist if we didn't have all these programming languages this is what makes computer programming and software engineering such a fascinating field to study so don't be overbearing bring it up every now and then but you know if you become annoying then it's not going to go over well so that's pretty much my story the results that I can say that I saw from this are within the first month 80 applications within the ecosystem were using the new archetype in production so there was a demand for Python whether or not people were openly saying it and there was a small group that was people did want it and then by other people showing other people in the company it kind of began to grow other people also then started using my development journey documentation to start building support for Golang so not only did I influence people using Python I started influencing other languages being supported within the company through this so proper evangelism of a good project and document proper documentation all this can actually help support the evangelism of other projects you can support Golang and you can support other things it's all a harmonious system and we all live very well together so that's that's my story like I said this is a very anecdotal story of just of my experience inside of the industry thank you for tuning in today feel free to tweet me if you have any questions I'm at Mason Egger on Twitter you can also message me in the discord I will be happily happy there to discuss anything regarding this talk or just anything general I like talking to people and if you like listening to me present and stuff I do a live stream at one o'clock on Tuesdays feel free to come and see me and if that is everything then I will turn it back over to our moderator