 We work at a company called Braintree. We make it really easy for you to accept credit cards, PayPal, and other payment methods online and in mobile apps. This talk is not currently on GitHub, but it will be shortly, and you'll be able to find it there. So let's talk about the title of the talk. Python not recommended. It might sound like it's a joke, but I mean it quite literally. At the company where I work, we have something called the Radar, and it contains information on the technologies we use, whether you should use them, whether you should never use them again, and sort of where to look for examples and other relevant information. And in the Radar, Python is specifically listed as currently in use, but not recommended. So this talk is about why we originally used it, why it's not recommended now, and what we still like it for and what we're gonna use it for in the future. So first some background. The obvious question is what do we use instead? If we don't use Python, there's gotta be something else we use as our general purpose language. And the answer is Ruby. Braintree loves Ruby. Braintree uses Ruby by default for pretty much everything. But I promise this isn't a Ruby talk. I personally am a Python person. I don't love Ruby, so we'll talk about it only as background. Braintree has bought into the Ruby ecosystem. We use Rails as our web framework. We use Capistrano for remote server management. We use Puppet for configuration management, and we use Rake for scripting, builds, and automation. So you probably wonder what's it like doing Python at a Ruby shop, something I've been doing for three years now. There's a lot of jokes about Python because it doesn't have quote unquote real lambdas, which people just mean that function definitions are statements and not expressions, and people are like just to be able to define a function wherever they want. And even though functionally they're the same, people don't like the syntax difference. It's seen as an elegant compared to Ruby or Elixir or Closer. This is both because of the statement expression dichotomy, but also because of the generally structured syntax. And because in these languages, magic is really easy with macros in the list or with everything and Ruby is magic. And it's just a little harder in Python and people don't like that. It's sometimes gets dismissed as a dying language because of some of the negative publicity around the slow adoption of Python 3. I don't see it as a failure, I don't think the Python community sees it as a failure, but people outside the Python community sometimes see it that way. The languages are also similar in a lot of ways, almost too similar for their own good when you're a Ruby programmer. A lot of people will sort of apply judgments that you'd apply to Ruby code to Python code without taking into account the difference in idiom. And so if you were to apply to brain tree as a Python developer, it might hurt you a little because people aren't gonna expect your Python code to be written like Ruby. And it also means that our Python code can be a little less than idiomatic. So despite using a lot of Ruby, first and foremost, we believe in using the right tool for the job. So you're not gonna find a ton of this at brain tree, although it does happen once in a while. So what do I mean? When is Ruby not the right tool for the job? And we've sort of found two main times. One is when you need the JVM ecosystem, you might say, oh, just use JRuby and you have access to everything. Well, we found that that gets a little messy and that it's not really a good long-term solution, although we have done it in the past when we needed Java in the short term. Primarily, we'll need to use Java if some third party who maybe wrote their API in 1999 only has, doesn't have an open API spec and they'll just give you a library and your choices are C++ or Java. Well, we're gonna choose Java. There's also a lot of great tools that work best with the JVM, like Apache Kafka, which is a message broker for handling high-volume data feeds, and Apache Cassandra, which is a distributed database for handling large amounts of data. So what do we use when we need the JVM? Well, like I said, we have tried JRuby historically along with Java. We haven't found it to be that great. And then more recently, over the past maybe year and a half, two years at this point, we have used Clojure pretty successfully, although we've still found that there are times that you really wanna use Java directly. So the other main time that we find that Ruby is in the right tool is when you need to write a smart proxy. Now, our business is basically to sit between somebody who wants to sell something online and the banks and card networks. So we basically are a smart proxy and so our big smart proxies build up of a lot of little smart proxies. You can see this is the logo from a party that Braintree and PayPal threw at South by Southwest a couple years ago. It's smart, it has a brain, so it fits. We have really high uptime requirements. We need to be available when there are temporary networking problems, when we need to fail over services, when we have to run database migrations, when we get huge traffic spikes like, I don't know, Uber is, it's New Year's and Uber is running tons of rides and they wanna charge people's credit cards. And we also have a big problem with the services behind us, the banks, the card networks, going down and we need, at the very worst case, requests to fail gracefully and preferably if the outage times are short, like on the order of few seconds, we want those requests to succeed even though the service behind us failed. So we use them to make our outgoing connections appeal, appear highly available to our internal services. We use them to pause incoming requests so that our internal services won't actually see any requests coming in and we can do whatever we want to them but actually clients are still able to connect to us. We do custom rate limiting, we have pretty complex rules for how much capacity, different clients and different types of requests can use and because we integrate with a lot of legacy services, we often find that we need really weird SSL or persistent connection configurations and this is something that we've often had to do ourselves. We also have very complex retry logic and a lot of other custom logic that we need. This is where we've historically used Python. It might seem a little odd since handling a large number of requests doing a lot of concurrency is not necessarily the first place you think of Python but it was actually a pretty good fit for a long time specifically because of Tornado. Tornado is a web server and framework for doing non-blocking IO and you get the benefits from Python of rapid development and easy to learn and you still get the necessary IO concurrency to handle tens of thousands of requests. So back in 2013, Python was in pretty good shape at Braintree. It was in use for several of these internal proxies and it served us well for a couple of years and it had several internal advocates not just me but other people too. So what happened that I'm giving a talk called Python not recommended? The platform really did fail us as we started to scale. Concurrency in the framework really isn't enough. Nowadays we sort of expect our languages to have concurrency built in as Python 3 now does and as Go and many other languages do and you really expect that concurrency logic to not get in your way when you're writing business logic which we found that the concurrency logic in Tornado really does. It was also too much work to keep up with changes in Tornado. We looked at new versions a couple of times and it would have really been a complete rewrite to use the new APIs and we really didn't wanna spend the time for that and we didn't trust that we would be able to make those changes without breaking anything. And so because we were using an outdated Tornado version, maintenance overhead could really be pretty high. You can't Google things easily because people have moved on to newer versions. It's hard to find the right docs and you end up in callback hell because you're still using old, less elegant APIs. We found that logging actually has an unsettlingly high overhead. Every time we logged a line, we saw requests pause and eventually those pauses added up to enough to be really significant as our volume scaled and then there's no SNI support in Python 2 historically. SNI lets you serve multiple SSL certificates in the same server port. A lot of our customers used it and it was only very recently introduced into Python 2 but the version of Tornado we're using is so old that even if the Python supports it, the Tornado version doesn't. The applications that we wrote also really failed us and a lot of this ties in with those platform failures but it's also to a large extent our fault. The smart proxies were really too smart. The logic was all mixed in with the concurrency boilerplate making it hard to understand and they were trying to do so many things that the code ended up that was meant to do one thing was coupled with the code that was meant to do something else so when we tried to rip out the code to do connection pausing, it completely broke rate limiting and we had to put it back and leave this completely now unused code in the code base. Straightforward Python implementations as we scaled were not fast enough so the rate limiting code started to add an unacceptable amount of overhead to every request that came in. We also found that in addition to being too smart the proxies were not smart enough. We couldn't just write certain pieces of business logic in the proxy. The business logic had to be duplicated in the main application and that's something that we really don't like. We don't like writing the same logic twice and none of these applications were really built for horizontal scalability. They all assumed that a single or in some cases a small number of instances would be enough forever and so they weren't designed for us to run 10 or 100 of these. They were designed for us to run two or three. So what solutions did we switch to? What made Python obsolete in these areas? So for our incoming request proxy, we've switched to a combination of nginx and haproxy along with pgbouncer. So in nginx and haproxy, we're able to do approximately the same complicated rate limiting logic and load balancing that we were able to do previously with our proxy in a pull layer and we've also moved pausing completely out of the proxy layer and into pgbouncer which is basically another proxy that sits between the applications in PostgreSQL. We then wrote our main outgoing proxy as in Node.js but this was actually a failed attempt. It had all the same types of problems as Python. It was still trying to write our own tool to do a job that we weren't experts at and we had problems with failed persistent connections and with memory leaks in Node that led us to abandon it and move to nginx and haproxy. Now the key here was timing is that haproxy 1.6 had features that we really needed and so we've now moved to that and it allows us to remove another custom piece of code from our system. Finally, our sort of most complicated outgoing proxy, we've decided to rewrite in closure using Apache Kafka and it allows us to centralize all the logic in a single application. We can build it pretty easily to horizontally scale almost linearly and of course you get SNI support with the JVM right out of the box and unfortunately we canceled that project because it wasn't a high priority, it was gonna take a lot of time and so instead I wrote a monkey patch of tornado to support SNI even though it's a really, really old version of tornado so it's not great. We still have logic duplication between different applications and we still have sort of lack of horizontal scalability, we basically run two of these proxies but for now it's okay but the problems are unsolved. So as of late 2014, all the smart proxies were on the way out or in use but not because we wanted to. Not recommended for new projects, it's official, it's in the documentation that we use at Braintree and there were fewer internal advocates. Now this isn't because all the Python developers got mad and left Braintree but a lot of the people who used to really like Python now have moved on and prefer languages like Go or Closure or Elixir, some of which have more in common with Ruby so you can understand it and some of which just have better concurrency primitives than Python too. Just to be clear, I don't fall into any of these groups, I still like Python and still use it outside of fork. So this is kind of sad and it makes it really sound like the state of Python at Braintree was really sad but that's because that's the point of the talk. We've definitely, there is something we will never use Python for again but actually things are looking up overall. We're now using Python in areas where it really shows its strength instead of just sort of the Swiss Army Knife glue code of our code base. So the first place is data analysis. This is probably surprising today, exactly no one. Our business analysts have really used it to replace Excel to write sort of one-off reports and do smaller monthly tasks that don't really need to be automated. Our data analysts have moved more and more from writing giant crazy blocks of SQL to putting more logic in Python so that the code is more maintainable and understandable to a larger group of people and these are people who've maybe done a little programming before but are really buying into Python very rapidly which is cool. And finally our data scientists are really using it to replace R. Part of that is because of the great modeling and analysis tools that Python now has but primarily it's because it's a lot easier to deploy your solution. It's a lot easier to do the sort of ETL steps that happen before you do the modeling in Python than it ever was in R. Finally, well, the next thing is really infrastructure management and this is somewhere that historically we bought into Puppet wholesale. We have huge repositories full of Puppet code. Puppet is a Ruby-based tool but recently we found that Python has a really good niche here and we use it to manage certain resources like IP address, physical ports, server locations. This is something we actually didn't do with Puppet. We did with a bunch of Google spreadsheets so the centralized application with a lot of the like better views into the data is really helpful. We've also used it to manage cloud instances because we run our own physical hardware for a lot of things. We had a pretty unsophisticated setup for managing cloud infrastructures namely we used the user interfaces. We basically log into the website and would start and snap instances. Now that we're starting to do more and more things automatically and need scaling, excuse me, use scaling, we had to have an automated tool and Python is the right way to do that. And finally, we use our switches for slightly complicated setups. We do try and make sure that even if switches fail in our data center, everything keeps going so everything is, all the networking is mesh and the switch configurations are pretty complicated and we didn't find a good way to manage them with Puppet. And we found that rather than writing a custom Puppet module which can be pretty complicated, it was actually easier to emulate what Puppet does in Python and pull from the same data source that our Puppet repository does. And so all the code is in Python and it's like 100 lines rather than writing a very complicated Puppet module that we'd never be able to change because no one understood how it worked. The Python community is sort of the final thing. It's a big advantage of the language in my opinion and something that has been very beneficial to Braintree. We host a lot of Python meetups. We have a monthly project night. We host Chippy, the Chicago Python meetup about twice a year. We've done a couple of events with Pi Ladies Now and one with Django Girls and we've also sponsored other events outside the office and this has really helped with our hiring. We've hired several people now who first heard about Braintree through these events and came to Braintree even though they don't get to write Python because they know we support the community and recruiting is one of the hardest things we do so this has been super helpful. We also find that giving talks is a great way to spread the word about Braintree and I'm not the only one giving Python talks but one of my colleagues has given a couple in Chicago and has given one with me at Northwestern University. Our customers also use Python so having us support the Python community makes them feel more connected to us. Some of the biggest startups in the world who are customers of ours have pretty large Python code bases and it's much easier for them to keep using Python to connect to us and that includes as they migrate to Python 3. Our Python library is single code base, Python 2 and Python 3 and a significant amount of Python 3 traffic comes through our API so having that and supporting sort of the next, the future of Python has helped us gain and keep merchants. So now we're at 2015 so what's the state of Python now? Python 2 is definitely showing its age internally and in general especially around concurrency I don't think anybody really disagrees with that and as the standard tools like HAProxy and Nginx improve more and more we're sort of losing a use case for Python. Being the jack of all trades the language you can use to write whatever tool you need is a little less important as the standard tools tend to be there for scaling and for high availability. Data science is really important for Python's future I think. It gets the foot in the door for Python at pretty much every company which is a great way to keep people who like Python interested and to sort of keep it in your mind for when to choose it for other projects. The community is also really important it's been great for us at Braintree and it's just one of the reasons that Python is as successful as it is. Thank you, that's all I have. I'm glad to take questions. Thanks, first great talk, thanks a lot. I have a question about like if you see yourself as a tech hub more or less if you have some... I'm sorry, I'm having trouble understanding you. Sorry. I'm a little hard of hearing so you're gonna have to speak up. Thanks, and the question is about if you have any interns, I mean people who like students or such walking into your company, are they keen on learning Python or something more, how does it act in your community? I'm sorry, could somebody else repeat the question? I just had trouble understanding that. Is it advice for students about choosing Python or something else? If they come to your company or your like environment, are they keen on learning Python or something else like Ruby or stuff? So what do they do? What do they choose? How do they act? Do they choose to learn Ruby or choose to learn Python? Yeah, yeah, more. So I think that we definitely look for people who don't wanna do one specific thing. We look for people who wanna learn and want to use whatever tool is best for the job at Prane Tree. So I think that a lot of the people we hire and a lot of the students we talk to are people who have maybe done Python at school but are open to learning anything. I'm not sure if that answers your question. The non-programmers or the non-professional program is like the analysts you talked about. Did they choose Python because they tried other options and like closure and didn't like them or because they already knew Python or because you suggested it or what was, how does it find its way into their job? So the sort of, the guy who started our business analytics team, he'd been doing a lot of our reporting manually in Excel for years and he decided to learn to program. And so first he actually learned Ruby and he wrote a web application that we used in production for years which was