 Thanks, thanks everybody. Hello. My name is Nikola Jipanov and I'm gonna be telling you about how we at Facebook use Python to help us build data centers at scale. Some of you may remember me from previous years from such venues as the riverbar I'm a production engineer at Facebook Production engineering as a role is kind of like Mix between software and systems engineering. I'm not gonna talk about this in a lot of details Now but if you're interested you can swing by our booth down at the marketplace and We can tell you more about this So this talk also has an alternative title which I'm sure is a theme that you've heard a lot about in this conference Still and it's automating the things that you didn't think you could automate So Facebook is growing really fast We have more and more users doing more and more things on the site This means that some things that we could get away as a company by doing by hand We can't do anymore because we're not gonna catch up with the requirements off off our scale Now I'm not saying that this applies to any company out there, but an interesting thought experiment would be If you consider that Computers are getting cheaper It's actually getting more and more expensive every year for you to not use computers to do things Because humans are not getting cheaper in theory So yeah, this may not apply to Everybody here, but at Facebook We really Needed to stop and think about how we can stop doing things that really are not gonna scale and this is one story of such a thing So a Fabric this is this is how Facebook designs data center networking This is the so-called spine leaf topology I think it's a design that somebody came up with in the 1950s So at the bleeding edge of of technology as always and What this allows us to do is add compute capacity without having to mess with the network Among other things. I'm not a networking expert But we do have some networking people here at the conference. So if somebody's interested in this particular thing You can again swing down by our boots And I'm sure some of the networking folks will try to help you and Interesting data point is that they're networking people at the Python conference. So yeah, that's kind of like part of the whole team theme Yeah, so but The Facebook we actually have Yeah We have many of these fabrics because we have a number of regions That have a number of buildings in them. So this adds up quickly at Facebook So question For the audience and a little bit of a nausea inducing animation Thanks to PowerPoint as how much fiber cabling connects all of these fabrics? like maybe venture guests and So, let me just say that for astronomy nerds in the audience, this is not an actual representation of planetary motion Just to you know, get it out there So turns out there's quite a lot of fiber cabling in our data centers and Some of my colleagues from the data center teams calculated that it's about a million kilometers of fiber cabling That we have to deal with So it's interior enough to In this make-believe universe go to the moon and back and around and Yeah, so We can think about this in terms of five reports that we have to deal with in our data centers and as we are adding more capacity We have less time to do the things we used to be able to do without automation and We can't do it that anymore because there's just more stuff happening today we are at Approximately between 10 and 20 million ports, but that number is gonna grow in the next couple of years rapidly Yeah, so this is This is really not where we want to be and When it comes to operating this stuff, we also don't want to have people deal with this kind of stuff and we need to we need to Do better and Then cheesy animation later We built robots with funny tattoos to help us do this and One of the tattoos is a python logo So it's on the slide yeah, so this is a nice Sunset California sunset photo of I think it's building 16 in the main Facebook campus in Menlo Park, California The address is one hacker way and this is kind of like What we at Facebook believe is a good way to to solve problems is to hack So we hold these hacks, which are also kind of like hackathons. They can be couple of days long Nowadays mostly people go home and sleep. I don't think this was the case all the time, but nowadays it is and We take this very seriously lots of engineers take a couple of days to think about Exactly the kind of things that that I try to talk about here like what things can we do better and Then maybe sit in a room for a day or two and try to come up with a quick prototype and many many internal projects and Even external facing products at Facebook started out as like crazy ideas that people people just set together in a room and hacked on for a for a day or two And yeah, this is an aerial shot of the of the campus building 16 is at the top right corner I think and you can see the San Francisco Bay at the back Like realistically you cannot just hack your way through some problems you can start out like that and that's a good way to To build a prototype test your ideas, but As you want to use this and make make better use of it you have to put it in proper production And There's there's one language that's actually quite good and helpful when you need to come up with a fast prototype, but actually it It works well in production as well You probably heard about this language But yeah at Facebook. We're kind of like thumbs up thumbs up in Python here's some numbers from our One single repo that we have that I pulled out to June These are rough numbers, but we actually have thousands of binaries running in production Decent number of them is Python 3 We have millions of lines of code 40% if you look at the line of code count is actually Python 3 so We are very invested in Python 3 and it's the second most used language for back-end services after C++ and Our like most people probably know that like the front end of Facebook the product is written in Slightly improved or a significantly improved version of PHP so we don't count this here but everything in the back end is mostly C++ and Python with some other interesting bits here and there So how does it all work together? You have all of these servers running Python C++ Some people like Haskell. It's apparently cool or used to be The Facebook we use drift Apache drift it's basically an RPC framework and What it does is like you can define types and you can find the interface of your service and Then you have a compiler that compiles this and generates language specific code so you can have services written in different languages That they can talk to each other seamlessly and all of your definition lives in a in like File like this. So this is obviously a very contrived example. This is not actual code but like our System to help us deal with a lot of fibers would maybe have an interface like this You would define a fiber type and maybe define a location type and then you would have a fiber server and Then you would be able to Ask what kind of fiber cabling is happening at in a certain location? and then like once you run the a thrift compiler and generates some code then you get Generated code which you can then import and it's fairly easy as you can see there's only like about six imports Then you have to actually write the business logic of your service. That's we still haven't cracked that problem. I mean how not to do it and then You instantiate your handler. There's a processor. There's some kind of there's two factories because one is enough just generally this is This is not very Engineer friendly and it's also there's a number of problems with this like there's no Signal handling. There's no monitoring. There's basically nothing so We came up with a solution to this and open-sourced it it's called sparks it's basically a services framework for Python and At Facebook we use drift for everything So it's a poor strift, but it also supports HTTP and it does a lot of cool things. You have periodic tasks background tasks. You have logging Command line arguments all sorts of nice things like that. So even if you're Not interested in using it as such. It's very nice readable code So you can take a look at it if you're interested like this is a good example of how you would write a Service framework in Python, which I'm sure like is becoming more and more common nowadays and then if you if we use parts this whole boilerplate becomes Much better. There's just you still have to write your code. We can't help you with that yet But then it's just like a couple of things and this thing would do way more than the previous example did because it will expose metrics so you can monitor your service and go in like five weeks of location like Hinnok did and All sorts of other things that you get for free and so for those of you who are paying attention I didn't mention binaries before and most people think of Python as a Interpreter language, which does not have binaries. It has source files that are then interpreted by the C Python interpreter And a Facebook we actually don't use virtual and pip or any kind of thing like that. We actually do build binaries and We distribute them to all of our machines also using Python, which I will tell you more about this in a moment so we have one build system to rule them all in and In Facebook it's called buck and it's also open source So if somebody's interested in this you can go to buck build calm and you will see you can read more about You can read more about buck and Basically we use buck to build Python binaries This is not the new technology like this has done their open source solutions to this But what we do is like we put it in a zip archive and then prepend it with a shell script that knows how to Extract and execute execute the code. So all of our code all of the binary dependencies all of Python pure Python dependencies would be in a one big blob that you then Sent to all of your servers buck can help you with a lot of other things that caches your build artifacts So you don't have to rebuild things all the time that don't change and it has a very nice reproducible way to define your builds and If you take a look at this, this is actually also Python code So this was actually very useful for me to come up with all the stats From couple of slides back because you can use things like the AST module to parse this and figure interesting things out About your build. So in this case, we define our drift library which tells the the which tells buck to build the and generate the Generated code and then we have a Python binary that depends on this library and Runs server pi which then solves all of our problems interior, of course So yeah, like I said now we have binaries Facebook we use bit aren't Which is also powered by Python and async IO to take those binaries and distribute them to a huge fleet of servers where they need to be so yeah, Python is heavily abused and This leads us to the kind of like the Tldr of the talk and if you take anything away from this, it's well actually it's two things that Some things about how we use Python at Facebook, but also that the robots are coming and I was really practicing this, but I couldn't come up with With a good good enough impression of an Austrian accent saying and they need your clothes So imagine I said it in an Austrian accent and But yeah, like I said as things scale you need to automate more and more things I think this has become a very big team in a lot of in a lot of talks that I heard today and and the previous days At this conference so It's something to think about it's very for me It's very interesting to think about how really a lot of things that you didn't even think you could be automating are starting to Pieces of it are starting to get eaten by computers and done by computers more and more and at Facebook We use hacks. It's it's a very fun interesting way to To come up with ideas of what things can go away and be done by robots and Yeah, and With that I ran a little bit short on time, but We're hiring and What's interesting for these folks? I think is London and Dublin offices for production engineering. So if you're interested to come and build robots with us and maybe do cheesy slide decks Yeah, come to the boot and and talk to us. That's it. Thank you as for questions I would prefer if like people grab me at the boot. I'm gonna be at the boot whole day tomorrow, so I Think it's a little bit awkward to talk like this. So come grab me after at the boot and and we can check Thanks a lot