 And be able to come and speak at FOSDEM about how much open source we use inside of Facebook the open source Infrastructure that we've created and released as well I sort of give a bit of an idea into what goes into scaling a site like Facebook So I'm David recording. I spend my time at Facebook focused on open source and web standards And my name's Scott McGehr and I work on anything open source that's PHP related within Facebook So the recently released hip-hop is my little baby at the moment So first under Okay, so first we should really talk about what do people do on Facebook? So you can sort of get the idea of the scaling challenges that we sort of have there for example eight billion minutes every day are spent on Facebook and It's over 3.5 billion pieces of content is shared every week be it photos writing on the walls And talking and actually 2.5 billion photos every month are added It's actually makes us bigger than the biggest photo sharing website out there And there's over and it's more than just the actual website There's also the API and the platform that people build with Facebook. There's a million uses of users of that And so that's sort of an idea of some of the amounts of content that's being shared on Facebook the amount of developers They're going and building with Facebook, but Facebook is also incredibly international over 70% of our traffic and our users come from outside of the United States We've gone and built a translation system which is focused on going and engaging the user base on translating the site I think we translated Facebook into French in just under 24 hours And so you can sort of see how we've gone and spread around the world Which also presented a set of scaling challenge challenges in terms of how do we go and serve all this information quickly? With only having a small number of data centers So when you think about scaling a traditional website, you're able to go and sort of break up that information So if you have an email application that user is generally interacting with their own email They're not interacting with email from lots of different places So you're able to go and shard your databases You're able to go and break this apart by your users of saying oh Bob's interacting with data here Felicia's interacting with data there And sort of continue just to break that apart as you have more users You add more databases and you scale from that perspective And so when Facebook started Mark Zuckerberg created it in his Harvard dorm room And it was sort of pretty similar in terms of all the users were on a Harvard server and then as more colleges were added You were able to go and shard your databases based on each college So Students at Harvard would interact with other students at Harvard at Stanford that would interact with other students at Stanford But then in 2006 Facebook sort of opened this up So you could go and have friend relationships across these schools and pretty soon you got to the point where it didn't matter Whether you were on the Harvard server or you were on the Stanford server or you weren't on the University of Washington server But you were going and interacting with people from college campuses all over the country And so you might think oh well You could just go and scale Facebook from a country perspective put all of the users in the United States in one place Put all the users in the United Kingdom in another put another server for Belgium We're pretty quickly Facebook got to the point where you couldn't even do that And so this is a visualization showing live friend requests around the world from a few years ago Really going and illustrating that point that on Facebook everyone is connecting to people all over the world It's not a site where you can go and scale broken up based on where your users are or just what information they're interacting with And so this is really that idea of that to go and render any page on Facebook We're pulling data from many different places on average a user has about a hundred and fifty different friends And so to go and render a page like that We're going and talking to all sorts of different pieces of our infrastructure Without being able to go and sort of separate it apart by user so if we take like a standard page on Facebook say the news feed and On average a person will have about 150 friends so to construct that we need to go grab data from 150 different friends and that's a split across multiple servers So we have to that in like less than as milliseconds. It has to be done in so we go do that grab all the data But it's not just your direct friends if other friend their friends have commented on their photos or The comments or any of their status updates that has to be pulled as well So we're pulling from 10 to thousands of different sources all just to render that one single page And it's also that every single page that we go and render is different not just per person But at what time they saw it but the graph has also been evolving I mean Facebook has always been based on this idea of that you're connected to other people and to other things that you have Relationships and so from a math perspective we think about it as a graph So we have nodes which are the people or the things and then we have edges which represent the relationships between them But we've also seen Facebook grow to no longer just be relationships between people But you might become a fan of a company or a person and it's a very different scaling challenge of going and connecting a person To a few hundred other people versus going and serving Michael Jackson's fan page, which has over 10 million people connected to it So that's also presented some different scaling challenges as the sites continue to grow Okay, so this is Facebook's architecture architecture at a high level And it more or less looks like this There's a load balancer on top and requests come in there and the requests are spread amongst a pool of web servers And then these web servers will either use one of our services to fetch data or fall into memcached Which is our in-memory fast database access data access before eventually falling back to the database And I mean this looks like pretty much any website that you're going and building today in terms of you have your web servers You have your PHP stack you have your services So at a high level Facebook isn't architected very differently than any other site that you would go and build So first we're going to look at the web server stack or Essentially what PHP is built on and that's a Facebook Facebook. What Facebook is built on which is PHP And we like PHP for a number of reasons the first being it's simple to learn There's not very much barrier to entry to learn PHP So if we hire new developers then we can teach them PHP relatively quickly because the syntax is very similar to that of C and It's an interpreted language So if you want to make changes to it you make your changes you deploy them or save them and you can see them live It's not compiling you're not waiting five ten fifteen minutes for things to compile so you can see it And one of the nice things about PHP is the hello world example The whole world example in PHP doesn't actually require any opening tags or closing tags because PHP itself is a templating language So this is the sort of the nice things that make PHP good for Facebook Because we have to develop fast and serve we move fast so we can't spend time waiting for other languages Okay, so there's a couple of things that are PHP problematic for Facebook the first being high CPU usage Because we do a lot of data assembly pulling from multiple sources and actually building it within the application server Then we actually spend a lot of time doing that with CPU We use a lot of CPU for that as well as memory usage memory is quite high because of the way PHP works It's sort of loosely typed which is well one of its benefits It also uses a lot of money for each variable because it has to represent a collection of six types So every variable gets a hundred and forty bytes of memory allocated for it We'd also potentially like to use a logic between multiple services We have things written in C++ and Python and Erlang as well as PHP So if it was just written in solid in PHP, we're gonna have to duplicate that logic across multiple services And finally if you want to sort of move this into PHP extension to get the speed benefit Then it's sort of hard to write because the macros aren't well documented for the Zend engine So to solve these issues especially the first two we come up with this thing called hip-hop and Hip-hop for PHP was this thing that we sort of start announced last Tuesday and tend to open source quite soon and what it does is it you point hip-hop at Your application framework and what it'll do is it'll translate your PHP and to highlight optimized C++ code and Then from that we can just pass it into the normal G plus plus compiler and it'll produce self-contained binary with the web server and your application and all the library logic built-in And hip-hop is really designed to give us sort of the best of both worlds in terms of the speed that we you can go and move with An interpreted language so that we can go and make changes really quickly From a development perspective, but then also the speed from a performance perspective of a language like C++ And so one of the ways that we think about the different types of PHP code You'll find is sort of this break down between the common stuff of like if else or the magic in terms of going and doing like variable in terms of like evaluation and Like dynamic variable and function assignment and so when you look at a PHP code base You actually generally find much more of the more mundane things It's very rare at least inside of our code base that we use things like a vowel or some of those other Functionality and so this is some of what makes hip-hop work really well Is that we can go and take the majority of our code base and greatly speed it up in terms of going and looking at Stack function calls and stack variables and then taking advantage of a lot of the optimizations that already exist inside of g++ The stuff that's magic is a little bit harder It's not going to give us as much of a performance optimization I won't give you as much of a performance optimization, but there's definitely some benefit there as well So roughly how hip-hop works as it performs a static analysis on your entire code base Looking to see what function what class is declared where and what variables are declared And from that then goes into our type inference engine which looks at the data types for each of the variables So if something was an integer and it was always assigned as an insurer There's always acted like it was an integer then at that point we can just say well This is always going to be an integer. So let's generate just a straight integer rather than our variant type So we're required support. All right, this is for all the variables So integers map to actual integers in C++ strings map to strings arrays map to the STL like the standard array types and Classes map to actual C++ classes And it's only when we have a bit of the magic stuff or when types are entered a type of variables an integer then it becomes a string then backed an integer again that will use the variant type and then eventually go to the code generation and The code generation is what puts actual C++ files and the transformation process for this looks a little Like this so it goes through a parser that's very similar to the standard PHP parser And then former static analysis on it and then a pre-optimizer and the pre-optimizer tries to remove things like dead code branches that won't be used or change Change some of the semantics when it knows that the execution is produced the exact same code regardless of the order So say you're doing a post-optimization. It's actually easy to do a pre-inc, sorry Post-increment. It's actually easy to do a pre-increment because then you don't have to use this third variable to store the result So it does like little optimizations like that before it was into the type inference engine And eventually there's some more post-optimization to remove any more dead codes that you could find Then at the end it's just code generation and then compiling is normal So we have some plans for hip-hop over the next Next year and at the moment only supports PHP 5.2 So we intend to get that back up to speed with the latest 5.2 release before eventually adding some of the new features from PHP 5.3 And our few things that we're thinking about that aren't currently planned for PHP But we want to sort of make sure it gets in there as well as in hip-hop so one of the things we're sort of dedicated to is making sure that the Features in PHP in the future in hip-hop don't differentiate too much. So you can actually use either We plan to add multi-threading support through a asynchronous function execution So you can execute multiple functions at the same time and eventually support for Apache will come in probably via fast CGI and Hopefully from that point on we start to encourage people other outside of Facebook to sort of use hip-hop We're sort of eager to create a community around that Yeah And so we announced hip-hop a little bit earlier this week We've been working on continuing to release it even a few hours before we got here So definitely look for that on Facebook's github account as well So we want to take a look at our services layer a little bit and give you a better idea of a few of the different Backend services that we both use and have created to scale different aspects of Facebook And so from a philosophical perspective We sort of think about this question of when do we need a service? So is it something that needs to be really quick? Is it something that Currently has a big overhead in terms of our application layer around a deployment and maintenance Is it yet? Do we want yet another failure point from our infrastructure sort of balancing some of those trade-offs? and we try to go and take advantage of as many bits of common functionality as we can between all these different services and Another reason why hip-hop is really interesting to us in terms of being able to go and reuse logic We've already written in PHP Compiled into C++ and be able to pull that into our back-end services Which are written in a variety of languages and so this is another piece from a philosophy perspective Is that we don't go and choose a single language when we're going and building out Facebook infrastructure our entire web server? Infrastructures PHP, but we use many different languages from a back-end infrastructure perspective Depending on what we're building depending on who's involved depending on when and some of the different Characteristics of these languages for what we're doing So to communicate between all the different services that we have we use this technology called thrift and thrift is something that we open sourced and It's essentially an RPC server That will generate It's an RPC server and you provide it with a Damage-agnostic RPC server So you provide it with a file of your thrift service and it's able to generate code for you in C++, PHP, Erlang, Python, Java, Ruby So it generates all of these for you and it also deals with the communication between each of the services as well as the actual structure So the structure of the communication So it'll do the serializing and the unsealizing of the data to pass it along the wire So to speak and the wire protocols that it supports is like socket the file Transport and then just standard in memory buffers So you can write something in C++ Serialize it with thrift and then talk to PHP thrift server and then it'll unsealize for you so it's the sort of how we pass data amongst our different services at Facebook and Thrift is actually in the Apache incubator So anyway can go download it and use it and that's one of the things that we gave back to The open source community. I'm not exactly Well, and thrift is also using a variety of different projects both I think companies like recapture making use of it as well as other projects So like thrift is used by some of some bits of Hadoop as well So one of the other services that we want to talk about is logging. How many people here have used syslog? Yeah, we did two And I mean So I think this is one of those themes of like we we always start with What is looking at outside of Facebook? What is either a really great open source solution or what is a good commercial solution? And then ultimately going and building on top of it And so we were using syslog for logging all of our data and as you saw behind me The logging server sort of exploded Because we were going and logging so much information from all these page views at this point I think we do about 400 billion page views per month on Facebook And we log about 25 terabytes a day So we went and created scribe and the basic idea behind scribe is that we're able to go and sort of break up this funnel from a logging perspective So you'll have this information coming from our web servers or coming from other back-end services That will get routed into scribe servers which sort of work at funneling it down and condensing it And then we ultimately go and store this information Inside of our Hadoop and Hive cluster so we can go and look at it from an analysis perspective later on and so scribe is one of those technologies where Logging is a common problem. Everyone goes and it has to have logging inside of any site that they build And we got to the point where we just weren't able to work within the scale of some of the other solutions that were there So we also open source scribe. It's also being used by Twitter right now and they were contributing back to the project And I think it's actually one of our projects that's sort of the least known about but Compared to sort of what it does and how much of a common problem it is So one of the more popular things to do on Facebook is actually to upload photos and At the moment, there's 40 billion photos that have been actually uploaded to Facebook And we store that in four different resolutions for each of the areas So there's the thumbnail for the wall then there's a thumbnail within the actual Gallery overview, then there's the actual photo that you look at And there's a fourth one as well because Original larger sizes kept so we store four different photos for different variants for every single photo And we have to and we serve that That's 1.2 million photos or second are served So that's how many photos we're serving at every second people browse around the site because you've seen the Facebook home page It's pretty a photo intensive Okay And one of the problems that are how you would normally try and scale photos a lot on a site It's so large would be sort of like an NFS share So you set up a central NFS share and then multiple servers would then read and write to that NFS share And you'd use HTTP just to serve the photos directly from NFS And the first thing we actually did was a commercial solution because you have to choose the battles that you're going to fight with and What will take more time to do so we went for the easiest solution was to get a commercial Vendor to come in get them set up NFS and use their technology But unfortunately it just didn't scale and the reason it didn't scale wasn't because the commercial technology was bad We just had so much IO We're just trying to read so much and write so much that it just simply didn't work So we had to look for another solution And so then we started optimizing so I mean we went and broke up in terms of where we're caching images so make sure that images that are more recent that are Smaller that we're being able to serve those out of a cache versus having to go back to disk every time Starting to use content delivery networks to go and spread across this load even pull some of that caching outside of our own infrastructure As well as then did some actual caching around the NFS infrastructure as well trying to remove some more of that metadata But fundamentally part of the challenge that we were running into was just looking at how an operating system stores information on disk So in order to to serve one of these files We were having to look up about three pieces of metadata for every file So you had to go and look at the directory I know the data inside of that then the file I know and then actually get at the data itself And so what we wanted to try to do was go and do this in one disk seek And so we developed a system called haystack And what it does is it allows us for any photo that we're going and looking up to go and serve it in In terms of one physical read on the disk So it doesn't matter from a random data access perspective It's always a consistent time of one physical read in order to serve a file Overall it takes about 300 megabytes of RAM to store this index for every terabyte of photo information that we store And so you're able to sort of see how this compares to some of the other solutions We started at a system which was about 10 physical disks to serve a photo We got down to three and then haystack ultimately got us to one And I think one of the other really interesting parts parts of a technology like haystack And this is true for a lot of our infrastructure is that it was built by three people So it's really one of those which is interesting for us in terms of being able to go and have an engineering culture Where small teams are really able to have a large impact or able to go and ship things And so haystack is a technology which isn't currently open source But we're working on open sourcing it because it's one of those that we really think is useful to all sorts of different sites From a variety of sizes and not just for Facebook as well So like talk another bit about one of part of our infrastructure and that's hive Which builds on top of Hadoop, so I'll talk a little bit about Hadoop first This is some of the Hadoop said Hadoop is map reduce So the first thing here that this example runs at the top is the map query Which looks for any key that's greater than a hundred and just outputs whatever that key was And then next the example is the actual reducer which looks for the count of unique items that were in the output from the map script And then actually prints the values like the key then the count now the problem with Hadoop is It's just not simple to use It's very hard to understand and we come up with another solution for that which would be hive and Hive is a very much an SQL syntax and what it does is it builds on top of oh, you can't quite see that there Hmm The people here can't on the left. Okay, you can stand up You can see that hive is very much an SQL syntax So you can just select key count from Val were key greater than a hundred and group by the key and in the background That actually translate it into Hadoop query and run it on your map reduce request and then run it on your behalf So hive is actually in the Apache incubation actually an Apache project now we open source that again put it into the incubator and Apache picked up and The support and development is actually happening outside of Facebook now, but we're still one of the main contributors to it so yeah And the interesting thing about Hadoop is Hadoop actually came from Yahoo So and that actually came from papers that Google would put out So the interesting thing here is that lots of different corporations are building the top of each other's open source technology And that's something we're kind of big on we want to make sure we can open source the things that we are finding useful within Facebook And I think like we really love Hadoop in terms of like what it's able to let us do being able is Scott said to go and build Infrastructure on top of it But I think some of it when you go back to like that syntax bit is just looking at The who's using it and so one of the things that we were truly trying to do was make Hadoop much more accessible inside of the company so that more people were able to go and work with it and Sort of use hive as a layer on top of the underlying Hadoop infrastructure as well And so this is sort of what our data flow architecture looks like This relates back to what we were talking about with scribe a bit in terms of we have our web servers going and sending data into scribe Going and figuring out where to file that putting that into our hive and Hadoop cluster Replicating that and allowing people to go and run jobs on top of this And so you also saw a picture in the previous slide which might look familiar and so this is what our hive and Hadoop cluster looks like for that But I mean overall we have a lot of information going into this cluster We couldn't have done it without Hadoop coming before us And I think that's really one of these important points from an open-source perspective is being able to go and build on top of software that others have released going and Changing in and being able to make it solve your use cases a little bit better going and releasing it again for others to continue to build on to Make better to innovate together And really with the idea with hive was going and simplifying Hadoop because we wanted people other than engineers to start working with it At this point about 250 people throughout the company use hive every month We're running about 7,500 jobs Every single day on that and a lot of people who aren't engineers are actually doing this So you have we have data analysts all throughout the company in different parts Which are going and taking advantage of hive syntax as well as some other web tools on top of it To really access this information which Hadoop is going and crunching for us So I think this is one of those really great cases of you have a technology like Hadoop, which is awesome It's a great piece of infrastructure and then being able to go and use Open source to continue to build another layer on top of that which makes it more accessible to more people So one of the other pieces of the infrastructure that we want to talk about is memcache I'm guessing if I ask how many people here have used memcache They will be about the same as syslog if not. Yep, exactly. So like memcache is awesome. We love memcache It was one of those technologies where when you go back and start thinking about how was Facebook built from the beginning I mean it was built on top of Linux and Apache and my sequel and PHP And then if we could we would add like one more M after that So it would be like the Lampum stack with memcache in there in terms of it's one of those technologies That's really robust. It's scalable. It lets you get a lot more out of your site If you're going and using it in a smart way So overall, I mean memcache was developed by Brad Fitzpatrick at danga originally for live journals being used by just about any Dynamic site that you can think about And it's something that gives us a lot of performance benefit But you it's also up to our engineers in terms of making sure that they use it and that they use it in a Smart manner, so we'll talk about that a little bit more So this is a slide which I think was originally created to just try to describe caches to a non-technical audience And so it's sort of interesting from that perspective But I think it's a little bit more interesting. We currently serve about 120 million memcache requests every second So we're incredibly reliant on memcache in order to make Facebook work And so if you go and think about that from how these different pieces of the stack come together Which Scott will talk about a little in a little bit memcache is really a critical piece of it And we've done some performance work around it as well One of the good examples of how we used memcache and how we how we do use it was when we launched usernames I think this was a little bit over a year ago I think there were about 200 million active users at the time on Facebook and We were trying to figure out what would be a fair way to go and give everyone a username So we thought about a few different options and ultimately we decided that the most fair way to do this would be just whoever Got there first So we went and asked 200 million people to go and access Facebook at exactly the same time So I think a lot of people probably are thinking of this as like a denial of service attack that you bring upon yourself But for us, it was really a product launch and So the team got together they tweak things they tune things and we served and it gave away over 200,000 usernames in the first three minutes and a million usernames were assigned in that first hour And so really that graph before was going and showing our memcache Hits in terms of leading up to this release and then at that point as well. So memcache is an incredibly important piece of technology for us and something that has a huge impact in terms of how we go and scale Memcached itself is generally quite robust. It's filled with Really nice features, but we wanted to make it better So a couple of things that we did with the memcache at Facebook is we made it 64 bit port Because most of our memcache machines have about 16 gigs of RAM and before we used to have to actually run three men For memcache processes so we could actually allocate all of the memory because it was only 32 bit port So what we did is reported to 64 bit and we open source that and that's now in the main memcache That you can get from most distributions We added a multi-threading so we could actually utilize all the cores that were on the processors It's all the cores and processors around the machine so we could fetch data and get it out there quicker and One of the more interesting things was actually adding UDP support Now when you add UDP support It means we actually have to do packet assembly on the memcache client rather than relying on the network stack to do And that was mainly because there was so much memory getting allocated to the buffers within the kernel that Was using more memory on the machine that we'd like to so we changed it to UDP and then we adjusted the memcache client So it could assemble UDP packets itself At the moment we haven't got around we released About a year ago, maybe a bit more we released all the changes we had up until that point and some of them have made it back into the actual memcache server and sometime this year we're gonna do another release of the memcache server and Hopefully they'll get merged back into the upstream version again Yeah, and I mean memcache is one of those examples where it works really great In terms of core memcache for most site that you go and build and there are actually some of these things We've gone and done which won't be as useful at a smaller scale And we're going to making some of those trade-offs between performance and features as well But overall like memcache is I think another one of those examples of it Incredible open-source technology that came before Anyone that came before Facebook that we were able to go and really take advantage of as well as then contribute back to and evolve So next I'm going to look a little bit at the database part of Facebook and Here at Facebook we use my SQL my SQL is our Database of choice and mainly because it's the share nothing architecture We have lots of master master databases that replicate themselves around and you can sort of think of this is like If one an army and if one sort of soldier loses its head like we have an example here at the top of the photo Then it doesn't matter because it'll keep going forward and at Facebook If Facebook it doesn't really matter if a single server disappears. We have other servers that will back it up Well, and those are that's really one of those choices that you make when you're thinking about how you scale is Do you go and invest in technologies? We're sort of like grow One piece of your infrastructure really large or do you go and find a way where you're able to sort of break that up? And I think that's really what we've done with our databases with my SQL in terms of being able to go and have a lot of More independent database clusters that we're able to continue growing that entire pool rather than having one really large database cluster by itself And overall, I mean we really like my SQL We found it to be a simple and fast and reliable and normally when we go and break our databases It's because one of us did something stupid not because the software wasn't working right And so we I mean we really think about my SQL is like that freight train Which is like really solid it keeps on running and a lot of that sort of is also based on how we go and use our databases In terms of we don't take advantage of a lot of the relational aspects. We don't go and use joins inside of our database but in but in general we found my sequel to be an incredibly reliable piece of Storage infrastructure and definitely is a good example of what you can go and build from an open source perspective say the No, it's gonna say the database that we actually use the NODB engine at engine because it's only one It's got full transactional support and we can make sure everything makes its way to disk but this is overall Okay, so this is sort of the high-level look of how the architecture is at Facebook and we'll look at databases again So most people see databases is sort of a secondary and a collection of second indices. That's where you got all your relational features That's where you do your joins But here it we more or less use it as a way to keep persistent data It's where we store the data that writes at the disk That's our persistence layer And if we want to do any sort of joins then the joins are actually done within the web servers And that's why hip-hop is important because we do a lot of processing there and reassembling the data from its different sources And then once we have that sort of put into memcache And memcache can be seen as a sort of our second indexes our secondary index. That's where our data is stored So really it looks a little more like this Yeah, so that's probably a more true representation of our stack where the web server looks at our distributed index to see if The data is there and if that fails it will look in their persistent storage But it's more than just their web servers that look within both memcache and my SQL We also have other search services that themselves will look in the various data locations Yeah, and so hopefully we gave you sort of a pretty good overview of a few of the different services and Technologies that go into scaling Facebook Both in terms of open-source software which we use some of the things that we've built on top of that as well And so that was really our goal in terms of going and looking at a few of these different places of our architecture and really How important open-source software is to go and scale Facebook since the beginning of the site I think one of the other things and we talked about this a little bit with haystack was that a lot of this actually starts as a hack project So it's not necessarily that we're going and set out trying to solve This specific problem and have a lot of people thinking about it and how to do it We're rather a few people will just like sit down or a single person. I mean haystack was started by one engineer high ping Or sorry, that was hip-hop haystack was started by Jason Sobel And it's really getting to the point where we prototype something and we're also from an infrastructure perspective When we're trying to solve a problem We might go and try two or three different things at the same time Prototype them out for a few months and then throw two of them away and move forward with the third one And I think that's something that's really important And it's sort of a piece of our culture as well from an engineering and company perspective in terms of really How these technologies help us go and do things like move fast And so that's a really important piece of hip-hop in terms of it allows us from an engineering perspective They go and make changes for with the development speed of PHP being able to go and sort of save something and refresh that immediately On your on your development server But then going and having a lot of the operational performance of a compiled language like C++ Being able to go and develop technology sort of in a small team and actually have them scale out to the entire site Releases open source and really that idea of continuing to be bold and innovating it goes into our engineering culture as well So everything that we talked about you can go and find at facebook.com slash open source We also have a variety of developer tools there So libraries such as 320 which are a lot of the UI aspects and some of the data Aspects beyond behind our iPhone app So let's you go and do things such as like have a list that which you scroll through which is actually going and pulling images from Server instead of having all that information being stored locally as well as tools Like tornado which is another preface of infrastructure which powers friend feeds web server It's designed to be a really like high-speed web server because they have a connection to open for every user That's actually on the side of the time As well as other development tools things like PHP shell for going and doing autocomplete and stuff like that So overall We're really excited to be here and have a little bit of time for questions as well as a bit of a pile of swag down at the front We want to come and grab any so thank you Yeah, my question is about he hope Why are you translating PHP to C++ instead of directly developing C++? So what was the last bit sorry? Yeah, why aren't you not developing their website in C++ directly? instead of developing PHP then Translating the C++ optimizing and doing a lot of stuff. Okay. The question was why are we just developing in straight C++? Why aren't we okay? So the current code base at Facebook is about four million lines of code So the first problem we'd four million lines of PHP codes So the first problem would be how do we translate all of that into C++ as well as allowing people to keep developing with the site We can't exactly hold up development while that's being done And we it comes back to how long does it take a C++ C++ project to compile We like the fact that we can move fast people can make little changes save it and then look onto the web server and the changes right there Yeah, so a lot of it is just that development speed of being able to like More people know PHP. It's easier to work inside of PHP. So we really want to keep it as a language from a development perspective Hi Thanks for the presentation What eventual consistency do you Tolerate So how much is so consistency do we tolerate? So, I mean we tried to I Don't know the exact answer to that question I mean so like we really try it for it I mean anything that you that you post any comment that you comment on like as soon as it comes in We record it we make sure that it we go and serve it again I think with services like newsfeed that you have the two views now So you have one which is going and showing that most recent view everything that's come in But then you also have a view which goes and sort of looks at what do we think is the most relevant thing for you to see right now? So I actually think it's probably a little bit less from a data storage perspective and more from a presentation perspective in terms of What specific feature on the side are you going and looking at and how are we going and optimizing that at the time? You guys were talking about You chose to put the joins and all the filters in a website in a web server level I suspect you guys did a lot of benchmarking and checks at what point that would be an advantage by doing that because for smaller Sides it probably won't be do you have figures for that? Can you distribute them because those would be really interesting to see and at what point it would finally be an advantage to do it? Yeah, so in terms of like at what point do different pieces of these technologies make sense Yeah, I don't think we necessarily have a lot of benchmarks in terms of something like do you do joins in your database? Or do you go just pull data together and join it together at your application level at your application layer? Some of that tolerance is also probably depending on how large do you want to scale your database cluster? And where are you wanting to spend some of that CPU time? In other question probably Date is very crucial for Facebook so how do you prevent the loss of the data and Probably you use the mechanism like duplicate and etc and the second question here about the data Did you have already the problems with it and how do you solve them if you had? So I think your question your first question was about replication and then what was it second? The second was if you if you had some data losses In the past and if you had how did you solve it and to know to have it in the future? I mean I think from it like a data loss perspective a lot of those Things that you learn about how do you go and run database clusters apply the same for us as well in terms of going and making sure that You're having backups that your replication is working that you're keeping your bin logs so that you go and can replay some of that information So a lot of I think the scaling Lessons that you learn at least in terms of how you go and run databases are pretty similar Whether you're at Facebook scale or smaller than that as well It's really a lot of those best practices applied at a much larger scale Hi, you've talked about scaling at technology level. How do you scale? Internally with your developers and you've got fairly high velocity releases. So how do you ensure all your developers on a consistent? Frame of reference when they're developing new features. Yeah, so really Scaling is more than technology as you said. It's also about scaling an organization scaling an engineering team And so we've developed I mean we've developed a variety of internal tools which also help us do that So we've gone and layered things on top of our Version control repositories So have tools to go and do things like code reviews to be able to go and trace a code review all the way back from that Release that we're doing to when was the code committed who reviewed it? What was happening inside of that every single commit at Facebook is reviewed by someone else? And so it's definitely one of those challenges that We focus on as well as in terms of how do we continue growing and moving at this pace Hello Yeah over here Excellent presentation by the way love it Really interesting in haystack So I guess the average size of a photo how big is that like a few hundred K or less Yeah, I mean I think an average size is certainly under 500 K Generally the size the larger size that we store is 640 by 480 and what we actually do is our photo uploader So when it's occurring on your computer is going and resizing that photo before it's even sent over the network But my question is then how will the could this scale for I don't know say or vorbis standard for Megabyte file or even movies or how do you how many discreeds per? Megabyte sort of so I think one of the things that to keep in mind there is that that Percentage in terms of the amount of time you spend going reading metadata compared to the file size is really important And so when you have a smaller file then the amount of time that you spend going and looking for that metadata It's actually much larger if you go and start to get into something like videos Then it becomes a much smaller Percentage of the time that you spend overall so we don't actually use haystack to store videos But only for for photos right now Yeah Yes Up here up here Okay, I was wondering can you give us some numbers on the breakdown of how many servers or web servers? How many servers are memcache and how many servers or The database No, I actually can't but I can tell you we have tens of thousands of servers overall What language is? It's hip-hop written in and what's your plans for growing a community around it? What was the second that's written in C++ and what was the second question and what's your plans for growing a community around hip-hop? What's our plans for creating a community around it? At the moment, we've open sourced it. We haven't released the code yet That's gonna be happening soon a couple of hours before this I was finishing the last of that with C make and Making sure it could build on non-facebook platforms We've created a mailing list and we've had a couple of thousand people joined the mailing list already Is that right? Yeah, there's about a couple of thousand people joined the mailing list already And we're hoping as people start using it and the community will just naturally develop around it You have to remember that hip-hop isn't designed for everyone to use It's they can if they want but it doesn't really make sense for the small person running their own shared wordpress blog It's more for those larger sites So we're hoping that they'll realize performance benefits that come from it and they'll start adopting it And from there, I'll just hopefully just snowball Yep. Hi. Yeah Thanks for the talk. What about HTTP servers? Do you work with the patch engines or? multiple So we've generally been using Apache in PHP But hip-hop actually has its own embedded web server, which is sort of really simple web server built on top of Lib event and so we're now been moving to using that How hard it is to port a Zend extension to hip-hop Hi To change an extension to hip-hop to basically just write a C++ class as normal And then it's got the similar parts of the PHP extensions initialization function There's a request initialization request destroy and request sorry module unload and Again, it's some more glue code with macros to expose your C++ class back into hip-hop So it's similar to the PHP extensions, but less random macros that might not understand Hi Thank you for the presentation first of all and my question was Where exactly do you use Erlang and do you make use of Indonesia and the concurrency that the Erlang provides? So we use Erlang for our chat service So for Facebook chat It's your generated C++ code using a boost library and what libraries do you use in the generated C++ code? Yeah, we missed that completely. Sorry. We couldn't understand. We couldn't hear you Is your generate C++ code using the boost library? yes, our generating C++ code uses the best libraries it uses file system system and A couple of the mutexing and the concurrent execution the threading ones If you have more questions come down and ask afterwards Hello, thank you for the presentation, but when will we get a dislike button? What when will we get what a dislike button? Are we let's talk about that later? Let me know if there's specific information you want off of them. Okay. I have another question about hip-hop Is it currently possible to cross compile it for example to get a windows binary? And if that's not possible would that be easy to implement that function? at the moment it only compiles on Linux Specifically last week only compiled on CentOS We've now got it to the point that should compile on all Linux distributions It uses CMake and CMake has support for cross compilation At the moment, there's no support on OS X or Windows But we're hoping because something from Microsoft is contacted us and hopefully they'll contribute the Windows part and Hopefully something the community will either contribute the Mac port or I myself will do it Hi, my understanding is that you only have data centers in the US, right? No Are you asking where servers are now? My question is how do you deal with the high latency from the US for users that are outside of the US? So dealing with latency So I mean in the US we have data centers on both the West Coast and the East Coast We also use CDNs to help distribute some of the like things like Images and CSS and JavaScript around the world I think that's been something that our operations team continues to work on as this site continues to grow internationally as well So there's still five minutes left are there any more questions and we'll stick around as well if you want to come ask down here We might be able to hear you better too Yeah, hi, you mentioned that you use CDNs. Do you use other people's CDNs or do you have your own CDN deployed around the world? So we use Akamai for a lot of our CDNs Hi, I was wondering what kind of open-source software you guys use for Monitoring capacity planning and trending and if you experienced issues scaling with those as well as the Facebook application Yeah, we know a little bit less about the monitoring technologies that we use I know that we use ganglia. I think we also use cacti We also have some tools that we've developed internally in terms of managing our overall infrastructure But I honestly know a little bit less about the scaling challenges that we've run into those You talked that you're using revision control systems for Fastly developing your site, but what kind of software are you using? Are you using only single revision control system or different teams with different revision control systems? So I think you were asking about like what software our developers use I mean in what so yeah, I mean We have a variety of like we we use then for a lot of our development environments Engineers then use I mean emacs or vi or whatever insert favorite text editor here And then we'll have some other tools built over on top of subversion and get our main repositories or subversion Over half of our engineering team now uses get on top of that And then we have some tools built on top around that in terms of things like code review and documentation and things like that Hi there, can you say a little bit about Cassandra and what using Oracle for please? Thank you So Cassandra. Yeah, so we do a team at Facebook developed Cassandra a few years ago It's also in the patchy incubator right now we use it for inbox search on Facebook and We haven't really been developing it actively in the past year But at this point I think dig in Twitter and Rackspace are really the largest contributors to Cassandra inside of the patchy project And so I mean I think from our perspective It's really great to continue to see it develop and have a community around it That's going and continuing to push it forward even though we're not actively developing it as well How do you manage the deployments of software? So software deployments. I mean we have a variety of tools built around that as well In terms of being able to go and push out information to different hosts going and running jobs on those hosts distributing information and And I mean I think that's something that we'll talk about a little bit more this summer Hopefully at the velocity conference Are there any more questions? Okay, I think not so thank you very much for your talk