Clang MapReduce -- Automatic C++ Refactoring at Google Scale

Google has over 100 million lines of code, and our biggest programming language is C++. We have a single, shared codebase developed primarily on mainline. We build every binary and all of its libraries from scratch every time, allowing us to incrementally evolve APIs and libraries over time. The entire development process is extremely incremental in nature, and even API-breaking changes are a regular occurrence. However, for core libraries used throughout the codebase, this development model is a huge challenge: how do we incrementally evolve an API in use by tens of thousands of other libraries? The answer is to use Clang to automatically refactor APIs and their users across the codebase. How do we scale Clang up to possibly the single largest unified codebase in the world? The same way Google scales anything else: MapReduce. By coupling Clang's library design and architecture to existing Google infrastructure we can automatically compile, analyze, and refactor the entire Google codebase in minutes. In this talk, I will dive into the challenges of refactoring C++ code, how we're using Clang and making it even better at solving them, and how we scale these solutions to the size of our codebase.

• Category

  Science & Technology

[MrPointless]

Thanks!

[AsianBorat]

Check BadVoidShip's comment. He didn't click "reply", so you probably didn't get a notification.

[BadVoidShip]

It's already out. There's another newer talk from May 2012 video id: yuIOGfcOH0k at 36min in he gives a link to the svn branch (although by now it might be merged into clang's main).

[MrPointless]

So... did it ever release?

[alim80]

Great talk, can't wait to see if a tool like that is ever released!

[otherwiseguy]

Is there any place that we can watch for when this tool is released?

[PixelAliensWorld]

Really interesting :)

[Manuel Klimek]

JuliaLLawall: we actually looked at Coccinelle before implementing the language, and found that it was not sufficient nor easy enough to extend for what we wanted to do in C++. Unfortunately C++ is a completely different beast from C - there are a lot more corner cases and little hidden things that are really important to match on.

[JuliaLLawall]

It looks similar to what Coccinelle does, although Coccinelle provides pattern matching on the concrete syntax and allows the transformation specification to be integrated directly into the pattern specification, using a patch-like notation. It doesn't seem possible to provide an example on the youtube comment page, but there are examples on the web site. Currently, however, Coccinelle only supports C code, with a very minimal treatment of C++.