 Good morning everyone. Welcome to OGG B3. Our first presentation for today is by Diego Briran and he'll be presenting all your multimedia, I belong to us. Please make him feel welcome. Thanks for the warm welcome. I will be speaking about LibAV and its twin sibling FFMPEG which are basically the core engines behind most of the world's multimedia processing, hence my title, all your multimedia are belong to us. So who am I? I started getting interested in free software around the end of the 90s and yeah, when quickly went down from then I started working on Mozilla, then on mPlayer, FFMPEG and I've been with LibAV since around 2011. So I've been in the multimedia free software scene for more than 10 years now. So I'm going to give you a technical overview about what LibAV is and its constitution libraries. Then we're going to look at legal issues that we had to face over the years because multimedia is known to be a legal minefield. Then I'm going to talk about the split that happened in 2011 between LibAV and FFMPEG and then we're going to take it to questions. So what is this LibAV FFMPEG thing all about? FFMPEG was started by a French man named Fabrice Bayard in the year 2000. It's a collection of libraries and programs for all things multimedia. It contains a bunch of command line utilities, a git repository weighs in at about 600,000 lines of code by now and the whole thing is licensed under the LGPL with some GPL video filters. And by the way, the FF stands for fast forward. So that's a little bit of trivia for you. What does the LibAV community look like? It's mostly made up of by volunteers, so it's very much unlike the Colonel where most of the people are professionals. There are some people that got hired or paid to do some work on LibAV or back then FFMPEG, but mostly are volunteers. But what binds us together is that we, for example, we dislike C++, we like our C without the plus plus. We dislike non-C99 compliant compilers and auto tools. So that's why we didn't support compiling with MSVC for the language time, and that's also why we have created our own build system about which I will talk later. What we do like is speed. Multimedia processing is something that's very speed critical even today and probably will in the future. And so we do assembly optimizations and things like that. We try to be standard compliance. For us, that means mostly C99 and POSIX, and we also try to be portable. So we run on all sorts of different operating systems. You can compile us with different tool chains, et cetera. But for us, portability means that we program to POSIX and then expect the platforms to catch up to that with some exceptions. This hasn't made us very popular with some BST portals, but over the years, it has proven fruitful because we are very portable now and we didn't have to hack to litter our code with compatibility hacks. In the multimedia community, it's notorious for having a lot of abrasive characters and flames and things like that. And yeah, that's not to be denied. It's gotten much better over the last few years, but clearly it's probably similar to the Linux kernel or maybe in the past it was even worse. What we also have is Trolldriven development, since most people have volunteers. It's hard to motivate them, but people still are proud and competitive. And so if you, for example, want to have some decoder speed improved, maybe you can find some Windows binary that does the decoding faster and show that to the developer that knows about that decoder. And maybe he will feel motivated and get trolled into actually implementing that. So our goal is total world domination. For the free world, we've basically already achieved that. So almost all free software multimedia is in some way based on one of our libraries, who all know VLC, G-streamer, employer, et cetera. But also in the proprietary world, since we have GPL, if you look at some shareware multimedia conversion tools, download them randomly or so and check which DLLs they use, you'll probably find our libraries. But our most important services, our most important users are probably web services like Vimeo, YouTube and Facebook that at this point probably handle most of the world's multimedia files. And they do use our libraries to convert whatever the users upload into the formats that they then present on the web. So this again is why I have chosen this title. So what do we have? We have three command line tools, AVcon, which stands for when we convert. This is a general purpose multimedia transcoding and transformation tool. So you can apply all sorts of transformations to multimedia files. We have a minimal SDL-based player that we use for development purposes, so it's not a comfortable full-featured player, but just something that you can use to quickly view files and see if they have artifacts or similar. And then we have AVPROPE, which is an analyzer tool that prints information about multimedia files. Then we have a set of libraries. I'm going to say a few words about each of them in turn. LibAVUtil contains our helper code, things like CPU detection, math, logging, things like that, general purpose data structures and data types. LibAVU format deals with containers and network protocols, which is probably all heard of AVI and MP4, but there are very many other things that are supported. And network protocols, of course, only the multimedia-related ones like RTP, RDSP, HTTP, ALS, that stuff. And yes, also Gopher. I don't see much point in that. I tried and failed to win consensus to get that removed, so we still have Gopher support. So then there's LibAVU device for dealing with hardware input devices like ALSA, X11, beautiful Linux, things like that. We have LibSW scale, which is a big eldritch horror that does color conversion and scaling, and it is a big blob of legacy ugly code. And there's LibAVU filter, which has video and audio filters to do things like deinterlay, scale, crop, video, apply effects, or maybe do channel and volume parameters to audio and also apply all sorts of effects. And there's LibAVU resample for audio resample and channel rematrix, things like that. And then there is LibAVU codec. So this is the big piece in a project. It contains a lot of decoders and encoders, interfaces to hardware acceleration like VDPIU, VAAPI, several others, support for subtitle formats like SRT, ASS, that's ASS, not what you think. And it's basically the project's count jewels. So it contains about 130 audio codecs, so decoders in this case, and about 180 video decoders. So the ones that you will recognize are things like MP3, ASC, VORBIS, media audio for audio and things like H264, MPEG4, MPEG12, Windows Media Video, etc. What we also support are lots of fringe formats. So in the 90s or early 2000s, practically everybody that had to deal with multimedia content in some way came up with their own format. And so there was a whole zoo of very specific fringe formats that were used, for example, in the gaming industry only in full motion video cutscenes for some games. But since some of the whole developers are gamers, gamers and like reverse engineering, they've implemented support for all these fringe and special formats in LibEvi codec, in LibEvi format as well. What we also have is architecture-specific optimizations to make things run fast, mostly for x86 ARM and PowerPC. So we have a Cindi assembly for 32 or 64-bit variants of these processors. So we generally have core routines that exist in C and then some optimized variants that are written in assembly. And the way it works at initialization time, we check for the CPU capabilities and the type of CPU, and then assign function pointers to the most optimized function that we are able to use on that CPU. And then at run time, the code is called or the optimized function is called the function pointer. What we also have, unfortunately, is a lot of legacy spaghetti code. So in the old days, the FFMPEG back then codebase was very monolithic. The MPEG, everything that deals with MPEG variants was all one big beast. That's the MPEG end context structure that is 308 lines still, and that is without comments, empty lines or defines or anything. But we are also working on reverse engineering, for example, with another big beast called DSPutil that we've managed to eliminate last year. Then there's, as I mentioned, lib SW scale, and we have things like that LibEvi format depends on LibEvi codec, which is not ideal because, for example, if you have to deal with hardware codecs, then maybe you want to use just LibEvi format and don't have to deal with LibEvi codec at all, and LibEvi codec is really big. So yeah, security is a topic that's come up a lot in the last few years, and well, our security track record is not exactly stellar. This has several reasons. The first is that we simply present a lot of attack surface because most of what LibEvi format and LibEvi codec do is input handling. Most programs have a small piece that deals with the input, and then they have some large piece of logic behind it. For us, it's different. Basically, every decoder, every demuxer in LibEvi format or LibEvi codec is pure input. So there's lots of opportunities to make mistakes or overlook corner cases, etc. But on top of that, only recently have we switched to development practices that really care about error checking and these kind of things. So in the old days, speed was often favored over correctness. And if you go back 10 or 15 years, you maybe remember that it was really hard to get full motion video to play at all on computers of the time. But maybe it was a bad trade-off even then. We also use automatic tools to fix all sorts of bugs, like automatic fuzzers and covariate. But really, if you want to look for bugs, then you can mostly grab around and you will still find a lot of things. But if our hearts remain pure, maybe we'll see the day that all our decoders and demuxers are robust. So the future. What will the future bring? In a way, so we are stabilizing and entering a phase of maturity because the multimedia world is moving away from this zoo of formats that you had in the 90s and early 2000s. And so, hardware codecs are becoming more important and we are focusing on refactoring and stability work. But still, for example, in this year, we will probably see DTHHD and go-to-medium codecs added to the delivery codec. Yeah, if fishes were fishes, then I would cast a net for implementation of the Cineform codec, which is what the video codec used by GoPro cameras, a replacement for the Libase WSK beast, new hardware acceleration framework and some paint-paintainers that help us out because we are still mostly volunteers. I mentioned that we have a custom build system. It consists of a configure script written in POSIX shell and some GNU make files. It does dependencies, cross compilation, you can switch between different tool chains, everything that you would expect from a build system. It's fast, so you don't have to wait one minute in order for it to tell you that nothing has to be recompiled. And it does actually work, unless other than most other custom build systems, which are mostly a bunch of non-working things. This one does actually work to the point that I'm long-taught with the idea of making a standalone project, but there's always the thing that it takes time and there's always so many other things to do. Another piece of infrastructure that we have built is FATE. FATE's first feature is that it has a recursive acronym as name. FATE stands for FATE Automatic Testing Environment. At the moment we have around 2,000 individual tests in our codebase. Some of these are unit tests, but mostly it's actually full system tests. So, for example, we would do a thing like, decode a video sample from our test suite, convert each or generate a checksum for each frame and compare that checksum to the reference that is stored in the codebase. And so if you break the decoding and probably some bits and some colors are going to be switched, you will notice that immediately. So this has been very useful in finding problems and has prevented literally thousands of bugs from entering the codebase. We also have some clients that run regularly and just check out, pull the git and compile and test, compile, run the test and then push the result to a collector, which then aggregates all the results on the web page. And we also have a FATE sandbox called Oracle, because the Oracle will tell you your FATE where you can push stuff and see if it passes the tests on platforms that you don't have at home. This is what it looks like. So the green bar in the middle indicates that most of the FATE stations are currently green, so all tests are passing, some are yellow, some tests are not passing and the red ones are not compiling. So it's eight architectures, 16 variants, 16 operating systems, 26 compilers, 262 configurations done by 11 contributors. So we run our stuff through a lot of different configurations. We also have a foundation of 50133, incorporated in Delaware a few years ago that sponsors development of codecs, optimizations, reverse engineering and costs of infrastructure, things like that. Unfortunately it has its non-profit status rejected last year by the IRS as has happened with many other free software foundations, not the free software foundation, not the free software related foundations. So that's this overview and then we switch to legal issues. As I said at the beginning, multimedia is a legal minefield. There are many entities collecting patent royalties. You may have heard of the Amperec LA and Fraunhofer 1P3. You may have heard of DTS and Dolby AC. There's also a lot of reverse engineering that is being done which is of dubious legality in many countries. There are trademarks and there are also, especially in the past, there were very many murky copyright and licensing situations. So let's look at the issues that we've had to face over the years because clearly we are all doomed and are facing jail time. So in the year 2000, FFMPEC was created and not really popular anymore, same 2001, nothing much happened, 2002, 2003, but in 2004 we already did have some commercial users, but still everything was quiet. Until in 2005, finally, there was legal threat number one and this is Mike Molensen, multimedia Mike celebrating the occasion on his blog. Do you have any idea how long I've been involved in multimedia hacking and reverse engineering? About five years now. All that while, folks have warned me sternly and constantly that this type of work would get me sued to death. I'm pleased to announce that today I received my first legal threat. I feel like my work has finally been validated. So what happened? On to technologies, maybe you know that company because it was recently acquired by Google. On to technologies has been producing a separate line of codecs, VP3, which was the basis for the Aura, VP4, VP5, VP6, VP7, VP8 and sold those codecs to their customers. For example, VP6 was used in Flash, VP7 was used by Skype and at some point in 2005 they made a demo application for VP6 decoding as a Flash applet and presented it on a website. Of course, Flash applets are notoriously easy to reverse engineer. Somebody took that Flash applet, extracted the Java code and converted it into a C++ library for VP6 decoding, which on to did not really like and so they sent those nasty grounds to the people hosting the VP6 library on their websites. So what happened was that all the people took that library down, but a few months later somebody committed a native decoder for VP5 and VP6 to LibEV correct. That was done on a Saturday and I've some years later spoken to a person that then worked for on to now a Google engineer and he told me that on the Monday following that Saturday they got wind about the decoder, but they never complained or did anything. So 2006 quite again, same as 2007 and then 2008 finally something again, Nelly Moser. Nelly Moser is a company that did the ASSAO or also Nelly Moser codec which is a voice codec that was used in Flash. So if you recorded something to your microphone in Flash then it gets compressed with Nelly Moser. So what happened there? Nelly Moser sent us a letter complaining that we were infringing their IP rights. Okay, so I answered them and asked what exactly was the problem and what exactly we were infringing because IP is loosely defined terms or were they talking about trademarks, copyrights, patents, whatever and also which part of a code base it was like half a million lines of code it can't all be infringing. So what exactly were they talking about and I asked them to be slightly more specific. The person promised to get back to talk to the engineers and get back to me and indeed a few weeks later we received another email where they outlined that the problematic parts were Nelly Moser dec.c, Nelly Moser ng.c and Nelly Moser.c. So basically the Nelly Moser decoder, the Nelly Moser encoder and the code shared between the two in Libre V Codec. So I replied again, okay, so now we know which three files we're talking about. So this is much more specific but still we don't know what the issue is. So is it trademarks, is it copyrights, is it patents, is it something else and the opposite? Never bother to follow up. 2009, 2010, 2011, 2012, 2013, there's 2013, 2014. Basically nothing happened. Wait, didn't I skip something there? Yes. In 2010 we received a letter from the MPEC LA, so that's the MPEC Licensing Association and again the tactic that we used was just ignore the letter and it went away. Probably they had just had some new person starting there and he or she was overzealous and found a new target to send a letter to but really nothing ever came out of it. So seriously nothing ever happens? Well, so the point is that yes there are legal issues in the field of multimedia, so this is not all completely made up, but there are some things to consider. For example trademarks are easy to avoid, especially for a backend library like us. We don't, we maybe print some trademarks to the console so we can just disable that and print some other word or whatever. And the other point is that all these these entities that go collect fees for their patents, they don't care if the software is proprietary or free and open source software. So they don't care what kind of software you use but what they do is go after the large commercial users, not after the providers of the software. So if you have deep pockets and you use some sort of multimedia technologies where these entities have patent pools or whatever, then probably somebody will come knocking at your door but if you're just a false project that doesn't have any money and everything that you can get is a bit of bad press. If you try to sue them then nothing would always happen. And just in case there's also what we call the GB defense and it goes like this. So if you receive a letter from some of these entities, you reply, je suis désolé mais je ne parle pas l'anglais. Est-ce que vous pouvez traduire ces lettres aux français si vous plaît? Cordialement, Jean-Baptiste Camp. For the non-French speakers in the audience that means hello I'm sorry but I don't speak English could you be so kind as to translate this letter to French for me please thank you and goodbye. And this also has worked in surprisingly many cases. So in summary don't be a coward just hack. The copyright situation let me say a few words about this. So LibAV is completely free software. It's licensed on the LGPL v2.1 or later. I did a full licensing audit of all files some years ago and we found some non-free pieces but these have since been taken out and replaced and we've also replaced some GPL bits that we had here and there because we wanted to have a simple streamlined licensing situation so that people have fewer possibilities to make mistakes and run afoul of the license. So we come to the third part of my talk. You're probably all aware that sometime in 2011 there was a split in the ffmpeg project. Yeah what happened? Well by the end of 2010 beginning of 2011 generally the atmosphere in the ffmpeg project was not in the best state. So people were unhappy because development was stagnating, patches were stalled, the leadership was dictatorial in a way and that overly centralized and many developers got frustrated and unhappy for very many reasons so that by the beginning of 2011 some people had silently left or stopped contributing as much as they had previously contributed. And this led to these people gathering on IRC and noticing that they were a group of people that basically had the same impression, both the same and this led to the decision to fork the project and so we spoke to more and more other developers and at some point we noticed that actually the unhappy people were not just a few but a large majority of the developer base. So at some point we decided to not just fork the project but replace the old leadership with a new team. So a new Git tree was created, there was a declaration of independence so to speak that was signed by 18 developers which instated the leadership team of seven developers and ousted the old leader Michael Nio Maier. So what was the result of that? Well with the benefit of insight it's easy to see but yeah it's probably best said with a picture. Yeah somewhat unsurprisingly the people that were not part of the crowd that decided to make changes in the project was not exactly happy because they had been left out of the decision or because they were just not, didn't have the same opinion and yeah they were a mix of shocked and angry and this led to them starting a vote on the main development mailing list in which was to be decided if the new team should remain in place or if the project took move back to the old leadership style. Yeah the seven people voted in favor of going back to the old leadership style but what nobody was told was that this vote also contains sort of a call to action after war. Yeah I can speak again. Yeah I'm going to call them the revolutionaries and the loyalists. So the loyalists had received control of the ffmpeg.org domain from Fabrice who was no longer active since maybe 10 years but still maintained the trademarks and control of the of the DNS. They set up a fresh project server and new infrastructure and then changed the DNS to point to that new infrastructure then effectively splitting the project taking the name ffmpeg with them and the revolutionaries kept the old infrastructure moved to the switch the name to libevi and moved to the libevi.org domain. So what has happened since then? So both groups of developers have stayed largely separate they run separate projects with different policies development speed has picked up in general so it's definitely not all been bad. There's a lot of bad blood still but somehow people are also working and working effectively. So how do the projects look like today? libevi is organized as a community of equal peers it has a strict review principle so everything has to be seen and approved by four eyes. You need an approval to push a change to the main repository. There's a code of conduct and the general goal is quality and maintain and long-term maintainability of the code base. You can often find libevi developers at events such as these as well as on irc and mailing lists etc. libevi makes no public statements about ffmpeg or the fork or whatever. ffmpeg is still run by an evil overload so this is not meant as an insult it's an old in-joke and it merges everything from libevi and also from other sources. The mailing list is run more with a free speech kind of style and the main development goal of ffmpeg is to add more features to the code base. Developers communicate mostly by mail also a bit by irc and are only rarely found at events such as these. Unfortunately although this has gotten slightly better at the end of last year there have been very many public insults for libevi and libevi developers from the ffmpeg side. My personal opinion about the situation well there's the Linux conference Australia 2015 code of conduct which says that language which is not appropriate for an all ages audience cannot be used in presentations so I'm not completely liberty to speak here so I would say that the general situation with ffmpeg and libevi is that and also and butterflies, roses, ice cream, yeah so I fear that we have to live with long-lived forks for the time being. I don't see the projects so development style is very different the culture of the project is different some of the people are not very compatible with each other and this if it should heal at all this will still take a long time so before I take it to questions I would say that yeah we always work on patches and contributors so if you're interested in free software multimedia then by all means come talk to me go to a mailing list to irc channels whatever and what we also need is paint maintainers so libevi is notoriously understaffed and it's really a very central piece in the free software stack and it's a pity that there's so many so few funds directed at it so if you work for a company that uses in some way then maybe get in contact and help us fund some developers so questions okay we'll start here you've said that the ffmpeg project merges all your changes what about the other direction how much of the code from ffmpeg do you merge since the split so the libevi to ffmpeg direction is 100 so ffmpeg merges everything and the the other direction so from ffmpeg back to libevi is cherry picked so not not all changes that land in ffmpeg get picked up by libevi next question so two things firstly you were talking about the fact that it's input that's controlled by third parties on the internet most of the content the formats are complicated the buffer overflows are potentially possible and then you talked about a test regime are you running any fuzz testing on the decoders to get some idea of what potential problems there might be that we can find before the third parties do yes yes there's this fuzz testing going on so google also does does fuzz sensing for for libevi and ffmpeg the problem is that there are just too many bugs so it's at the moment the situation is a bit like like starling said for for death so one death is a tragedy and a million deaths is a statistics and for bugs it's a bit similar so if you have one bug then you want to fill it fix it if you have like 1000 bucks then maybe you go see what's on the tv and so it's it's it's a long and tedious process and yeah many of these are surely though a class of bugs yes many are a class of bugs but still takes a long of a long time to fix them all so to fix all just basically going in there and fixing all the mallocs is just checking that all the mallocs are done all the failing mallocs are caught is is something that hasn't been done yet and there's a lot of work and it's not exactly the most fun and rewarding work i believe last year that google um put in about a thousand bug fixes into the code for avian ffmpeg um can you comment on that in terms of did they just put it into both or did they put it into one and then it got merged and also have you had any contact with google in terms of obviously they must have people are looking at the code quite extensively in terms of maybe getting one of them to help as a paid developer on on the project yes they're not helping with paid developers also so we managed to also get access to the the google cluster for samples and results um but um yeah we're not exactly happy with all the fixes that go into ffmpeg um it's yeah we try to to be a bit more thorough in many cases and so it's it's a slow process um but yeah we also find that not all the things that get fixed in ffmpeg actually apply to us two questions one is very quick um does they leave every project receive donations from the public or are there any plans for it uh yes well you can donate through the ffmpeg foundation or through through videolan and things like that um yeah we could get better in this way um finding ways to accept donations it's i think it's not terribly easy right now does that answer your question yes and the second question is slightly more technical you mentioned in your slides that leave every format and leave every codec are um well they have a dependency in there yeah have you also thought about maybe uh all the legacy codecs that are probably not going to be used in the future um to be pushed to some uh different library that you can leave out a compile time if you so choose this um so the the build system is is fully um i say fully uh modular and configure and configurable so at configure time you can select so you can turn on and turn off all components so you can build uh the baby exactly to your liking so you can just um compile in two or three codecs or you can just turn off two or three codecs just as you want so but we're not going to to remove the decoders for for these formats so because leave every codec or leave every is also about dealing with these these legacy formats and um probably preserving a bit of of history so in in many years these files will still exist and we will still have still have a way to to play those files from the 1990s so it's kind of a preservation of multimedia heritage what do you see is the next big thing for the baby the next big thing for the baby well we'll see what the general in what direction the general multimedia world moves so um things that we added recently is a native hgbc decoder and a vp9 decoder if you've seen the talk about dala um hopefully we'll see some movement in that direction and maybe dala will be able to compete with the hgbc and vp9 at some point in the not so distant future i believe it's uh yeah there's going to be much more importance for for hardware accelerated codecs things like that um so unfortunately we've reached the end of the session but just before we close um Diego we have a small gift to say thank you for your presentation today so thank you very much