 I'm Jeremy Bennett. This is a bit of an experimental first for the LLVM rem. It's a panel session and we have one throat microphone. It's intended to stimulate a discussion. Quick show of hands, how many people here have worked or are working on LLVM? I'd include in that anything in the LLVM toolchain. How many people here are working on GCC or anything in the GNU toolchain? How many people here have dipped into or are dipping into both? Actually, there's a number. The point is there are quite a few people who work or have worked on both projects and it's probably time we tried to work out how to work better together. We had a first session at this at the GNU tools cauldron. You can go and watch the videos online. The link is on the abstract for this. But we wanted to go into a primarily LLVM audience and the LLVM foundation suggested that FOSDEM was where you were going to find the broadest spectrum of views. Not in California. That's what we've got. We have three panellists to lead the discussion. Arno de Grommeysong, Pedra Alves and Tom Tromey. I'm going to let each of them briefly introduce themselves. Then what we're looking for is a discussion about how we can work together. In particular, afterwards we'll look at the recording and see if there are some concrete ideas of things we can actually take away and do rather than having a nice comfortable chat about them and then nothing happens. Here's the aid memoir and we can cover anything else. It's not just GCC and Clang LLVM. It's the whole tool chains, LLDB if you're a LLD, GDP, binutils and so forth. For that further ado, let me ask each of the panellists to just say a couple of words about themselves. Hi, my name is Tom Tromey. I work at ADECOR primarily on GDP, but in the past I've worked on GCC and I also worked on Rust for a while, so I've worked on LLVM and LLDB as well. Hi, my name is Pedro Alves and I work at Red Hat on the debug team. I've been working on GDP for a while and I'm a GDP maintainer and contributor. I'm Arnaud de Gromaison. I'm working at ARM. I spend a long time in the compilation team at ARM working on LLVM and before joining ARM I was in startups doing custom DSP, working on a custom DSP and a custom processor and I was in charge of the tool chain, which was obviously LLVM based. He just goes to the camera. Okay. Okay, so this is intended to be a session where everyone participates, the three at the front, to stimulate discussion. I've put up on the basis of the meeting we had at the GNU cauldron, some areas that I think matter and where we could work close together. Language standardisation, ABI compatibility, interoperability between the tool chains and also channels of communication. How do we talk together? Do we need different conferences and so forth? What like is each of the panellists now to give their first thought on the one most important thing we could do to improve cooperation? I should have expected to be on the spot. Yeah, I'm very bad at planning. My primary focus is on debugging. That's what I know the most about. When I look at the debugging world, I think it's actually in a worse state than the user language world, like writing C or C++. I think that one thing that would be very good to do is to have GCC and LLVM explicitly co-operate on improving the dwarf standard and documenting and standardising the extensions that they both use. I think that would be a very fruitful way of cooperation. Wow, you've said everything. I come from a debug background as well. Actually, my work on the compiler side is not very meaningful. Looking at the current state and the current like duopoly in the debugger land in open source, I see two aspects that would be nice to improve. One of them is an area of focus right now for our group. It's on the quality of the bug information. We've been looking at the quality of the dwarf that the compilers emit. Does it represent the original source accurately? Does it support the whole set of dwarf, the whole features, which compiler emits the correct output, which one needs to be fixed? It would be nice if we joined efforts in the testing side in the testing frameworks that validate the quality of the bug information. I know that there have been efforts on the LLVM side about this. It would be nice to chat about it. That was the second point, which I forgot right now. It will come back. Sorry. It's a bit hard to be in the third position there. Maybe to differentiate a bit. I think it tightly links to what you've said. I think we could probably better cooperate as communities around the language standards. My understanding of the language standards or the standards, if we consider dwarf as a language or not, I don't know. My impression is that mostly in the standards committees companies are represented. Whether this is LLVM or GCC, this is more tools for experimenting with the new standards. But this is not representing the whole GCC or the whole LLVM community as a whole. I don't know if I'm right there, but I don't think we have LLVM only representative or GCC only representative at those standards. On a kind of a tangent that touches standards, again from the debug side, because that's my expertise, for example, there's been a push on LLVM for open MP support, I think, and yeah, I'm sure. For debugging open MP, there's a standard called OMPD, which is open MP debugging, and that requires implementing a library that exposes a standard interface. What's happening right now is that LLVM is implementing that library, and GNU side also needs to implement something like that, exposing the same library and other tool chains like the Intel compiler also is doing the same, and we have a mixed match of a matrix of the burgers across run times like GDB against the LLVM run time, GDB against OMP from the GNU side, and LLVB against GCC and the matrix. It would be nice to see about sharing that infrastructure, maybe sharing even the code, and that can only work if we cooperate and discuss and experiment together. The other point which I had forgotten earlier came back to me. I wanted to mention like the LLVM uses the remote protocol from GDB, and it has its own extensions, and it would be nice to cooperate in the sense of the ones that are generically usable would be standardised and documented in a single place, and the longer we take to get to that point, the more we will end up diverging, and we don't want to end up in a place where tools end up more incompatible. Sorry about talking only about debug stuff. Okay, thank you. At this point, I'd like to throw out the same question to the assembled multitude. If you'd like to put your hand up so we catch it on tape, if you make your point, I will try and summarise it and repeat it. Anyone here have a view on the most important areas where we could improve co-operation between GCC and LLVM? I think somewhere of a neutral forum for discussion would help in that you've got a sort of basically LLVM mailing list, you have GCC mailing list, you can be known in one and not the other, and there's sometimes intimidating to cross the gap. So let's say more places where you can sort of tread on a more neutral place. If you just say it, I'll repeat it for you. I do not know if he has a very big test, but if you accept the test by the GCC community, we could throw together the test for maintaining co-operation. So the suggestion is that GCC's big regression test suite could be brought together and have it as a test suite for LLVM. I have to say that professionally we've been doing that for 10 years, so it's there, but it will be good to formalise that so everyone can pick it up, they don't have to come up with our magic copy of it. Any other comments? Excellent. So we now have some of those there, so that's great news, the torture tests are there. Which are the ones that cause pain? Let's talk about that first. I'm really curious about this. Like I know GCC incorporates some things from LLVM, sanitisers or whatever, and those are just imported periodically, and maybe there's local hacks I don't really know. On what basis are the GCC torture tests integrated into LLVM? Do patches go back to GCC? Is that? Because I feel like this is an important thing where I feel like, like for instance this thing about co-operating on dwarf, which is territory I'm really comfortable with, right? Like there, part of the process has to be like a social commitment by the maintainers to say, if you're going to extend the dwarf, you have to follow that are agreed upon thing. You have to upstream it or document it or whatever. But that requires like a commitment from both sides. But I'm concerned like just hearing that and if you don't know the answer, like is it a fork of the GCC test suite that to me seems like worse somehow. It's a bad outcome. I didn't do the work. I believe it's actually, it is a fork and it works by actually having a blacklist of tests that won't run. That's been done once before and of course then people end up running a 10-year out-of-date test suite. So I think there are two good points there. One, it's good that the co-operation is happening, but the second is actually if it just leads to another fork, then it's possibly not long-term valuable. More comments from the assembled multitude? I think one thing that works well that I see occur sometimes between projects is just practicing careful communication and respect kind of thing. It's very common for people, like I tried to dissuade people from saying, oh, why was this implemented this way or not that way kind of thing and generally be respectful of the competition. I think helps in all discussions on either sides of things. One thing that I wish we had more of was more of like an RFC proposal would be interesting for certain language extensions that like right now either side will ship extensions without input from the other and not necessarily take these to the ISO standards bodies and I think that's okay, but then typically multiple implementations will work out different kinks or interesting edge cases or things don't compose a certain way with other features and these typically only kind of shake out once you get more than one implementation. So the comment being made is that if you have divergence of extensions as part of the language exploration, then people writing application code have a nightmare of making sure it works on both GCC and LLVM. Just about the documentation thing, I think one issue there is, if you work on GCC, you're familiar with GCC and if you work on Clang, you're familiar with Clang, there may be no one who can write that document who understands the subtleties of the divergence in some particular feature. But I really liked your point and it really reminds me actually of what happens in the web world where different browsers collaborate and features sort of don't become web standards until they're implemented in multiple browsers and they have, like you said, more or less an RFC process and I think that would be an excellent idea. So on the RFC process, maybe one thought I have of where it might get hard to implement it is that the more people you involve into a discussion, the longer it takes and it seems something... I've seen the past incompatibilities between two tool chains appear because in one community there's a quick need to implement something would be relatively straightforward and in the other community they don't see the need as quickly and so one community moves ahead because there's a genuine need for it and the other community doesn't react as quickly. So that might be something to overcome, maybe not necessarily completely impossible. From my point of view, I would say my personal experiences, I find most of the difficulties to come from probably the ABI site in that whenever a new feature gets implemented quite often without realising there's some small binary interface all of a sudden that gets created and it sets a defective standard and it doesn't get documented. Maybe that's because I work more on back-end sites stuff and it's just... probably every week, at least every month some small extra addition happens to a binary interface somewhere and I would say the majority of binary interfaces go completely undocumented and so I'm starting to wonder would it make a difference if we tried to recognise there is a new binary interface that's at least just documented so it becomes more visible and it's not just three people who implemented it and you have to reverse engineer from code would that be helpful? I have the impression that just documenting it would not solve it because when you're... it's only when the other team goes and tries to implement it even based on documentation that they realise oh, this corner case wasn't considered and it's only them that will notice it because the original team just implicitly thought it worked that way didn't even think it would be a possible design point change so it feels to me like a lot of this is going to be based around communication and just reaching out and being friends making bridges and break away from that mindset of them versus us we're all just toolchain people who work on the same kinds of problems and we should I'm adding this extension for the C language I should reach out to that friend on the other side and see what he thinks about it and hash it out a little bit at least in public in preference and make that a way to shake out some things before they are done and documented it and then what frequently happens is it's released and shipped and out and only released later is the other side aware of... oh, there's an extension for that let's try it and then it's already too late because it's already in the wild so the point is reaching out and to build on the point you raised earlier on and also on what Christophe was saying with the fact that the two compiler teams may not be working with exactly the same agenda for different reasons I think what would really be important apart from having some documentation for the ABI would also be to have some tests because just the description of the ABI is not enough ABI can be quite subtle and the corner cases are not often not described in the specification or why you are doing things this way so there you really need tests that's an excellent point that's an excellent point and it's something that you can do in your own team without reaching outside and it feels to me like that's a good engineering if you're not doing that, you're doing it wrong you've got a question at that patiently waiting so the question was the small device it's a very good question so Microsoft have outli open sourced their library last year yes, I think the whole point is to bring in all voices Microsoft is changing they were at the GNU Tools Cauldron this year which I think is a first very positive and big support they are changing as a company and of course the whole point about free and open is it's inclusive that's a great point two questions here so that's somehow related to specification and testing but writing test is not the most tremendous task or the most enjoyable task and whenever we develop a new security feature we could write the tests as a whole community and in a generic way and then everybody could implement it and take advantage of the test suite that's some area where both from an engineering point of view and from a time spent point of view collaboration is fruitful and not very difficult to set up I wish that for 45 souls there would be a test suite that I could use that's a good part from my personal experience the GNU regression test suite is based on a deja-gannu features that say is this feature supported and if you write those generally above you can give them GCC or LLVM and it will work out whether the test can run the problem is the GCC test suite is huge and very old and only the best and newest tests actually work like that so the real world hits a bit there I have a similar question he made a talk about changing the compiler from GCC to Clang and I saw many projects that tried to standardize don't use GCC specific stuff and I think it would be great as a user to have more interoperability so RLVM can just change my compiler without having to think much and I think it goes currently the other way around that some projects just require to use Clang and if I had a platform that isn't supported by Clang I have no chance to do anything and that would be great as a user Here I'd like to say that on one hand interoperability is a nice goal to have the other thing I mean if you try to do embedded software it's often a good idea to test it with multiple compilers because they will make different implementation choice and you need to be to accept that what you have written is not portable so I mean that's the point of interoperability just a subset will be interoperable so that's I mean it goes both ways I'd like to ask the panel what they think about deprecation or it's not deprecation I'm talking re-implementation so if like it or not all the calls for collaboration get pushed ahead further in one than the other and then the other will try and catch up as such in some ways it's akin to code review you do something downstream you then move it upstream and it gets code reviewed it gets changed is there a way of deploying newer features as maybe some kind of tentative extensions or is it anything that you put in binary for life I guess is there rule for say communities to introduce things a sort of ok you can use this but it's not standardised yet whatever it may change in the future that type of thing that's a good question I think we ought to ask the panel and I'll start with Arna this time so I'm not sure I understood fully your question but so around for the deprecation it's a bit hard because you as long as you have one user who does not step-bort near refuses to upgrade his codebase you cannot fully deprecate the stuff or you will have to stick with an old version of the tools I'm far from an expert on the compiler side and the runtime side there are some things that have been deprecated like I'm thinking for example on the G-Lipsy side there are facilities to if you really really need to you can create a new entry point and use help me Mark the symbol versions exactly thank you to like the old binary will still run and link at load time with the original deprecated version but if you try to compile again the program it will be using the newer ABI entry points so it's possible to deprecate things on the runtime side on the compiler side I'm not sure what the answer is I kind of feel like if it's there it's there forever I'm sure he may have found some compiler flags which are dating from very very old versions of the compiler and the compilers are just ignoring them just so that the build scripts don't fail this is that it's not specific to compilers I mean LS has obsolete flags and they are still there and just enough so that's common practice although software was named soft opposed to hardware once in there it becomes a hard stuff that's the command line aspect on the coding side on the language support side is there any like when we will get rid of C89 or before in 40 years I think one thing is you have to differentiate between the different cases of compatibility like for source language in a way it's easier to just say we will no longer accept K&R declarations or whatever it's just errors now I mean I don't know if they are I'm just saying it hypothetically and then people who want to have a 30 year old C compiler should go find one what's hard is ABI compatibility especially if what happens pretty commonly is someone implements the ABI and they think they did a good job but they made a mistake and it's not found out till later if you just have one toolchain like GCC made a mistake it's better to just leave the mistake now it's the standard even though it's ridiculous but it's more difficult when there's two compilers involved because you can have this thing where you have parallel implementations and one or the other makes a mistake or they disagree about what's a mistake or it's undocumented and so they both made a choice and even those cases can be treated differently you know you could say well as communities we'll have a commitment to following the standard so if you catch us in a mistake we'll like make our user suffer a little and change but that has to be like a two way street everyone has to agree to that like as a social thing and then for the case of like implementing a new feature like it's not documented and you want to make some choice you have to also make it like it's a social problem you have to say we'll commit to sending a note to some ABI list to say hey this is what we're planning to do stop us before we strike or whatever Yeah, a question I guess to Peter to reinterpret your question were you asking more about things with thought of potentially deprecating it in the future? Yes, so I'm thinking less about implementing things that are already standardised like things in the front end I'm thinking about say I've got some new binary security features I'm just going to pull stack protection out of the out of the hat here and it will work in a certain way in a compiler that's a downstream binary thing the compiler can choose how it implements it but it has as Christof mentioned some form of ABI clang comes up to that does it need to match precisely it might not be possible to match precisely at the moment I'm talking about for things like that that are kind of not in the area of anything that's really written down how do you get it so that say for example you can maybe transition to a form that both can accept and that might involve changing and I think partly it's also got to be driven by the community I think when there's demand from users that clang and GCC work together they will do it when there's not demand in areas that nobody cares about it's not going to happen so I think it will be to the community to decide what happens so yeah one of my the reasons why I remarked writing trying to produce some documentation if you realise you're introducing some new binary interface even if it's small amount was just to try to make it a little bit easier for other projects to start using that interface one thing I so it would indeed be really nice if we could come up with introducing binary interface without from the start it being forever making it more evolvable I don't think there's an easy way to solve that problem and the only partial solution I think maybe for some binary interface might help a little bit just try to make sure there's always some kind of versioning there so if you need to change at least detectable something like that might be one step in the right direction but then it would need like general agreement oh yes every time we recognise we're introducing binary interface we'll also need we'll make sure there's some metadata somewhere easily interpretable because of version something like that at this point we're quite a long way through our discussion a lot of issues raised here I want to turn to solutions now we've had one suggestion which is for having a neutral mailing list so you don't have to that is neither GCC nor LLVM for discussing neutrally that's a good one I think to take away and see what is the organisation that can host such a list that will be trusted and respected and we can take that one away I'd like to now open up to the panel for solutions we've got one LLVM director here I don't think we've actually got her GCC steering committee member have we? no no no but start with you bedroom solutions oh man start from scratch third compiler solution you standard right I don't know solutions you were mentioning ABI versioning and it reminded me of work that Red Hat is doing and maybe I'll ask Mark we'll learn to help you with this the NICS project what's the name of that the binary tagging thing not that no that's doji's nicklifton's anubin exactly can you talk a little bit about that see what I've done there tell us the solution no it's not a solution but what anubin does is record all the compiler flags and all the the ABI's it can recognize in object files so it's not a solution but you're right I hadn't thought about that there is I don't know how far Nick is with the anubin plug-in for Clang but he was working on that okay anubin is a plug-in currently for GC where it will output ELF nodes for all the object files in which it records all the compiler flags and ABI's it's currently respecting so when you link your object files together your binaries have all the flags ever used in all the compiler versions and he also did that for Clang I don't know how far he is it's at least an interesting thing to do Fedora does it now distro wide so that you can see what is actually being used in a whole distribution which is nice I don't know whether the linker is extended I don't know whether that work progressed to a point where the linker can reject incompatible ABI's but it could be done like it could be used to you can use it to query the whole distribution see what's the ABI's you're using and I think that for versioning we could use it to validate whether the ABI's are compatible which version are you using feel like that could help if we want collaboration or better collaboration not to be just wishful thinking because I think everyone is in agreement that it's a good thing to do if we want to be pragmatic it only happens on real projects and I think both our communities are already collaborating on a need be basis I know for example at ARM the LLVM and the GCC people are working I mean are in the same building it was not the case for some time but now they are all together and we try to to make sure because ARM has an interest that it to have proper support for their product and there are other cases where I think there is collaboration so are there other projects where we could have a broader collaboration and then okay yeah I think when I listen to this I think in some cases an RFC process something like that a shared way to communicate directly between compiler developers is good for back ends like for ABI's I think there are already existing institutions that handle many of these things there is like the C committee and the C++ committee there is dwarf there is like I know more about dwarf which I think is like institutionally kind of weak that need to be strengthened and supported by like these communities but some of these areas like I think the ABI situation is similar it needs a little more commitment from developers and stuff and then some of the areas are terra incognita right like linkers I recommend it all as far as I can tell and just work by magic and like that would be a good thing to change and maybe create a new institution you know to handle that right so I don't know we're running just a bit short of time so I'm going to actually draw things to a conclusion now we've heard issues where we could do better failing us to do better we've heard some concrete suggestions mailing lists projects that we work together some detailed technology from Mark and Pedro the possibility of needing new institutions there's one other I think we've missed which we did once we had an LLVM cauldron the day before a GNU tools cauldron and 40% of the people who went to one went to the other and that was a very good particularly the evening in between when we all came to the same and all drank and ate together my two penithaer I think possibly we need to revive that idea so actually we do talk to each other because sometimes human interaction only matters it does matter I'd like to thank you all for your time I'd like to thank our panellists for giving up their time Arno in particular who came up here just for this session so I really do appreciate you travelling up here please carry on the discussion send your feedback if you can't find anyone else to tell send me an email thank you very much