 Hello everybody, I'm going to get started there's not a lot of people here I did not foresee a lot of people coming so you in the back can scoot forward if you'd like You don't have to I'm going to talk today about something called not invented here syndrome I'll start off with a short amount about myself. My name is Dmitri Gaskin as you can see up there And hopefully you saw in the program I am a Drupal developer I've been programming with Drupal for about five years now and appearing at Drupal cons for four or so and Today I'm going to talk about not invented here or me for short So you may ask what is not invented here or me or me depending on how you want to say it Not invented here is basically when a project or a company or group Doesn't want to use things that were not invented here. That is to say they like to you know come up with their own solutions for things It's talked a lot about in corporate settings You know how Companies like to come up with their own tools their own systems But it's also important for us to talk about it in free software Like Drupal where you know we could be using we could be leveraging a lot more other projects than we are now This last bullet point I'll come back to later, but basically There's sort of a difference between Integrating with other people's code that that is to say sort of working with Optionally versus building on top of other people's code So I'm going to talk about when we build on top of other people's code or rather when we don't and instead we invented ourselves So Not invented here is sort of a problem Here's one one thing that can come from that problem Reinventing the wheel. It's it's a little problematic as you can see it doesn't always work. I wouldn't want to ride that bicycle But I'm gonna so I'm gonna look at There's so what the way I went about making this presentation was I I thought you know Okay, it's a big problem in the Drupal community, right? Well, it turns out there's that's a sort of a question. Is it a problem? I asked a couple of my my friends within the community and about what they thought of it and about Like three of the three of four responses thought it wasn't a problem and I disagreed and I still disagree even given their justification, however, I'm just sort of going to explore this problem and Hopefully we can reach a conclusion So the question is do we suffer from me or not invented here? I'll go through a few examples of Not invented here or what may be classified as that And We'll see, you know, do we suffer from this problem? All right start off with the first example Hooks or the hook system so it could be argued that it's a It's a version of the observer or the mediator design pattern It could also be argued that it's not and however one thing that's not arguable is that it we completely uniquely implemented this You know this design pattern has been implemented for years and years and years and all sorts of different systems various different ways but Sort of none nobody does it like Drupal and we decided to invent it ourselves So, you know, if you if you have some some knowledge of the observer of the mediator pattern You might recognize aspects of it in our hook system. However, that doesn't necessarily mean that that's you know What it is and that your knowledge will completely apply to this Another example is the XML RPC library This was originally known as IXR. I forget what that stands for. It was just looking it up earlier But there was this XML RPC library sort of the one the main one for PHP That we pulled into Drupal. I think 2005 2006 When in the process of pulling it into Drupal, we completely rewrote it from scratch and It's had several releases since it just had one in 2010 was the most recent release and we haven't been able To pull in any of those updates because now our code is not compatible with their code Yeah, so basically we haven't incorporated their updates since 2005 So we're stuck with this old version if this old version were to have a security hole we would fall, you know It would it would be a problem and we'd have to fix that problem Instead of you know letting them fix it letting them do the work Another problem with something like this is that you know if you have learned how to use this this library And then you come to Drupal and you want to write some XML RPC code You know you're kind of out of luck you have to learn something new so Kind of begs the question why did we do this well? I'll get to that later. I still have a few more examples There's this function the filter XSS function This was also. This is not originally Drupal code. This came from a different place. It came from a library called KSES Which then became filter XSS Again, we rewrite it for Drupal In this case, you know It's the fact that you you can't port your knowledge over like if you learned this library and then came to Drupal I wouldn't say that's such a big problem because it's literally one function So, you know, it's and it's pretty easy to use it's one function with two parameters one of which is optional But I didn't have room on the slide We rewrote it for Drupal and As it says there it it's now on maintain and it has been unmaintained for about ten years and There was a and pretty much now it's become like the XSS filtering library For most PHP applications WordPress uses it. I think jubilee uses it Tons and tons of you know smaller PHP applications also use it And and I think 2010 there was a major security hole in it And so, you know, it did it failed its one job which was protecting people from grass-eyed scripting attacks And but due to the fact that we rewrote it ourselves It didn't affect us at all so, you know, there was there is some some benefit to to writing our writing rewriting things ourselves or writing things ourselves Simple test So you see here these are sample simple test results With simple tests simple test is another case of where we pulled in some third-party code But there is there was a problem or there wasn't really a problem But we started sort of at first we were just using simple tests and we had a like very light wrapper on top Then we started doing more and more and more of our own code on top of simple tests to the point where Using simple test itself sort of became useless because we were doing so much of our own stuff On top of that and then I'll take guilt for this Or I'll admit guilt for this sort of starting this effort to rewrite simple test completely just for Drupal and Now it's a problem because you know if simple test is one of the probably two or three ways that people test PHP And if you learn Simple test or another one of those ways but say you learn simple test and you come to Drupal And you turn on the simple test module and it's not simple test And you know that knowledge that you gained learning how to use simple test now you need to learn something new So that's you know, we're putting people through unnecessary work here I think it'd be you know, we could gain more developers gain more traction if we used Code from others because other people would already have that knowledge And you know you could come into a project knowing simple tests and be able to write tests instead of having to learn something new Just for Drupal All right Next one jQuery JQuery we pretty much succeeded. You know, we just we use jQuery verbatim We even use some jQuery libraries on top of that, you know cookie form. You can see them there We also use a library called farb tastic, which was actually written for Drupal Funnily enough it was written for Drupal and there is now a colorpifker and jQuery UI Which we don't use and said we use our own thing which is Right now it's it's separately maintained and it has it is used outside of Drupal However, it's definitely not the only jQuery color picker solution So along with you know, just using jQuery. We've had to write jQuery code on top of that But our jQuery code still feels like PHP code It doesn't really feel like fluent JavaScript. So if you were some sort of You know, if you were a JavaScript programmer you came in It wouldn't be the most intuitive thing that you know to help out and to pitch in because you're what you're sort of reading is JavaScript written like PHP and that's you know, that's problematic because You know again, we want to be able to have as many people helping us out as as possible as many new developers coming in the project To get started with Drupal. It means you have to learn all these new things This is a bit of more of a historical one The X template library was used before we had PHP in our templates We actually went through a couple of iterations on the theme system one of them where we had theme engines and X template was the main engine in use. I believe we did bring this actually in as is But unfortunately later it became unsupported the maintainers of the X template just dropped it and We sort of we got bitten by that because now we were stuck with this pretty big library that was unsupported That you know, we weren't the biggest fans of we could have still used it and maintained it But that's extra maintenance work for us and we really want to do that when we can spend our time doing other things Next thing is our database layer. Our database layer has gone through quite a few different revisions We started out using a pair database compatibility actually That was like way back in the Dark Ages We said after that we moved off and we used our own compatibility layer Where we simply swapped out a function name depending on what database you were using not the greatest solution and you know that definitely All you need to know is Or you basically like you used sequel But if you wanted to use any sort of database compatibility layer Or rather, you know query builder or whatever. Well, you couldn't instead. We wrote our own We did base it off of PDO. So if you know PDO the PHP database library, then No, it doesn't get you very far actually We still wrote something on top of it. So oh Look, there's the author right. He just stepped in our guilty of not invented here syndrome Because other solutions did and do exist So as he likes to point out it is not in fact exactly the same doctrine is a Sort of database compatibility layer that's also an ORM But it does sort of focus on the same problem space So, you know while Larry's not necessarily completely guilty, you know, it's it is a problem He will admit that We do use a couple of other libraries just doing minor things for example archived our library that down, you know unzips files when you download them from the downloader and The PHP password hashing library, which we also use successfully So, you know clearly we have had some success incorporating other people's code, so maybe it's not a problem Other things we have a pluggable subsystems So these don't really solve the problem of not invented here But they do leave the opportunity open for somebody to create a library that you know integrates with other systems However, I think that oops. That's supposed to say utilization at untilization You know that that's not really the solution because we still had to invent whatever lies under those pluggable areas And just integrating with the library is not the same really as you know using it fully because the knowledge still doesn't transfer and you know just because you know how to use say What's it like for a session's redis cash Just because you know how to use that like sure that'll help you a bit But for example, if you want to use it you've got to write the compatibility or throughput. You still got to learn that There We Do support our measly list of two open standards. I'm sure there are others But open ID, which is currently on the way out of the internet and RFRDF sorry RDF So these are examples of where we have used ideas from other people You know and incorporated them in our own code base I think that these are you know good examples for us to look to on how we can continue integrating that Another sort of problem area for us is using Outside idea or not using outside ideas rather So we've ignored break best practices such as you know encapsulation abstract interfaces also, you know globals and That's when we ignore outside ideas like that then it makes it harder for other people who come into the project to sort of grasp on and to To build you to build using Drupal because they you know they have a different knowledge base They have to learn something new But if that's not enough to convince you there are still questions. Is it a problem? So So these are sort of a couple of reasons why it is a problem it does take time and energy To build our own solution and then we have we have the responsibility of maintaining it So, you know, it's we could be spending that time building more components on top of those systems Instead of just building those systems or maintaining those systems It also raises learning curve. So if you learn one system, you know, for example, you learn simple test Like the simple test not Drupal simple test and then came to Drupal, you know That knowledge doesn't completely transfer sure at the basic concept of what is a test and how does it work does? but You've got it, you know You still have something to learn in fact quite a bit to learn There is another side of it. However, there are some benefits and we have had some successes with writing our own stuff The unmaintained projects like in the case of X template Could you know, that's a You know that that did that did hurt us. So maybe you know writing our own thing which we did was the right solution and We could end up creating more work for ourselves In terms of integrating other code Like it could be more more work to integrate other code rather than to rewrite it ourselves however, that's not I don't think that's that justified of an argument in itself because You still fall can fall victim to other problems such as then later having to maintain that code or Having to deal with security issues which could go either way. So if there is a security problem in it And the maintainers of whatever outside library you're using don't fix it then that's you know, that's a big problem And so that's one of the arguments for writing your own code because we do have a very very big and very active security team However, if that you know that other project is responsible, then you can also then we can also save ourselves a lot of work And then of course you have to deal with stuff like ugly code ugly interfaces Do we want to be using that or do we want to just write our own? so However, as I said like I don't think these are enough reason for us to believe that you know It's a it's just a bad thing like I think or rather just it's a good thing I think it is a problem that we've written so much ourselves And with that let's look at why this is a problem. So here's the it's a problem face There are I think sort of two categories of issues They self-taught coders Or rather cultural issues so as you can see like when when When Drupal like Drupal came from a background of people who taught themselves to code and people who are hackers not really people who you know Learned to be programmers in school and learned you know learn their list of design patterns that kind of thing And in fact people have sort of been antagonizing that and saying like no we want to keep our code base approachable and We want to be it like we want to have something that said somebody can you know Spend a weekend, you know digging really deep in a Drupal Without having to have a huge amount of computer science background instead. They can just you know They can look at this. Maybe it's a pile of hacks. Maybe it's clean code We don't know because we're not computer scientists There's also, you know, we have been burned in the past as I keep returning to X template We have been burned in the past by outside code and So now that makes it sort of less incentivizing for us to want to continue adding more code Also, there are there were other issues in the past that prevented us from using Sort of more outside code Ecosystem issues so the PHP ecosystem wasn't there that is to say most of the third-party code sucked anyway, so You know, we we could look to to integrate something else to that solved that one of our problems But it sucked so we couldn't use it And you know, that's an interesting sort of point of contention there Like if it sucks, should we still use it because it's third-party code and we don't have to maintain it ourselves And it it sort of requires a case-by-case evaluation, but it could go either way and We could experiment with writing our own solutions. In fact There's a sort of mentality among Drupal developers of really liking to experiment When the when the internet like was sort of still I mean sure it was it was moving, but It's moving now. I would say faster than ever before However, so well, yes, so that's why we sort of we've had that mentality of you know Let's do us do it ourselves because you know, we can we like to try things out We're just hackers hacking our hacking our way through things and the other code sucks anyway But I think that's changing And so where are we headed? I think we're headed towards more integration and more than that more Utilization of other systems. So The web is moving faster than ever and we have to team together to succeed a great example of this is how we've been teaming up with the symphony team I'll show that in a couple of slides But we've been we've been working together with the symphony team now You know, they're helping us write code. We're helping them write code And the things like symphony the PHP ecosystem is finally matured to where it has good quality software Our developer base has been shifting to now where there are people like Larry who you know our champions of the let's use design patterns instead of hacks and People are more sort of cognizant of these things and they want to move in this direction more so we've had sort of cultural changes and Drupal 7 is huge. It's really really big and to come into it like not You know at least for me. It's like Drupal 7. I still don't understand Drupal 7 And I think that building Drupal 7 on top of other rather building free future versions on top of other components Would help in the sense that you know It would be easier to understand what is Drupal as opposed to what is something else instead of having to understand this humongous monster that is Drupal There's also actually an interesting parallel with how Assembly like people used to write in assembly languages first they you know before that, you know, they do the punch cards, but they There's sort of this movement from lower level to higher level So now people are writing in higher level languages like PHP or Ruby or stuff like that And so I think the Drupal community has sort of moved in a similar direction Where we're no longer interested in just building, you know these lower level parts But instead interested in building bigger stuff and just and building on top of other people's things Where we can let somebody else handle the lower level parts and instead we're more interested in focusing on the bigger things In a sense, we're growing up as you know people did with programming languages as Dries did from that picture you saw earlier So You know one one big place we've done this is a symphony so We have integrated plenty of Third-party components. I think this is by far the biggest one we've incorporated and in the most fundamental way I mean you could argue that are you know, for example cross-site scripting filtering is pretty fundamental, but It's you know takes up nowhere near as much code as symphony is going to help us out with It's saving us time energy and maintenance like there's a whole team dedicated to symphony that's here at Drupal con But mostly dedicated to symphony that's You know, they're they're the ones taking the time into developing symphony into maintaining it You know pushing out better better versions security updates that sort of thing So we don't have to work on that instead we can focus on building on top of symphony and building You know, you know all sorts of crazy things So we can you know pull parts of Drupal out and make them rely on external components like symphonies So instead we could spend our time writing bigger And better things all right There's still a bunch of outstanding issues not everybody thinks it's a good idea Some people think it's too radical of a change as with any culture our culture shift slowly And you know Are we has our culture shifted enough to a point where we can integrate something as big as symphony yet Or are we still taking those those baby steps in order to? you know build up our sort of our ability to our ability to do that our developers becoming Or rather have they become yet enough sort of wanting to use more Like less experimenting themselves and more using other people's components have people really gotten to the point where they want to Start building higher-level things And start using for example design patterns Or is that too radical of a change? and Then the last question That's my own question is is Drupal 7 approachable So I mentioned earlier like a big thing has been we want to maintain approachable code and something that someone could just pick up on the weekend and start hacking on and I Think that you know we've already sort of crossed that threshold with triple seven And to the point where you know I can't pick up triple seven in a weekend I can't spend three months and learn triple seven I'm still battling with triple seven trying to figure out because there's like so much new stuff added That I can't figure every little part out like I could in Drupal 6 So I think We might have even already crossed that line with triple seven and now it's just a matter of sort of recognizing that and Realizing that okay We've already crossed the approachability line that should make it the decision easier if anything about whether or not we want to Start and continue integrating others code All right that that's it and that's a little bit shorter than I had managed imagined it would be But thank you for coming and thanks to these people for helping out questions No questions If you could step up to the mic Like the natural follow-up question is like, you know, what's next what can we For those interested in kind of pursuing these things and experimenting around different ways we could Pull things out or chop things off or add things You have any ideas or you know, is there additional buffs or how how can people get involved in the discussion more I guess That's a very good question and it's a very hard question because You know, as I said, you know, we're sort of in the process of a cultural shift And it's not just that we can just start adding things. We've slowly worked to this point and are continuing to work on this You know, are we ready to be adding things? Maybe we are I'm in that case It's pretty much a matter of just finding like something that you want to add and adding it or you know proposing it Suggestions, so I can't you know give you a definitive answer but then I mean pretty much do what you want see if people like it and People might hate you but it's a worth a try This actually came up yesterday in the core conversation around theming And let me tell you that was an interesting conversation, but The kind of general consensus in the room was that we are basically going to have to scrap a lot of how we built the theme layer and kind of start over and I made the observation that the symphony framework does actually include a Templating framework. I believe it's called twig and So this might be a really good opportunity to do to kind of push more of this kind of code You know outside code utilization that could probably actually really solve a huge problem that we've been having with the theme layer I think the real question going forward is How do we convince a community of very very invested? stakeholders who have invested in some case Nearly a decade of their life into writing this thing To kind of get over the ego factor and say well, you know what? Maybe we didn't do this the best way. Maybe there's a learning opportunity here and Yeah, I think that's I think that's the big hurdle is kind of getting people to realize Hey, you know what? We don't know everything. Let's let's play around and see we can find that might actually solve our problems a little better than we did Yeah, I mean I definitely totally agree with that. I think the only thing is like people Almost have the feeling like okay. We got it wrong the first time. That's okay So let's try again instead of letting you know like somebody else try again. It's already solved the problem You know we should try again, and I don't think that's the solution I think that yeah, we need to sort of recognize that maybe other people already have solved the problem already will solve the problem for us just want to speak just not nearly as deep a cultural problem, but One area where I think we are gonna need to look outside and Kind of broken record on this this triple con is at the JavaScript framework level in the client-side frameworks where As we look more and more towards services, we're gonna need to have a really great robust JavaScript framework layer That works flawlessly with Drupal. I got a horse in the race the backbone module. I think backbones perfect But I think that's one where there's just no there's no reason why we'd want to make it ourselves great ones exist You know I think I guess there's two sides to it There's there's refactoring the existing code and then there's as the new problem spaces open up You know really looking aggressively and I think you know the contrips space is really wonderful for that so if anything we do have a pretty robust culture of You know taking things that were invented outside into the contrips space Maybe it's just about being very conscious about developing them in a way that They're really robust API modules Yeah, I mean I agree there that Contrib is definitely been sort of more open to using other people's code for stuff Then core has It could be any number of reasons mostly probably a factor of the fact that could remove so fast and is able to do that But I do think yeah, I mean it's it's a good opportunity to you know To start exploring other other places that we can use other people's code And yeah, I mean that's definitely JavaScript framework is could be something huge that we probably don't want to be writing ourselves before you speak First guy on the slide there is right standing at the mic. Thank you very much to him for giving me many ideas for this talk To answer the earlier question about where to get involved in this in this discussion