 Good afternoon everybody. Is this the right volume? Everybody can hear me all right? Okay, cool. So welcome to my talk about the Triple 8 render pipeline. My name is Wim Leers. I work at Acreo in the office of the CTO. And in the past one year and a half approximately I've been working on performance issues and on criticals. And in the end of 2014 we started looking at the render pipeline and the state of it, the understandability of it as well. And unfortunately at the time for understandable reasons because Triple 8 has changed a lot of things. But sadly the state of the render pipeline was not that happy in the sense that it was very, very complex to understand. It took me full time for about a week to understand. And I'm already working in core all the time. So that was not great obviously. So we worked together and came up with a path forward and make it actually understandable, clear, less confusion and so on. And so at this point the render pipeline is in a good state. And what I'm presenting here is a state that the pipeline has been in for the past several months. So it is very likely that this is also the state that it will ship in. Maybe with some smaller changes. But in any case this will be very, very close. And so hopefully by the end of this talk you will be able to understand the Triple 8 render pipeline. It will not be scary anymore. The diagram that I listed on the slide here that was being shown while I was waiting for you guys to arrive. That is the diagram that we'll be using. And so that diagram shouldn't be scary anymore. So let's get started. So our guide is not sadly this hipster llama. But a diagram that looks slightly intimidating at first sight, which is this one. But basically that diagram is just a map of Triple 8, how it works at a high level. And it comes with driving instructions. That was the thing on the right-hand side there, the column on the right-hand side. So in this talk we're going to take a look at each of the areas of the map, just like a regular map if you will. And at the end you should have a pretty complete understanding and you should feel like at a high level you know how Triple 8 works, how it renders pages, and so on. So if you haven't yet already, please download the PDF and follow along, I would say. I created this together with others to hopefully have that be a very useful reference while working in Triple 8. While debugging, while developing, while whenever. So if we remove the drive instructions for a second and just keep the actual map, then it looks like this, slightly less intimidating. And like any map we need orientation points. So in the top left corner we have a request that is coming in. Because the web was all about requests and answering those requests with responses. So from the top left, where we have the incoming requests, towards the bottom, where we have the response. So from top to bottom we are processing the request and formulating a response and then actually sending the response from left to right. We are at the very surface of Triple and towards the right we are getting deeper into Triple. And the various areas that you could see roughly speaking are the ones that are here marked in blue rectangles. So those are the steps that we'll be looking at. Now I know that this talk is at the front-end track and this may seem very back-end-y and you would not be wrong. But a front-end is meaningless without a back-end, right? A front-end needs to talk to a back-end to be able to show something meaningful unless you're creating a static website in which case you probably wouldn't be sitting here. So I think that as a front-end developer it is useful to have a high level knowledge of how Drupal 8 works, how it serves responses, the things that you are working with. But also given the whole headless Drupal slash decouple Drupal hype, it is useful to know which things Drupal is capable of doing, which kinds of transformations it can do, what kind of expectations you can have, and maybe what kinds of responses that you as a front-end person would really like with Drupal 8 isn't delivering. Well, hopefully this will give you the insight or the sufficient amount of knowledge but not too much to allow you to talk to developers and say, hey, I want my data in this kind of format, in this shape, make sure it happens, and hopefully this will help you understand at a high level how that works and that should hopefully help you as a front-end developer as well. So a quick show of hands who would consider him or herself a front-end person in the room, and back-ends preferred. Okay, so it's approximately 50-50, cool. Okay, let's get started then. So the first step, and I think that everybody in this room already knows this, if you're a, or actually one more, is anybody of you more of a symphony person than a Drupal person? Okay, very few, that's fine. Okay, so let's get started. In the Exit PHP, we've had this file for a long time, actually since forever in Drupal, and what it really is, so we have had it for a long time, and what it is, it's a front controller. That's what symphony calls it. That's the design pattern, and maybe the design pattern is a bit over the top here, but basically all it is, is it's a central point, a entry point where all requests are coming in through, and from there, we are deciding what to do and what response to send. By symphony convention, this would be called app.php, not in Exit PHP, but basically the name is irrelevant, it's just a name, that doesn't matter, so just for the symphony people here. So the responsibilities are of that in Exit PHP thing, and this may seem overwhelming at first maybe for some front-end people, but really it is high-level, so no worries. Symphony request object, that's what Drupal 8 is using. So instead of talking to a global PHP object, global state is bad, so we and symphony have this concept of a request object that encapsulates that so that we can pass it around and not muck with global state. So we create a request object. The second thing we do in the index of PHP is creating an HTTP kernel, and we'll talk a bit more about HTTP kernel later, but for the back-end folks, in the initialization, the creation of the HTTP kernel, what we do is we initialize a container, and basically we read some settings from settings.php to know how to talk to the database, to talk to which database, and that's basically HTTP kernel. Next is, as a next step, we use both. So we give the HTTP kernel a request object, and the HTTP kernel will make sure that we get a response back. And I think that this is, by the way, the key thing here, and this is really what the web is all about. You're sending requests, HTTP requests, and you're getting responses back. So at a high level, this is basically what Drupal is. You give a request, and you get back a response. And so this is what every single answer that Drup provides, every single response is thanks to this. So once we have a response, we need to prepare it, maybe do some calculations on rendering, depends on what kind of response you have, and as a next step, you send the response. That is making sure that the thing you've constructed actually makes it back to the end user. And finally, once the response is sent, it's possible to still do some final bits after the request has been sent, so things like logging or statistics. But that's it. So basically, this is how Drupal works at a very, very high level. And this hopefully will be familiar to any symphony developer because it's exactly the same in any symphony application. So the next step, HTTP kernel handle. So remember that here we got the request, we gave it to the kernel, and we get back a response. So we're going to look at in a bit more detail HTTP at the HTTP kernel, what it does. So what it is, is basically an arbitrator decides what should happen in order to answer that request. HTTP kernel is really the heart of both symphony and Drupal, and so it is used in unmodified shape. So the documentation that symphony provides, the one in the link there, is also applicable to Drupal. So it should make for more cross-project collaboration. So the responsibilities here are negotiation and routing, and that's happening during the request event, but more about events a bit later. So we need to negotiate a format to know whether we need to send a response in JSON, in HTML, in an image format in PDF, whatever is needed. And then routing, determining given that we have a certain format that we need to answer to, figuring out which code should actually generate that. And for the background developers in the room, in Drupal 8 we have routing.yml files, and there the underscore controller thing that determines which is the code that will be used to formulate the response. Once we know which controller should be used, there is another event that allows for a last second overriding so that still something else could be returned. And once we know for sure which controller is the one that will be used, we need to provide that controller with the arguments that it expects. So introspection is happening to figure out which arguments are needed, and then make sure that the parameters are converted to match that. So for example, node slash one, one means that node with ID one should be loaded. And if my controller expects a node object, then the node one URL will make sure that through a parameter conversion that one is converted to node object one. So basically just mapping those things, that's all this is about, fancy words, but it's really simple. So given that we know which controller to call, given that we have the arguments, we can now finally call the controller, the thing that will formulate a response. And so that we'll be looking at in more detail later. And if the result of that controller, that the return value is a response object, because remember we have in symphony that request object and as well as a response object. So if the controller returns a response object, then we're all done. Basically we have the entirety of information that we need to know what to send back to the end user. Because the response object encapsulates all of that information. But if it was something else, so for example if your controller returns an object or a renderer way, we need to still turn that into a response object. Because as we saw in index of PHP, the only thing that knows how to deal with is response objects. So eventually we always need to get a response because again that's what the web is all about. So in that case there is a view event subscriber that allows you to turn whatever is returned by the controller into a response object. And then finally we have the response event for last second modifying. And there are some good use cases out there, but the funniest one I've thought of yet is that every year there is Pirate Day. And on Pirate Day you want lots of YAR and FiddleDD on your website. Well through a response event subscriber you could easily just inject that in several places if you want. So basically this is the diagram directly from the symphony documentation and it's very similar. On the left hand side a request is coming in. On the right hand side a response is coming out. And the blue boxes, the blue rectangles are the events that are being triggered along the way. Purple things are where stuff is happening if you will. And then the most important bit again is the controller there where the controller is called. If it's a response object that we get back we go immediately to the right. If something else returned we go down below to the view event. But eventually we always end up with a response event. And so really at a high level this is all that Drupal is about and even all that symphony is about if you will. This is basically how every single thing that you see on the Drupal website eventually is working internally. And so the possibilities of this, the HTTP kernel are also pretty interesting. So you have these various events that I mentioned. And basically anything is possible with those. For example the response event subscriber. The response event allows you to do things like the annual pirate day inject FiddleDD stuff. But for example at a request event subscriber you could do some very early overriding and determine that based on some dynamic condition you instead want to return a picture of a llama instead of an actual piece of content. I don't know what you want to do but basically anything is possible. So hopefully that is all clear at a high level. So we've dealt with the first two columns if you will. And then the third one is the one we'll be talking about next. And those are events. We've covered a few events already in the HTTP kernel. So several or if you will most are triggered by the HTTP kernel. And they're really the glue between symphony and Drupal. Because remember that the first two columns there is nothing Drupal specific in there yet. So the top layers are really just standard symphony. And then with event handling we are able to pull in the Drupal concepts and make Drupal work in similar ways as it used to work in the past. So Drupal can also of course trigger events and it does. And we'll see one of those later on. But remember events are the glue between symphony and Drupal. So the next step is the controller. And this is the most interesting bit even though it's the smallest rectangle. But this is the thing that most of you will be interacting with the most, at least backend developers. Because this is basically the equivalent of the Drupal 7 page callback. So hookmen you have a page callback. The equivalent in Drupal 8 is a controller. So it's just the application logic and it determines the contents of your response. And it's already slightly alluded to. You can return different things from a controller. You can return either a response object, a render array which is the thing that we pretty much always return in Drupal 7 and you still can as you can see. Or an object. So in that case we'll need an associated view event subscriber because remember we always in the end need to return a response object. So it's keeping things simple and setting the same expectations always at a higher level. I will go in more detail into the render array later because that's still the most commonly used thing. So the possibilities here are sending a response. If you return a response object from the controller then you have full control over what is sent. So there is no more need for manual printing stuff and then calling exit which was pretty commonplace sadly in Drupal 7 but there was also no other way so it was acceptable. But now there is no more need for that. Everything is cleanly encapsulated. Returning a render array then you basically let Drupal take care of sending and we'll look more at that later. So Drupal will take care of turning it into a response object. But now the really interesting thing and where the first important tie into frontend comes in. If your controller returns a object meaning some kind of semantic data then there can be many multiple view subscribers that operate on that. So for example if you have a tabular data object some kind of spreadsheet like data the controller returns that kind of spreadsheet like data but then it can be rendered by one view subscriber into CSV, another into an HTML table, another into a PDF file, another into yet something else. And this is where things become really interesting because yes, Headless Drupal may be good for some sites but not for every use case. It makes a whole lot of sense. For some things Drupal as it is today or it works today is just fine and I think that we will be seeing quite a bit of Headless Drupal but I think there will be also combinations of both parts that are headed and parts that are headless if you will. And this kind of flexibility in what you return allows for those hybrids to exist because parts of the website maybe if you're building a website for a library let's say then they will have a news section, a contact page and whatnot and that could be regular Drupal but then the parts that are more interactive, more dynamic like for example browsing a library as in the books in the library and looking at the metadata about the books. Well in that case maybe the controller should be returning a library book object and that can then be rendered into a JSON response that is exactly in the format that the front end expects or maybe an image for the front cover of the book or maybe various shapes of JSON is really up to you and this also makes it then pretty easy to do things like integrating with external systems and making it pass through Drupal. So in the case of a library for example I can imagine that usually there are old huge systems that contain all of the book information which ones are lent out and whatnot but then exposing that information to the visitors to the users is kind of difficult because they don't want to interact with old shady systems that are difficult to read and understand so in that case Drupal could talk to that external thing the controller could return a semantic object called for example library book and then there could be multiple view subscribers to return data in the various shapes, forms, formats that the front end expects so the front end people could just ask the developers to add another view subscriber to return this particular shape of JSON or this particular shape of hell JSON or maybe something else completely like if anything goes you can do whatever you need at once. So I think the possibility to return both due to force both the entire response to be something which is option one to be to have the return value be a render array which is the classical thing and which is how Drupal 8 still works for for example all of its UI or as a third case objects, semantical objects that are turned into various formats and shapes that is then the other case and the ability to use all three in one site I think is very interesting and will open new doors. So if you look at the render array part in specific for a while so render arrays as returned by controllers or page callbacks in Drupal 7 usually represent the main content if you will usually there are blocks surrounding it but then the render array that was returned by the controller is a thing centrally on the page that's why we call it main contents in Drupal 8 as well as in Drupal 7 and so as we were saying before if the thing that the controller returns is something else than a response object it has to be turned into a response object eventually and this is a thing that turns render arrays into such response objects and what it does is selects and calls a main content renderer and which main content renderer is used depends on the negotiated format so if the negotiated format is HTML we will use HTML main content renderer and so on and we'll look at that more in more detail next and another interesting possibility here is that a view subscriber could turn one of those semantic data objects still into a render array nothing prevents you from doing that and then because it's a render array and still not a response object the main content view subscriber will kick in and turn that render array into HTML for example so you could do something like the tabular data object thing that I mentioned earlier that could be turned into a type table or theme table render array and then Drupal would still take care of it the usual way but that means that you have the flexibility of choosing how to format and render things on multiple levels so you can do lots of interesting ways and reusing or even gradually shifting from the Drupal way of doing frontend towards something else because your controllers are returning semantic data objects if you will and then maybe at one stage they're turning those into render arrays and then letting Drupal do the usual thing or you can do the JSON and JSON hell JSON or whatever thing so you have a lot of flexibility there so looking at the main content renderers these are the actual things responsible for turning a render array into a response object and the responsibilities are the basic ones that are supported in Drupal a core are HTML obviously Ajax for Ajax responses remember those dynamic loading things that Drupal 7 did use very often but in some places does that's the Ajax type and Drupal 8 also has the ability to render any content into a dialogue or a modal dialogue so those are supported as well now this means that you could also add more main content renderers so for example pdf if you have a need for articles being printed or downloaded as pdf then you could have a module that adds a pdf main content renderer or maybe even something else depends on your needs you can add as many as you like so taking a step back and looking at the big picture at the diagram this is what the entire thing looked like and it may still look a little bit intimidating but again let's look at the orientation point stop left the request comes in bottom end of request left to right getting deeper into Drupal if we flipped it around a little bit then we can see the various layers that their request passes through so first there's index of PHP which is symphony like if you will then there is HTTP kernel which is definitely symphony then there is the glue between symphony and Drupal the events next there is a controller which is where your logic lives and then depending on what the controller returns so for example let's say return the response object then we skip five and six and we go immediately to the bottom because we have a response we can send it if the controller returns something else a render array or an object then we still need to do some work in case of an object that's not listed here because I don't know which few subscribers you will have but in case of a render array it will go through five and then six and then we have a response object and it will go to the end and send the response so hopefully this makes it much clearer to give you a high level picture of how Drupal 8 processes requests now the most frequently used main content renderer is of course the HTML one because the web is all about HTML so let's take a deeper look at that part so the interesting thing here is and it's new to Drupal 8 is that it supports page display variants which sounds complex again but it's just about alternative ways of rendering main content and basically that just means alternative ways of filling the page.html.twig examples and examples there are the block module which is about rendering the main content centrally and rendering other blocks whether they're pure decoration or in relation to the main content no matter but basically blocks surrounding the main content panels has a different way of doing things but that is another page display variant page manager is already ported to Drupal 8 and is already using this abstraction and so that is another example of a way of filling the page of the HTML.twig template so the default is a simple page variant and that is basically just the main content and zero decoration so you just get the title and the main content and that's it but of course the Drupal 8 standard profile still uses block module and therefore the block page variant is still being used so that's the main content with blocks around it so the responsibilities here are basically filling the regions of the page template and render that into a response and it detects or determines rather the the page variant to use through an event the selected page display variant event and then whichever one is selected simple block page manager something else that one is built and then it's rendered now the interesting things here are that you can have panels page managers and other things without any hacks and that also means that you don't need the very very deep knowledge anymore that was necessary to be able to build panels and page manager in Drupal 7 because that was not easy at all it was quite a daunting undertaking and now it is just listening to an event saying I want my thing to be used not the Drupal stupid default thing and then you have that be used and that's all there is to it so if you if one of you has maybe in a different opinion on how pages should be constructed or what the key elements of a page are and you've never liked blocks or you'd like to experiment with something else that used to be very very difficult and now it really is a matter of probably 20 lines of code in total so it opens the door for new paradigms maybe even and maybe even something like a partially decapitated page variant so it makes between headless Drupal and something that isn't which still lets Drupal render the main content but then instead of using something like blocks or panels which are still conceptually relatively similar instead you rendered block like things but using js using react js angular whatever you prefer client site stack client site caching and local storage and that allows you then to render more dynamic or interactive alternatives to blocks such as for example showing live statistics like that are updated while you're looking at the page of how many people are looking at it live common thing that shows up right away anything you want any fancy things anything becomes possible that you are that you have come to expect maybe of the more modern js rich frameworks so as you can see you can very you can you can create very interesting new combinations that would have been very very painful to do in the past um so hopefully that will help you with some interesting experiments and so finally a little bit more about render arrays um render arrays are still very much the same but they are improved in many ways um and so I will cover some of the higher level points in this in these slides but not details for those I have other documentation pages on Drupal dork and talks so first assets in Drupal 7 what we were doing and this is very much relevant to to front end as well what we were basically doing was always loads jgrad.js and Drupal.js even if nothing on that page was using it and as we all know loading js comes not without a cost parsing it also and executing it definitely so we were blocking we were delaying showing the page because we were loading that js even though we were maybe not even using it so in Drupal 8 what you what we have is dependencies between assets so we make sure that the that we only load those assets css or javascript that are actually being used on that page so in Drupal 8 we have the concept of asset libraries where you declare which css and js logically belong together which kind of form a logical unit which are components in some lingo um a forget the word basically reusable components of css and javascript that work together and that supports a semantical piece of content and then the ability to declare that that css and javascript also expects for example to that jQuery is pressing on the page or jQuery.once or something else you can declare your dependencies and that allows Drupal to only load those things that are actually necessary. The next one is cache tags so some of you may have heard that Drupal 8 now ships with page caching enabled by default for anonymous users and the reason that we didn't do that in Drupal 7 was that we wouldn't have been able to reliably invalidate cached pages so basically what we did was whenever any node was updated whenever any comment was posted we cleared the entire cache so we had to rebuild every single thing and that's why it wasn't enabled by default but the reason it worked that way was because it couldn't have worked in any other way the APIs were lagging and this is where cache tags come in because the problem was how to clear all cache items containing something the title the body field whatever from node 42 the answer was it's impossible to know and that's where cache tags come in so cache tags declare this render rate this piece of markup depends on this data in Drupal so when that data changes please invalidate me also so that I'm always up to date that's basically what cache tags are cache conflicts are related but different they are about the variations so the question is does the representation of the thing I'm rendering vary per permissions per URL per interface language for something the answer is cache contacts and they're very similar to the HTTP vary header if you know that one so a concrete example at the bottom I'm not sure if it's legible in the back but the comment says associate the user doc permissions cache contacts with the render array and then the logic says if the current user has the permission pet llamas this is for a zoo site a petting zoo site um and the market was done how many llamas will you bet today question mark um so that that contents is only shown if the user has the permission to pet llamas and how do we make sure that then if I have permission to pet llamas but somebody else doesn't how do we make sure that we get to see different contents that's where cache contacts come in we say that the content varies depending on the permissions and that then makes sure that we don't show the wrong content possibly sensitive content to the wrong user and then bubbling this is where the things come together so we had assets we had cash tax and we had cash contacts and all of those things really affect the the overall html because for example if my fancy forum needs certain assets maybe some javascript to make things more usable some css to make it look nicer then we cannot really load that css and that javascript within the html for that forum we have to load it at the level of the html itself similarly if my cash tax for a field for example so here for example i'm looking at the field for a node and i'm showing the current use like the author for example so the author is being shown so it is tagged with user two this is just a representation to make it easier to explain this is obviously not actual html or triple renders so the node that i'm rendering when that is rendered it contains the fields so it actually also depends on the things that the field depends on so for example my node shows the body field as well as the author that means that the node which is rendered also depends on the user not having a changed name in the meantime so i'm tagging it with the user and so the cash tax for the field bubble up to the node level but when rendering the block that then it is containing the node that block actually also depends on everything it contains so that that's where bubbling comes in and it works just like javascript events so basically just deeper in the tree everything deeper in the tree bubbles up to a higher level to a yet higher level to a yet higher level until we eventually get to the html level and that is how we make sure that we know with absolute certainty and with complete with complete knowledge of of everything that is on the page which assets we need to load which things it depends upon and by what things it should vary and this is how we can make sure that troupel 8 is actually fast and faster than troupel 7 in many cases and even beyond html we can bubble that this is really for the backend people we can even bubble that to the response level so to a new level to a new header called the xdrupel cash tax header and that allows us to know with absolute certainty that we can invalidate all responses that contain something from node 5 whenever we modify node 5 so we can purge all of our varnish instances all over the world instantly and have that content appear instantly and not invalid anything else so render caching that's where all of these things come together really this is basically fragment caching is what is called in many other places this allows us to cache expensive complex portions of the page for example a node can be quite complex maybe contains many fields and whatnot that is expensive to gather all that information and then call the templates to put them all in the right order and structure this is where render caching comes in and it uses all the previous things to make sure that we can cache correctly and invalidate correctly and so on and if you're interested this is where the intersection between front end and back end definitely is very present these things cash tax contacts and assets they're all present and they're all very understandable once you've started to use them but it would be great if we could visualize it while developing so that you could see oh this portion of the page actually depends on these assets and that's also very valuable for front end people to make sure that they haven't associated the wrong assets css and javascripts with something that actually doesn't need it but then forgot to add it to the one that does need it so I've created a prototype a very early prototype of visualizing that tree of information and how it all bubbles up so you can see where the associated assets are coming from and where the cash tax are coming from and so on so I'm not a great front end developer so it would be great if somebody is also very interested and passionate about this this is using css 3d and css transitions so defensiest stuff around so if you're interested please ping me and so that's all I've had so if you have any questions please shoot it is very much supported by the render pipeline but at this stage not yet by blocks for example the problem is that we don't have a solid enough way to to know which things a block depends upon so maybe a block is shown that depends on the node because a block is showing for example the file attachments to that node but then the block needs to be aware about which node is being shown and that kind of interaction knowing those dependencies knowing that it depends on that node is being shown that is where the current problem lies but in general yes the render pipeline completely supports that it's just that for example for blocks that is where it's still a bit hairy and unclear if that will actually happen but work is being done right now to make that happen so the markup that I was showing you I think you're specifically referring to this possibly yeah so this is just a way of conveying how it works but you have to imagine that each of those tags are actually render arrays nested render arrays so a tree of arrays which is what Drupal has always used to represent which things should be rendered and so besides markup and maybe declaring which theme function or rather which template to use there is also a pound cache thing which contains that metadata and point attached for assets okay can you repeat that the final sentence fields and entities yes absolutely that works perfectly fine but as I said the problem with blocks is that many or at least a bunch of them depend on some things on the page and that is where it's very where it's hairy but if you have blocks that are self-contained then yes absolutely interesting question I like the analogy to hook page alter but are you sure that you're referring to the view event subscriber not a response okay so were you asking about the pirate example the our horror one yeah okay no definitely there won't be a whole lot of those that's why I gave the kind of silly example of the talk as a parent day annually so no I wouldn't expect there to be many currently there is one in core that is used only in one special case so no there there wouldn't that shouldn't be many but maybe as a way of quickly hacking things like you could even do very very nasty things like replacing a certain class that you dislike as a front-end person and then just do a string replace and replace all of them in your html source it's possible but no I wouldn't expect there to be a whole lot it should be indeed for the very special cases or it as a last resort probably looks like there are no more questions thank you very much hope you liked it so was it useful but please