 Good afternoon everyone. My name is Nigel Cunningham. I have the privilege of working for a boom called Technocrat. Absolutely love it. If you're looking for a job, talk to us at the desk over there. There are some websites that should maybe just not exist anymore. Maybe some of them have, you know, general problems that just make you want to cringe when you look at the design. Maybe the design was dated now or it was just never a good idea. This one is still live. I could do, I'll take it to Chrome and show you and then I'll leave you to discover it later. In other cases, perhaps there were poor decisions that were made when the website was being built. Last minute additions that maybe were a bit of a hurried build, or maybe the implementation of a website was just constrained by what was available at the time when it was made. Sometimes a website doesn't do things that are wanted in the modern environment. Maybe it doesn't match, for example, what the competition are providing today. For the most part, the Master Builders Association in New South Wales that our company takes care of is none of these things. I'm very glad to say. But there is one section of the website that is an exception to that. The Master Builders Association is an association, as you probably guessed from the name, of builders. They exist to provide the builders in New South Wales with a bunch of services, representation before government, promoting excellence among builders, providing them with services. One of the services that they provide to their builders is support and providing contracts to their customers. If I'm a builder in the Master Builders Association in New South Wales and I have a customer that I'm going to be doing a residential build for, or maybe a renovation of their house, maybe I'm a larger builder and I'm going to be building a skyscraper. I can make use of my membership in the MBA to get access to contracts that can help me with taking care of that need to fulfill my contractual obligations with the client. When we were building the MBA website that went live early 2019, one of the areas that was added as a pretty last-minute addition was the eContract support. The eContracts had existed before that. It had been done in a completely different system, not Drupal, and because it was done as a last-minute, hurried addition, it was done without a lot of thought of theming or such like it was just ported as it is, as it was. And so, he says pushing the button again, they ended up with something that looks like this. It looks like it belongs in 2010 still, doesn't it, apart from the bit at the top which is slightly cut off, which is just the normal page here. It's quite JavaScript heavy and it's really in a lot of ways quite ugly. Behind the scenes, it uses two twig templates, one for this display and another one for the PDF. It has to use the two twig templates because we're using WKHTML to PDF to render the PDFs, and that is a binary that is pretty dated, doesn't always have support for the most recent CSS and so on that you might use in the browser itself when people are filling out these forms. It also means, of course, that if they want to make a change to one of these contracts, it requires a deployment. I, as a developer, have to go in and edit one twig template to add whatever the field is, make whatever the change is, and then I have to go into another template, make edits to that, and because the version of WKHTML to PDF on your dev environment might not match the version on the server, I also have to do a bit of an iterative approach making sure that it's going to look right when the PDF is generated on the server. There's no electronic signing with us, which, of course, every man and his dog wants nowadays, not even remotely possible to support it. So MBA came to us, I think it was towards the end of last year, with a shopping list for changes. They wanted the system rebuilt, and on their shopping list were things like, we want the administrators of our website, so MBA staff members, to be able to add new fields to the contracts, to edit them, and to be able to modify what the PDF content looks like. We'd like to be able to do calculations in there. So if the builder enters in some fields saying, here are the payments that need to be made, and here's the deposit that's going to be provided to start with, then we'd like to be able to calculate the total price automatically, but then also lead you override that if you need to. We'd like to make it more faithful to what the paper version of the contract looks like, and we'd like to support electronic sign. Truth be told, they'd been having a look at the electronic contracts that are provided by MBA in Queensland, and they wanted what they'd seen there. But it wasn't just them looking at it. The builders were looking at what was possible too, and saying, why are you providing us with this really old looking, horrible interface when we know we can get something that's so much better? There are other requirements they had as well. Things like being able to take the front page of the contract and put the builder's own logo as an overlay on that, adding that to the front page of the contract. They wanted to be able to add additional files, so if the builder wants to upload the designs, maybe some images, they can include that in the PDF that's generated for the contract too. I had my own little shopping list. I wanted to drop the WK HTML to PDF support. I wanted to find a better way of creating the PDFs. In addition to that, I wanted to make the generation of the PDF much more debuggable, much more developer-friendly, so that when we're adding new CSS or adding some new way of doing things, we don't have to do that iterative, painful process. I wanted to make sure that it was secure and scalable. I wanted to make sure that whatever we use as much as possible is open source, and I wanted to make sure that whatever we built was going to work well on Ionstar's service, our sister companies hosting that we use. It needed to work with the docker containers that they use, so that of course limits in a good way some of the things that you can do. Initial ideas that I had. Well, if you want to be able to add fields and remove them and modify settings as an end user, web forms pretty much is your go-to, isn't it? So of course it was the first thing I thought of for how are they going to be able to edit the forms. In addition to that, I thought maybe we could use puppeteer to generate the PDFs. I learned pretty early on in my investigation of the possibilities that Google Chrome has an option where if you run it headlessly, so no screen attached, you can control it through a web socket, and in that mode it can generate PDFs. Of course that raises questions. How good is the quality of the PDFs that is generated? How good an outcome are we going to get? Is it going to support everything that we want? But it was a possibility to look at. Could we use CSS to generate the table of contents and the page numbering? I looked up specifications and did a bit of Googling and so on, and I found that there has been support in the W3 specifications, at least to draft spec, in browsers or for browsers to support CSS page numbering and all sorts of things like that since 2015. Sounds like it should be a go-to, doesn't it? Let's find out. And then we had the requirement for DocuSign integration. So I thought, okay, probably a REST interface will make use of there. Left that to one of our other developers to figure out. So we had a general implementation plan, something to work through and see what would work. The web form part of things came together pretty quickly and easily. Your normal sort of wizard progress bar that you see across the top of a web form when you've got multiple pages, all we've really done is some CSS on it and I think maybe a little bit of a change to the templates but that's it still just down the side instead and providing tabs. Of course with web forms you get out-of-the-box support for a wide range of fields so we were pretty covered in what we needed. Also composites were there. So I talked about multiple payments before. That was how we could set up being able to say there are going to be multiple values for this field and it's going to have a description and a number. We did some things with some custom modules though. We've got seven e-contracts that we want to implement and we want to try and minimize the amount of hard coding of stuff throughout this so that the MBA administrators can have maximum flexibility. But we also want to ensure that the code that we do use works across all these web forms consistently. We don't want to have something not work because in one form they added the workflow filled with an underscore and in another one they put in two underscores. So what we did was create a module called nestedwebs. What it lets you do is take one web form and embed it in all the others. Which of course means you don't edit them in the other ones you just edit them in the first one that's getting embedded. But it means that all of those fields you put in the one that gets nested you can consistently get the same outcome in all the places it's nested. So we've used that for putting in our workflow state for recording where the status of the contract is with DocuSign and with our payment process. We've used it for recording where the DocuSign's been used various other things that we want across all the versions of the different contracts. We used a module called web form workflows. I'm going to say straight out it's painful. It is designed mainly for a person who is a user to be making modifications to the workflow. It's not designed for you to be programmatically changing the workflow and so we've had difficulties with that area. We need to submit some patches to try and improve things there. We've also used the web form navigation module. It is what is letting us make those tabs so that you can just click on one to go to the page. It also makes it nice and convenient to do validation across the whole web form so that if you're coming straight in from the dashboard you can see what the required fields are that aren't filled in yet even if you're not on that page here. So a bunch of modules that we've used there. We also used a module called Config Entity Revision that I spoke about a few years ago at a couple of Drupal conferences ago I think so that we can do revisions of the web forms over time. It needs a bit of love itself I say as the maintainer but hopefully we'll get there. The next question is okay you've got your fields set up. How are you going to get them into a PDF? So what we did was we built a new interface in the a new tab in the web form configuration and we created plugins and so we've got four kinds I'm remembering correctly at least three kinds of plugins that we've created. One is a plugin that just lets you upload a file a PDF and that's going to be included in the generated PDF. Another one is a plugin that lets you fill out a WYSIWYG so this is where you're going to put in the content that can be changed and that will come out from your web form. A third one is that the builders will be allowed to upload images and I talked about before in PDFs perhaps of their own so we want to be able to embed them in the PDF that's generated. That plugin lets them do that. I think that is all four of them. Yeah rocks like this. Haven't forgotten anything. So then what does the individual plugins look like? Well this is the WYSIWYG one which I'll mainly show you because it's the most interesting. So you can see that it's pretty standard CK editor plugin but we've created a couple of additions there. There's a drop down up the top that lets you pick tokens from the web form so the fields that you've created you can select them from the drop down and insert a token substitution and where it should go in the web form. So if they've typed in the builders address this lets you place the builders address in the place it's going to appear in the PDF. That's the examples up here with the square brackets that maybe you can see or can't see. It also lets us do inserting templates. So we used a template module that lets you insert a bunch of code or content quickly that we could use that for tables that we're going to look consistent throughout the PDF that people would want to add for different purposes and then also styles. So we needed a way of saying at this point here start a new page. So this lets us put in a bit of text and say select the style for it and when the PDF gets generated that's how it knows to create a new page. We've also got one for a table of contents entry and so because your entry in the table of contents might not match the heading that you actually want to display in the text at that point we've made them separate things so your table of contents can say foo and the heading there can say far and then it won't matter. But then there's the question of how are we going to do the actual PDF generation. We've got our content set up how do we get to the PDF generation? Well this is where that headless chrome comes in. So you've got whatever you call as in Drupal that you want that you want to get a PDF from. What we're using first is a module called printable. This is another module that I've maintained for a little while since I worked for Solstice some years ago. I've extended that so that it adds new support through the PDF API module for a module called Puffpeteer. It's a bit hard to pronounce isn't it? Puffpeteer with PHP in the middle of it. My take is Puffpeteer. You can come up with your own if you like. So that's a set of code that sits in the vendor directory. Puffpeteer or whatever you want to call it talks to node starts a node instance. And then node talks to Puffpeteer itself which talks to Google Chrome and Chrome we feed the credentials and the URL we need so that it can then come back to Drupal and get the content to create the PDF. Bit of a convoluted looking procedure isn't it? It's almost amazing that it works but it does. I tried this very early in the project. I wanted to establish that it works before doing any of the other work because of course it didn't want to waste the effort on getting it going if it on doing everything else if this wasn't going to work. It's not the only thing we're using though. We also have a bunch of command line tools most of which we already needed for the existing PDF generation. So it's things like QPDF and popular new tools as well. We also needed some libraries. So we're using the iMagic library to do stuff like resizing the images, making sure they come out of sensor size in the PDF and also to do that overlaying of the builder's logo on the on the front cover that I talked about before. As far as PHP extensions went the main addition that we needed to get all this going was the PHP sockets extension. Everything else we already had in from what we've done with WKHTML to PDF right of this. What does the node support for it look like? Well this is simply adding four requirements in package.json in the top directory of the repo. So it's out of the dot route. Pretty straightforward and simple. It was a lot less painful to set up and get going than expected it might be. And that's all the code you're going to see by the way if I remember correctly. I think I've already mentioned some of these things. So yeah the normal libraries for PDF generation that you would need stuff like fonts and so on. PHP sockets, iMagic, QPF, yep I think I've mentioned all of those. So the process of actually generating the PDF. We've got our plugins that we've previously set up. One's got the PDF for the the cover page, then we've got our WYSIWYG, we've got some attachments that the builder's providers and we've got our appendices which is just another PDF like the cover page. We created first a method in each plugin that lets us create some HTML content for for Chrome. So for the cover page we want to tell it how many pages it's going to have in that PDF that it's including. So the call is just generating some empty sections each one's going to take a whole page. Then with our WYSIWYG we need to take the content and do things like replacing the content of the web form submission into your into your tokens that have been entered in the WYSIWYG. We also need to take our composite. So if you've got your builder's list of payments that they're going to provide in our editor we've made it nice and easy for the administrators. So you don't have to enter five spaces let's say for there to be five possible payments because of course you don't know how many payments they're going to be for each one. So we have in our PHP code some scripting that adds looks at how many instances of the composite field there are on the submission and duplicates those lines to create the extra instances that needed. So it makes it much easier for the end user to set up a contract. We also need to somehow be able to tell DocuSign this is the place in the contract where the person needs to be able to checkbox or this is the place where they need to be able to put in their signature or this is a place where there are two options one saying yes I've done something or no I haven't and the person has to be able to pick one or the other and it has to be required. So as part of our rendering we need to put in tags for DocuSign to be able to do that. Adding headers and footers of course people like like them so we've got to have a way of doing that all of that gets handled in this first step of rendering the submission and then there's pagination. I mentioned the possibility before of using CSS to do the pagination turned out that wasn't a possibility even though the spec has been there since 2015 I don't think any browser implements it like it's written very little of it is implemented at all and even and since of course we're using Chrome we're really uninterested in what Chrome is doing anyway and unfortunately it's not supporting it. So I had to find another solution. My solution ended up being using a live JavaScript library called page.js. It had some issues and one of them was an important lesson I learned that when you're telling page.js to do the pagination you must supply it with the style sheets. It's not enough to just have them added into the HTML that's pulling anyway and one of the parameters to say when you say do the pagination is style sheets and you have to give it that I can it gave me a good few hours at least I forget how long now of grief try to figure out why were my pages not coming out the way I expected before I learned that that was the problem. One of the other things that we have in here is the making it easier for a developer to debug things so because Chrome is having to access this all through a through a browser of course to a URL that means the developer could potentially go to the same URL doesn't it and and visit the page which is fine for debugging but you don't want your end user to be able to go to that page and get the content of the PDF without buying the support that you're preparing commercially to either do you. So the way around that was to implement CSRF tokens so when we're creating the PDF we give Chrome a token that it can use to access the content and then as a developer I can turn off the use of those tokens so that I can do my debugging check out the CSS figure out why something's not appearing how it is but still have it nice and secure when it goes to production. Table of contents generation was done via the page.js library as well using some CSS but it's CSS that page.js goes and looks at and figures out how to generate table of contents properly couldn't just do that automatically because of the same reasons that pagination wouldn't put it. For the other pages again we just need to know how many blank pages so this then gives it gives Chrome one great big long piece of HTML that it can generate the initial version of PDF from but it doesn't yet have in it each of the pages that are on the left and the two sections on the right so then we need to do some post processing after it has created the initial version and so then we're just using the PDF unite tools like that command line tools to take our initial PDF and substitute in the bits for putting the logo in we're using iMagic as I mentioned previously to first overlay that to build his logo onto the PDF a copy of course so we don't go sticking with all on one after another as we build different contracts and then we insert that into the content on top of that the WYSIWYG needs no further processing that content and so that's the way that we're building the PDF the docuSign integration I'm only going to mention this briefly because the focus of course is mainly the PDF building I mentioned the token replacement earlier that's the way that we position the fields and say what the fields are um we're using a REST API as I initially expended to talk to docuSign that has docuSign can make callbacks and in turn back to us to update the state of the the signing so that means that we don't then require that each builder has their own account of docuSign they can see the status of the project perform whatever actions they need to entirely in Drupal it also gives us the possibility of doing a integration with docu with Drupal commerce the way we've done things so we were able to get them to pay a different amount if they're using docuSign or not and if they canceled the signing partway through and come back and they need to buy you envelope from docuSign we can take care of that as well so what does it end up looking like we our generated PDF looks very much like the original paper copy that MBA sell this is what it looks like in docuSign so you can see the person is partway through signing they've got a little prompt there here is the place we need to sign they click they can do a signing using a tablet or use the provided uh default that docuSign provide the content looks just like it would be on a paper copy so MBA were very happy with the way it turned out they can also have sections that can have checkboxes so here you can see the person is required to tick each of those checkboxes down the side to say yes these things have been provided and then at the bottom here they need to pick one or the other of these and sign them to say that they that they have in this case what it would seem that complete copy of all the documents that relate to it so that's the end of my talk do people have any questions yes are the pvx generated accessible do they make weekend accessible i have not tested that i'm not sure probably not there for but we we could probably do work to make them accessible because we've got complete control over the hd not okay i work i'm a government and obviously any people who have generated files have to be accessible yeah government standard and now obviously we can't stand it as well so that can be yes yeah yeah it's not something we have had to focus on yet but i'm sure it will be doable given however much else at pinch you um you have to spin up that note service every time you use it or background um i have created a patch so that it can just stay running in the background at the moment we we're not using that and we're just putting it spin up each time we'll see how much it gets used and whether it becomes an issue but it's relatively quick to do the pdf generation the old system takes a few seconds this probably is about the same so pdf generation is not fast anyway and we've got a lot of processing to do in there so um hey thanks for making um do you just come here to brag about it or you can share properly that whole process of how you do things with the community with oh no we're not we're not just here to brag about it as you know i i like to contribute as much as i can that's true and i'm working with mba so that we'll contribute as many of these uh what custom modules we've made as we can yeah i must sound like a very complex thing you will take like a knowledge base article something to explain everything yeah yeah certainly want to contribute them and some of the modules have been explicitly designed so that they're stand alone and contribute so much for the talk with you talk a little bit more about the integration with the document and how you're able to position those a few science specific fields yeah it is an interesting problem because i found that yeah that their suggestion is that you put little tags in with unique pieces of text and then when you are sending the API you tell them what the unique pieces of text are to look for but the difficulty is laying things out so that those little pieces of text don't distort the appearance of the rest of the p yet so i think in the end i've been a while since i did it so i don't quite guarantee that i remember this correctly i think i chose to use a very small font size like point one of a pixel or something and then also either making it transparent or yeah and i think it was transparent as well the font size being important so that the size of the text doesn't go pushing other things out i tried absolute positioning that sort of thing um and was not successful in using them to to achieve mike hello excuse me uh thanks for a great talk i have two questions uh the first is you mentioned that you went through the process of doing a bit of a critical concept to make sure everything would work the way that you hoped it would before you did the build yeah having now gone through and done the build and having this live in production what would you do differently what what what do you wish you had approached differently and the second question really quickly is how many hours did this take you okay um let me first say it's not in production yet we had planned that it would be by the time i did this talk but uh things have delayed it and so it will be in production in a couple of weeks hopefully um maybe next weekend i've forgotten the details uh things that i wish we'd done differently i think i can honestly say i don't wish that we'd done anything differently i'm happy with very happy with the option yeah you don't often say that um how many hours most of the work as you probably gathered has been has been my time on the project but i've had at points i've had at some points i've had two developers working with me sometimes only one it's i wouldn't want wouldn't know how many hours i've been calculating but i think i've been working on this pretty much since january maybe maybe just single um so it has been a lot of work yeah yeah but it's worth it they're really happy with the outcome and and i'm really happy with the outcome so are they now the envy of other master builder associations well it's not in production yet so we'll see the the queensland one does look really good i've only seen screenshots of it but the cream one important difference is the queensland one uses a create system to pay for your your um contracts you'll be a docusign part of things whereas here we're sending them to um Drupal commerce and so they can make a whole bunch of contracts and then put them all into a card at once and pay for them once together but it's not a copy of credits so maybe that will make the difference between them being the envy of not well we'll see i don't know how the queensland one is built thank you all for your time