 So my name's James and I work with Drupal These things always seem like some sort of support group where you go. Hi, my name's James And if you've been working for Drupal with Drupal for any amount of time Sometimes you do need to be around a group of like-minded people going help me Anyway, this presentation is gonna help me presentation. This is something that where Catalyst have worked on with one of our major clients to make Drupal the center of a Web of different applications So hence why the title slightly different to what's on the it's it's an integration hub, but anyway change the title Okay in the beginning there was Drupal and it was good Well, it was okay, and then suddenly design fuse and forms and panels and everything else and it got better You know it's Drupal is a framework. It provides content management libraries use of management Comprehensive API. We all know this sort of stuff Large 30 pod third-party module said if you've ever done a contra module, I thank you From the bottom of my developer heart because it means that I haven't had to However, sometimes Drupal needs to play with others most of the time when you launch a Drupal site It's pretty self-contained. You have all your user management or your content management everything else all in one box The project that we had to work on however Had to talk to a number of different services Moodle being one of them and I'll go through the stack or the horde as we like to call it So we had Drupal as a CMS Moodle which is an open source. Who's had a moodle before? Okay, so I don't need to explain that So we had Moodle as our LMS. We had a proprietary student management system, which was Developed by another company based out of Geelong We had yep as soon as I figure it out Where's that? Okay, what I might do is just lean over a bit. Is that a bit better? Okay So Drupal was a CMS Moodle was a LMS. We had a student management system developed by a company out in Geelong two different payment gateways Salesforce as a CRM Simple SAML, which is a lie. If anybody's dealt with simple SAML it is a lie as our identity provider and Amazon Web Services and they were all meant to present themselves as this single platform and The site's been running for a year and a half now and the users don't know that it they're different applications So we've done that pretty well, I think However, what we decided was that we need a control center We need a we needed a spot where we could where the functionality was reaching out to all the different things but on one spot and Drupal presented as the ideal. I think the volumes come up Okay so we decided on Drupal as being the core of this sort of content management system and This integration area because of two major reasons the comprehensive API meant that we could build out against it and The large third-party module set meant that we didn't have to build everything We could extend on it, but we didn't actually have to start out from scratch on everything Whereas if we had to say if we decided on Moodle as being that point there would have been a lot more work in turning that into an integration hub Also these things Is that better or I turned off everything okay, we'll try that So Web services now web chick talk about web services this morning in Drupal 8 mother sounds about they're going to be doing web services on steroids Everything will become a JSON object, which is a joy to hear Hooks the hook system allows us to build Multiple different parts of say the user save area and trust me that'll come up You the user save an enrollment was a massive process We had to build and using user save hooks and those hooks around that area allowed us to build it in parts without having to build monolithic sections Believe it or not PHP is actually mature and As we see it's actually moving towards a much more object-oriented sort of design with the later versions as somebody has recently told me it's been Dragged kicking and screaming, but it is being dragged And drush who doesn't love drush? I mean I Personally live on the command line as a developer And being able to do things in drush means that I don't have to use the interface the interface is great If you're an admin, how many people are admins, but okay first off how many people are developers? Okay, how many people are admins? Okay So for all the developers here who know who know and love drush. Thank you But it is great in turn because not only can we use it when we're developing but we can use it when we're deploying Our deploy process actually use it utilizes drush to do feature updates feature avert's and all that sort of stuff and in automatic manner And it works When I first met features wasn't such a fan, but now I am Also the fact that Drupal is a framework It means that developing new features is a lot easier than if it was a straight CMS It's designed to be extendable. It's designed to be built on it's not designed as a Here you go. Oh, by the way, we can charge you You know several hundred dollars an hour just to build this extra bit because it's going to take months to do so so When we decided that Drupal was going to be the core there was a reason for that our entire setup is a As a learning system, it's a learning platform The user comes to Drupal enrolls in a course and then gets taken to Moodle. Let's say I'm very simple Except well as far as the user users concerned that is all that happens user comes in picks the course They want to go to fills out the e-commerce stuff, you know credit card all the rest of it Succeeds it, you know, and then that gets taken to their Moodle class and they can start learning In the background, however, there's a number of different services being utilized So this real-time comm story is pretty much what happens when a user is created now This is just creating a user not enrolling So we have a creating user new user registered with Drupal. That is stock standard Drupal registration. Okay, we've read a Drupal user However, Drupal then using real-time web services talks to Moodle says I need to create this user, right? Good, let's synchronize the IDs. So the ID and Moodle and Drupal now matches Then they go to the student management system and say hey, here's the Drupal ID of this user I need you to create him or her sorry So they create the user and then they send back some information about that user So that's with the identifiers then they go to talk to sales force and say, okay Here's a new user you need to create a lead for this user so you can track them Sale stuff around them then they send back in our ID and identify about that user So in this user creation, you're actually talking just the creation you're talking to three different services To actually complete a user creation system. Now. This is all real-time using web services We actually have this reversed as well There is a way that in sales force they can go in click register user and They can create and it basically sends the information to Drupal Drupal via web service Drupal then says okay No need to create the user in this in this students Create the user in this student management center and create the user and Moodle and then set manage that information And that's all done by real-time web services Now we have another type of integration which is queuing now Queuing is also known as batch processing Asynchronous any other name you can call it but basically there's a delay. There's an expected delay It's not expected to happen now So if we go back to here if one of these systems is not available Especially the student management system. We have a problem because it's real-time We need that information to complete an enrollment and if we can't don't have the information we can't enroll With this one which is queuing for an invoice This is after the e-commerce process has happened and after the enrollment so user decides to enroll e-commerce process user registration has happened Payment gateway says yes, I've got their money now. You can give them what they want We need to give them an invoice so student management system Which does the enrolling part of the system which talks to Moodle and says enroll them in this user Then has to put a puts a message on a queue which in this case is an AWS SQS queue Which is just a packet of information about the enrollments and on the rest of it Drupal via cron which basically just schedules every 12 15 however long you want to do it Says okay. Is there a message on that queue if there is I'll pull it down Process the information and then generate an email with the invoice attached and shoot it off So this is not happening instantly this happens every five minutes So the invoit the time sending off an invoice is not as essential as actually getting them into the classroom I mean it is essential, but it doesn't have to happen in real time So we've seen two different examples and it comes down to there's a number of questions about which one you'd actually pick when deciding Whether you're going to into whether you're going to use a real-time service or a cute service So, I mean the key one here is Can you wear a delay? Do you need this happening in real time in our example of the user enrollment? That has to happen in real time the user expectation is that we need to go Okay, I'm filling out my details. I filled out my credit card details have all my monies. I want the classroom now So that happens. That's why we do it in a real-time system The invoice on the other hand can come in five minutes. That's not something that needs to happen right now It also comes down to how much information you're trying to process at a time if you're trying to do large chunks of work like say Thousands of messages are hitting your system all at once in a real-time system Would it be easier and would it be better to actually batch them so that you can actually tell the system to say okay instead of taking off This chunk of work all at once try and do it in bite-sized chunks That's where the queuing and batching system really comes into its own because it means that instead of loading your system up and causing massive spikes you're actually processing things in a sort of Regular pattern in a regular way that sort of which leads on to the resources question So how many people use AWS or other cloud service? Okay, what happens when your load spikes? Okay Your bills go up for a start You're having to force you having to run up new instances to To actually support everything, but you want to avoid those sort of load spikes so You either spec it out so that you So you take into account load spikes so they say you get enough CPU to deal with a load spike But that's on an ongoing basis or you can if you're dealing with real-time Or if you queue it you can bring it down lower because you're only dealing with it in bite-sized chunks Which can be processed in that sort of smaller ram CPU load now The other thing is how would this how will the service effect your applications operation in the example I gave you of the real-time comm story if the student management system goes down when we're a bit stuck Because we can't enroll the user Okay, there is because it's real-time because it happens has to happen now We have a problem so Basically we need to account for that In When you're looking at how you're designing your service if you're looking at Your system and it's real-time because it has to be there It just means extra work in making sure that that service is always available It becomes 99.9 end to the power of infinity rather than five nines because as soon as one part goes down That's essential You're a bit burned Which leads on to tolerance failure? The other thing you have to worry about is what data governance requirements you have now the project that we were that we worked on is An education project it has certain government requirements when it comes to actually Storing data what sort of data you store How it's how it's stored and all the rest of it so we need to take that into account when it's processing this information Also, it comes down to a It comes out to taste. I mean through all that It can come down to what you're comfortable with are you more comfortable with dealing with Real-time services. Are you more comfortable dealing with cues? It's I mean as developers we all have a preference for some for one or the other I mean we've all seen the flame wars on stack on well basically any technical side Personally, I have a I have a leaning towards cues because it's much more Tolerant of failure because if you stack up instead of the message is dying if a service dies You've got a stack of them that can be processed later on so you can pick up again so We've we've services to deathmatch. This is basically just a list of Areas where the they Especially present cons, so I haven't had my second coffee. I'm not I Need three Okay, so web services real-time part of the page building Drupal So they are with the web services you bitch-strapping Drupal a lot if you're hitting it every time You're if you're doing it real time You're hitting the page you need to load up a lot of Drupal functionality and it's been done all the time You're doing it in cues. That means you're doing it when you want it to happen So you can account for that So the cues are the cues are lightweight cues are the messages are all stored off elsewhere All you need is a curl request to go and get it and pull it down See the cues are more highly available than web services I was a bit hesitant about putting that one in because cues are like every other service There is a potential for them to go down The thing about cues is the data will still be there when they come back up Whereas a web service goes down you just got packets hitting a wall and dropping So web services however incredibly widely supported So you know as we say Drupal already supports web services out of the box in Drupal 7 Sorry Drupal supports web services out of the box. You can build your own web services. There's a format to doing that There are multiple protocols. We've got soap. We've got rest. We've got any number of XML formats So they have Wait, they have status already. They've already been developed. Whereas cues are Not so much. I mean you have to build to support a cue if you want to say we had to build a Basically a library of call calls to say, okay, pull the packet down from SQS Web services are not particularly failure tolerant. I think I've gone over that enough But they also are a potential attack vector because you're dealing with Data which is being injected straight into your system without knowing without knowing where it's coming from you could validate But it's a much easier It's a much easier to use a web service to attack than a cue if the cue is particularly if the cue is something that you've set up with an expected format with that you're The only applications accessing it are the ones that you've said can access it. It's a lot easier Web services are a bit fiddly to test How many people have tried to PHP unit testing web services? Yeah, how many people haven't Okay, because it's a pain in the butt We've sort of got round that by cheating and I'll explain a bit later on about how we've done that But we cues you can test all to the house to your heart's content because all you do is you say here's the packet format That you're expecting here's the response that you're expecting if the what happens if the response is this And do that Okay, one of the things that we discovered is Once we had this integration hub set up. We thought okay now we need a way to determine All asset the health of all our services we can heartbeat all of them individually, you know Which is basically go out set up a whole bunch of Nagios checks which individually tags each one and goes through and says are you up? Are you alive? Can I connect? We could do that, but we decided that we've already got this Integration hub set up. We've already got this bit that says I can now talk to these services So why not take advantage of that? So we built a Drupal module, which I'm not I don't think has been released. I think something needs a whole lot of cleanup but What it does is it actually creates a series of functions which output JSON So and also creates a nice pretty page so at a glance we can go and say oh the student management system is down We need to find out why or Sales forces reporting that it can't connect for some reason Or even Drupal is talking to Moodle and Moodle is saying the user IDs are out of sync. We need to fix that You know these are all ten and we've added all these tests as we've gone along So we built this module we integrated now We have Nagios is hitting this hitting these web services these JSON elements and saying okay based on the contents of this JSON This service is up. This service is down. This variable was wrong. We can alert to that so we can now do Nagios checks which say This Cron has taken too long to run on a particular system, so Cron's died for some reason and The drip and or Drupal Cron or Moodle Cron or something like that stuff Which is a lot harder to actually do unless you manually go in write your Nagios check and so on so That's one example of taking advantage of this all this sort of web server stuff that we do Okay now some lessons learnt I'm going too fast on a Right, we're positive as I said before I prefer batching. I prefer Q systems because As I said if the real if one of part of the process system goes down then everything goes down I mean you can you can account for that But at the end of the day if you're if one of your systems is down you're losing data With a Q the data is still there and you can pick it up and process it again Batch processing allows you to schedule how you process requests which allows you to manage your resources Which means that instead of saying holy crap nuts We've got you know Spike loads of a hundred percent on all our instances because we've got a thousand messages coming in at once We can say okay. We have a queue of a thousand messages. Let's process them at ten at a time You know we can schedule it once a minute to go through just schedule instead of actually trying to do it all at once Debugging is easier debugging your web service means that you're trying to set up a Essentially a mock web service set up where you do the call and you're coding a fake endpoint Which returns an expected result Whereas when you're doing queues you can say okay. Here's the expected packet Here's the packet that you want and Just test against that Document all the things I know this is something that everybody says with every project But when you're running systems which are multiple elements as I said, we've got an LMS a CMS a student management system To payment gateways with two different payment gateways and all the rest of it They need how they interact needs to be documented Especially when you because we built a lot of this stuff from scratch in terms of the protocols that they use the packets Packet structures that they use they all need to be documented properly Because otherwise somebody's going to come along in six months and say What's going on and break something because they don't quite understand how it all works together? Don't use XML. I have a personal hatred of XML. I'm sure some of you also do Jason is pretty Jason is a lot easier to read and it's a lot easier to process So that again that comes down to a personal preference you may like XML in which case I apologize Sorry, you're not sorry Unit testing your services now. This is one thing that we did At the start of this project. We decided that when we were going to unit test We were going to unit test using PHP unit tests now this in Drupal is in Drupal 7 is not supported officially But we had a look at simple test and it really wasn't doing what we needed it to do So in terms of so we went through So web chick went through unit testing this morning on Drupal 8 a lot of that we've actually ported backwards So we've got a lot of the interfaces. We've got mocks. We've got classes the whole setup is now set up in In this projects and they are run automatically via a continuous integration service and they work they work very well They've also allowed us to do things like setting up proper web service testing Because we can mock up an interface which says yes hit I'm pretending to be a web service endpoint that you can call data from Make sure that you've got a graceful failover if you're going to web service it make sure that there's a way that it's not going to bring everything down That's one of the lessons that we really learnt Early on was that we needed to make a way that okay sometimes services aren't there You know, it's the nature of the beast Sometimes the service that you're relying on for whatever reason will not be available. You need to be able to say, okay It's not there. I'm sorry. We can't do that at the moment instead of saying Oh, I'm blessing everything to the screen and the user says it So you need to be able to say so in the case of say Salesforce becomes unavailable for whatever reason What we do is we say, okay. Yes, we've enrolled your user, but we need to In the back end So the user the user's fine. The user goes through doesn't enrollment goes to the classroom But in the back end what we do is we actually make a note and send a message saying okay We've created this user, but you're going to need to manually create the lead in Salesforce for them So that way they get everything synced up again So it's graceful failover Test your definitions are correct. I mean this is all basic stuff and that's it. I've gone through very quickly Oh, you apologize Anyway, that's catalyst. We're New Zealand based Sydney based Melbourne based do all the open source things there's the clients There's some the text we use so Have you got any questions? Yep No, the the project that we worked on was actually Basically a second iteration of an earlier project So there was a lot of stuff that we were getting for free for this because the work had already been done in terms Especially the Drupal Mural side that was already working. So we took that and said, okay now How do we extend it to work with everything else? So yeah, the Drupal Mural side was is all web servers based We don't do anything like drop into each other's database to do anything like that We do everything by web servers. So yeah, basically it was got this working. Let's see what else we can do with it So simple single PHP basically when you log out you log out of Drupal Mural So the student doesn't see the student management system The student doesn't see Salesforce or anything like that any changes that are made to their profiles addresses or that sort of stuff It gets communicated by a web service to the other systems Because otherwise Yep, yep We use so the prop so we use AWS SQSQs for our queuing system. The main reason being that AWS is a hell of a lot more scalable than using the internal system. So our internal system is All database based which carries its own load problems And if we can farm that off to AWS and say you deal with it I mean we can have thousands upon thousands of messages on the queue and it's a blip It really is. I I'd actually recommend it. I mean it does come with a caveat that it's an external service at some point. They may not be available But Amazon's pretty good at keeping up time Until it's not So we built a lot of I mean we relied on Drupal's underlying web services sort of core, but we built built a lot of it ourselves We for the e-commerce stuff, we use that a lot. We use Drupal commerce with coupons and all the rest of it In terms of the web services Yeah, we built a lot of that ourselves, especially the queue processing stuff because basically that is essentially a series of curl calls That we just say okay. Here's your SQS key. Here's the Process, you know go out get the message come back process it So I mean where we used a lot of the contrary modules is more in the content management side of things In fact content management is another example of where we're using web services as well I should have put that up there The course information is actually stored in the student management system So all the information goes in there all the modules all the units and they press a button and via web services They export that down into Drupal So all that information gets pushed into Drupal that automatically set up in Drupal saying okay Here's the new content. Here's start end dates for the courses and all the rest of it Moodle Moodle also talks to Drupal about course information about this bit it pulls more of the Content management side of the course information across so we have a student dashboard which says okay Here's your new next module But yada yada yada and that information gets pulled across from Drupal via web services Yeah Yeah Yeah So that's what we do with the course information Basically, so they do their bit and student management then they push it out Okay, and then we have a you know, basically it blats the database saying here's the course information replace the old stuff Here's the new stuff. We keep track of all the old stuff So it creates a new version or a new a new copy of the course information And we have all the old stuff in there so it for again the auditing purposes So there's you know government requirements around what we store Which one of those you recommend They are and it comes down again it see I was being a bit flippant when I said it came down to develop a preference, but In many ways that's an act that's an accurate description because as I said, I prefer cues I would much rather be able to work out a system that did the enrollments using cues rather than real time Because we've had situations where for whatever reason one of the parts has fallen down You know whether there's been a routing issue or a problem at their end and we've had to deal with that And that is bad for that bad for the business is bad for the user But if we can support basically have a seamless file over which says okay We've created this will enroll you in a few minutes because we're just having a little bit of trouble And but the user knows that they're going to be enrolled Rather than please come back in a couple of minutes and try again Yep, we do with it. We do with the sales for stuff So we do with the sales for stuff the student management system is a bit more complicated So because there's that whole enrollment stuff as well And so sales force is sort of considered a secondary thing. Yes, we can go back afterwards and rebuild They're their lead the student management system has to be there has to be working sort of an integral part It's the three core parts are Moodle because they got to learn Drupal because they got to pay and student management Because that's where all the other information lives. That's where all the government auditing information lives That's where all the course information lives Well, we rely on things like Drupal watchdog and stuff like that We can do things like error mess, you know error messaging to the admins emails If if the No, we don't do like we don't do local Q and we just say please come back later in a little while Yep user management starts in Drupal. So that's where the user enrolls. That's where the user registers Drupal is basically So we have a couple of different Strong sources of user information Drupal is where the user puts all their information because a user logs in user fills out their address information Then they get replicated out to the other areas, but we can push back From sales force say if they call up on a call center and say I need to change my details They go, okay, push click the da da da click pushes back into Drupal and pushes back into the student management system sort of In terms of basic user details. Yes, but there's a bunch of stuff called a vet miss Which is government required information around this sort of stuff that comes actually from a form in Moodle That gets pushed straight to the student in And that's where that leaves is in the student management system Authentication is Drupal. Yep, they log in via Drupal. That's it I find Jason easy to work with it's Yes, I think it's it's easier and quicker to work with a Jason packet and an XML packet It's not as heavy to process Personally, I like looking at Jason better than XML It's just easier for me to read and it's easier to define So you can and so yeah, it's just easier to use an XML Especially when you're dealing with okay, I'll be using rest are we using soap or we're using a New definition a new XML definition that we're setting up for this packet. Whereas with Jason we can just say, okay da da da da there you go, sorry Just write out a new packet structure in a way you go You don't need to build a validator for it. You don't need to do anything like that. You can just say does it match Yes Well, not okay, so we case the front end a lot Because we get a lot of people using the system But once you're logged in not so much with the casing Because a lot of the interactions are real-time interactions So if they go to the course they need to be able to see their their answers their stuff They don't generally live in dripple land once they've logged in once they've logged in they're off to Moodle. Yeah Yeah If the okay if the queue is lost as in the queue gets deleted or purged Then we have a problem Okay, we do need to set up redundancy in our queues The idea being that if somebody goes in and purges the queue, they're gonna get fired very quickly Yeah, no, I understand. Yeah, so basically sometimes, you know crap happens and Things go away and we need to back we need to work out how to do that. It's it's an ongoing process of what we're doing So there was a yes and no So we built this site This site's been running for every year and a half almost a year and a half now And there are lessons that we've learned about how I mean we've made changes since the site was launched We put more stuff in the queues we put more logging into queues as well We have multiple instances running so logging mean and we might input watchdog into queues So that we can pull that off and process that in other ways Rather than having to go through and try and do a drush watchdog show or whatever Still have while I said I would like to find a way to make Enrollment and lot and user registration queue based. I still haven't found a way that would make it give you the same experience of user enrolls user gets classroom So still investigation But it's a learning it's always a learning process and we are constantly moving on with you know, okay That's not quite working. Let's try it this way and we iterative improvements Sorry. Yep. No, they're pulled off first off the queue. So if then Well, one of the things that we learned was don't do into don't do one. So basically when we started you could do student info updates to the information from all three sources peppy So the sales force student management system and Drupal We decided that's probably not a good idea. So we remove the student management system as a way of actually pushing out student information Because the the student management system should only be getting student information from either sales force or Drupal so basically If Drupal pushes information it goes to both if sales force pushes information it goes to both But so the student management system doesn't actually push anything It otherwise. Yes, we'd have to have some way of actually doing it to which is the canonical source of good Yeah Yeah Look, we've run two different projects based off the sort of Drupal Moodle project that we we originally built the first one Handles a lot of concurrent users. It's a one of it. It's a free MOOC sort of set up and it handles a lot of users we got something like 500,000 registered users with maybe a thousand or 2,000 concurrent users that are saying at any one time And It handles that load quite well I mean we have spikes especially when we do rollovers for enrollment when people everybody goes to the class at once You know when you get 30,000 users go, yeah, I can learn you've got to be able to support that But you know it sort of patterns out and because we do The way we work is we have an instance with Moodle and Drupal on it and When we spike we add new instances and using the load balancer. It all works quite well Yep We use ops works and AWS for the Drupal instance and it is a Drupal Moodle on the same server sort of set up So when it's when Drupal and Moodle are talking to each other, they're not talking to each other via The urls are talking to each other by a local host so It's that way the communication happens without any possible interruption Yep No, most of the reports come out of the student management system and that's because they're actually set up to handle all the government requirements So again when I mentioned a vet mess that's all the government required information. So when you do a TAFE course or some sort of Technical course, there's a questionnaire you fill out which contains a bunch of information. I you know, do you speak English at home? Are you of Aboriginal or Torres Strait Islander? Heritages that sort of information that all is required information by the government so that they can better target Their information their services and that's all handled by the student management system Any more questions? Well, thank you very much. Oh, no, sorry. We got a That's what we use simple SAML for so simple SAML gives you a session outside of the Drupal Moodle session set up And they all and they all pull from okay So this is sort of less of the web services and more in the user stuff But basically what we do is we use men case to store our session information and that what we have a single men case instance Which handles all that information But so basically we go through we say okay, this is a session ID. It goes okay check with them case. Yep That's valid regardless of what instance you hit So Drupal itself on the server isn't doing session information. It's just asking men case if that's a valid session To be honest, we didn't have time It was one of those okay, we're gonna do this. Oh look, we're gonna do it now But yeah, we could quite possibly port it. It's at the moment our module is essentially a series of curl calls But we could quite ease well when he says quite easily he says, you know with difficulty We could we could put that to part of the core API as well If if there isn't already an sqs one that somebody's popped up in the last year and a half I'll have a look but yeah, but we we had these particular set up that we needed to do so Any other questions or remarks comments? All right. Well, thank you very much