 So, thank you very much for coming here. This is Mike's in my first time presenting at DrupalCon, so we're noobs here, so hopefully we'll have some good stuff for you. So this is going to be kind of an architectural overview of a lot of the different things and considerations in actually using Drupal to do digital science. Just a quick introduction. My name is Adam Weingarten. I'm a technical architect over at Acquia in our professional services group. So I help do discoveries, estimations, project executions, supervising rollouts, and just keeping large scale websites up and running and have some interesting stories to share today. And I'm Mike Madison. I'm also a technical architect at Acquia and I'm particularly excited about this talk because I have been living and breathing digital signs for the last six months. So all signs all the time and I get to share some of that with you guys here today. And he has me to thank for it. I told Mike that when he started this project he would either love me or hate me for the architecture I left him with at the beginning of the project and he's still talking to me and I'm not quite sure why, but yeah, so you'll get some of the information about that. So why are we here today? We're here to talk about some next generation digital experiences that go beyond the normal standard web browser. So you're no longer accessing things necessarily with a keyboard, with a mouse. You aren't just able to reload things. We're hitting data and consuming data in ways that we're just not used to for those of us who are web developers. And we're not used to thinking about it. So what could those different things look like? So you could have a voice interface, something like a Siri, a Amazon dot, I love mine, Cortana. So all these voice aids have become really, really popular and we're seeing a huge emergence of them online and with different retailers, different vendors, different institutions are all rushing to create these. We also might have a touch screen on a car or an airplane, like another type of digital experience that could be powered by a website, a mall kiosk. The other day I was walking around the Pentagon City Mall outside of Washington and they have signs all over where you can actually interact with the sign to get information about what's going on. So they're not just printing anymore, you can actually go reach up and touch. And it's a little bit different than a normal browsing experience. And finally, you have public transit. So you consume information every time you go on to a train, a subway, a bus. That's another place where you see information that could be powered by a web server. So ultimately, why are we really here? We're not here to talk about digital experiences in the abstract. We're here to talk about signs. That's the purpose. That's what we're here for. So today what we're really going to try and give you is sort of the overview of the entire process, as Adam said, this is more an architectural overview than like a developer's deep dive. So if you don't have a great deal of experience developing for Drupal or PHP, that is perfectly all right. What we are going to look at though is quite a long list of things that we've had to take into consideration as we're actually building a system to do what we're talking about here today, right? So some of this is going to be very Drupal centric. Some of this is going to be much more sort of broad and general guidelines. But the important thing is that if you either have a digital signage system now that perhaps isn't powered by Drupal, or if you're thinking about moving in that direction at some point in the future, hopefully this is going to give you a solid starting point that you could go out and approach this on your own. One of the first things that I always get asked when I sort of describe what I'm working on in this sort of scenario is people ask me, well, why the hell are you even using Drupal? I mean, that just doesn't sort of immediately click. And that's a valid question, right? It was my first reaction when I got on this project. Someone told me we're using Drupal. I was like, why are we doing this? Why aren't we doing something else? And we're not just talking about this because it's DrupalCon, right? I mean, there is a legitimate use case for using Drupal in this picture. But, I mean, just to play devil's advocate for a moment, right? I mean, if you really wanted to, you could go out and build a really cool angular app and just do it in angular, right? Or whatever else. I mean, that is a totally legitimate use case, as is any other modern website, right? I mean, you don't really need Drupal to build a website. But, as we all know, each of these technologies gives all of the developers building them a really slick experience, but all of the users that actually have to then maintain what you're building a really terrible user experience. And this actually is a fascinating problem with signs because literally the only users that are going to be interacting with anything other than staring at your sign are your people that are managing the content on the sign. So in many ways, we're talking about the most critical section of people here being able to log in and actually manage these devices. So just building them something in one of these other technologies while perhaps easier on the developer is ultimately not going to solve any problems for the end content people. And, you know, I'm sure that your stakeholders won't be very happy if you have built them a sign that can't ever be changed. If all it does is do one thing and then the next week they decide that they want to do something else, they won't be very happy if it's going to take months and months of development to do the next other thing. They want to be able to have some flexibility and you should think about that. That's why we're doing it digitally, right? Yeah, that's right. We could just print it out. You might be onto something there. So, you know, obviously this is kind of an obvious statement, but I mean Drupal does provide great user experience for managing content and even in a headless world, right, if it's about React or Angular or something else, actually displaying the end result, there's a lot that your content creator can do by logging into a Drupal UI and using those forms. So, it's important to go back to why do we use Drupal to manage content again? So, and most of the people in this room, I expect know this information, but we're reiterating it mostly so that you can take it to the people that you have to talk to and you have to justify to remind them why you're doing this and why you're taking this technical approach. So, first thing that we do with Drupal is we do content modeling. First thing you do when you create a new Drupal site, you go and you create content types. Drupal does a great job of that. It has access controls and permissions, so you can control who can do what to which node that you have created. So, some people can edit, some people can't, some people can publish and make things live, other people cannot. You have revisioning. So, one person can get started, they can create a draft, the draft is ready to be published, maybe at a certain time, so you can keep tabs on all those things. We have a great diverse community here, so it's no surprise that translation is really, really important in Drupal and especially in Drupal 8. So, we have some really robust tools in D8 to help us with those. And redoing all of these things that Drupal gives us is really, really expensive to do someplace else. So, the minute that you start with Drupal, you already have a leg up on a lot of other solutions. At the end of the day, it lets normal people actually interact with websites. That's why we use our content management system and that's why we're not just editing straight up HTML anymore. I mean, this is a silly slide in some ways because you guys have all seen this, it's really obvious, like this is a node edit screen in D8, but the fact that we have a revision message, we have a wissy wig where users can do stuff, we have titles, we have drop downs, people can upload files, that's all really, really powerful and ultimately, all of this stuff is about content. So, let's look at an example. So, we're in Baltimore, I'm from D.C., so I figured I'll grab the D.C. subway system as a great example to talk through. So, you look at a map here and there are different color lines. So, our content model starts with subway lines. The lines have direction, you can go in two different ways. So, if we're looking at the red line in your local, you might go to Shady Grove or you may go in the direction of Glenmont. You also notice that there are stations. So, that might be another part of our content model. So, you might go to Metro Center, you might go to maybe the Smithsonian, maybe, you know, go check out some museums, maybe some of you did the weekend before, maybe some of you will do it this weekend also. Or, you know, go to Farragut North and I don't see the lobbyist up there. And then finally, trains have, train stations have platforms. So, maybe you have to go to one platform or another, maybe different platforms serve as different lines and different directions. So, all of these things make up your content model. And D8 makes it really, really easy to syndicate all of this data using services. So, you know, you can use the built-in services or you can also use, you know, something like JSON API to get the data out to some other system like an Angular or something else to consume it. So, as with the modern web, we're constantly seeing, right? Drupal is not always going to be the only data that you have, right? I mean, you may have some additional data coming from other places. And I mean, Drupal may not even know about some of that other content, right? I mean, Drupal may not even touch it. You know, that's okay. But the really cool thing about the content model that Adam was just talking about is you can still use Drupal to inform the parsing of that data. So, you know, let's imagine for a moment that we might want to use, right? So maybe we're going to go get data from social media. That can be incredibly dangerous if you're not appropriately filtering and reviewing it before it goes up on your sign. There's lots of fairly famous debacles in that arena. But still very powerful data that can be very useful on a sign. You could be talking about train or other public transit data. You could be talking about weather data. Stocks, top news, et cetera. And each of these different items could actually represent a data provider for your screens. And in fact, the way we handle this in Drupal is we literally have a provider API that can go out and talk to each of these different sources at some interval and get the data from them. We parse that data based on our content model, meaning that if there's news that is affecting a particular area, perhaps it's news that is affecting a particular train line, in Adam's example, we would want to ship that data to an appropriate screen or set of screens based on our content model. We happen to be doing most of that with object-oriented PHP, but pick your poison. You can do that in a Drupal module. You can do that with Node.js. You can do that with many different technologies. The important thing, though, with all of those providers, with all of that data, is that you want to pull data for the entire U.S. if you're just trying to show the temperature in Baltimore. You don't necessarily want to show news for some other convention than Drupalcon to people that are attending Drupalcon. So data without context is meaningless data. So this is where that Drupal content model can really come in as part of that the parsing as the providers actually go out and get the data. The content model can actually inform that parsing makes sure that it gets to the right place. So if we're talking about our trains, let's say, for example, we have an arrival feed that's actually monitoring the red line as it moves through Washington, D.C. As that train is coming into a particular station or approaching a particular platform, we can make sure that the signs on that platform get notified, hey, the train is three minutes away. It's two minutes away. It's arriving now. That data is only relevant to that platform. And we can use all of this to make sure that the signs know when some data, whether it's the arrival or news, or something else has changed. In our example, we're going to go a little farther and just talk a little bit about how this works. So we actually use Drush to go out and get our data. So each of these providers gets wired into a Drush script. Drush runs and processes. We'll talk about that in a moment. Since we're using Drupal to go out and get all of this data, that lets us store all of our credentials on the Drupal side. That's much more secure than having 10,000 screens that have credentials embedded in them. And it also lets us be very smart about our APIs in terms of rate limiting and how many times we actually go out and ask for this data. We'll talk about that here in a little bit. Caching is both critical and incredibly difficult when you're talking about signs. It's critical because you can't necessarily hit an API 10,000 times a minute and expect it to stay up, but a sign is only relevant if it has valid data. So the whole point of doing like a CDN or a caching model is to increase performance, but it does that by storing data. And if we're talking about timely data, a cache kind of immediately defeats that purpose. So you want to be smart about where you cache. You will have to cache some things, but you can't cache everything. Or again, you may as well just print out something and hang it on the wall. It's not going to be up to date anymore. And then as I said, we do parse and normalize the data to make sure that the right data is getting to the right places. Question I got fairly frequently is if you have CRON, how do you make sure that Drush is actually running fast enough? For those of you who are more technical in the room, you run CRON like once a minute, best case. What we actually do is we kick off our CRON jobs and then we let Drush actually take over from there and just loop, loop, loop, loop. And every time we do a complete loop, Drush is actually going out, getting the data, loading our cached content model, separating the data based on that content model, and then pushing data out to the appropriate signage. And I do want to throw a warning out there, right? This assumes that what you're doing can run very quickly. If you're parsing data for thousands and thousands of screens and you want that to happen in like near real time, you have to have a script that can run very, very, very quickly. I think we're running ours 20 times a minute just for context. That's for about 3,000 signs. So speed is critical. And again, caching can help you in this scenario. So ultimately, one of the bigger challenges is getting data to your signs. So, you know, we have all this stuff in Drupal. How do we get it out? There are two ways that we can do this. We can push or we can pull. So the most traditional model on the web is pull. So you initiate a request. The request goes to the server. The server then responds. And, you know, we can generalize that for all the signs. So you have every single sign can just be on a loop that just goes and asks for help. Pull sucks a lot. There are many reasons why pull model is not ideal. There is a lot of traffic. Imagine that you're on a road trip with your kids who are hopped up on sugar and they're really, really bored and they're asking you, are we there yet? Are we there yet? Are we there yet? That's what pulling is like. So you have your signs knocking on the servers every second saying, do you have something new? Do you have something new? Not ideal. It means, first off, that the devices or the kids are running kind of hot. They're excited. They're doing stuff. They're taking a lot of energy, sugar, whatever you want to call it. So it puts more load on the devices. It also puts some load on your servers. They can only handle so much attention before they snap. And this can actually mimic a DDoS attack, ironically enough. If you have a broad enough network of servers who are all polling at once, maybe your web servers aren't robust enough to handle that much traffic. It also makes it really, really difficult to employ a WAF because the WAF may not be able to distinguish between the polling model and legitimate traffic. From its point of view, you may have a whole network of bots that are just hammering you. So that isn't what we want to do. So what is the goal? Ideally, we want to send data from lots of APIs to lots of devices somehow in near real time. How do we do it? So this is what it would look like. We would get data from Facebook, from Twitter, or weather, or train arrivals. We process it somehow with that content model that we talked about. And then we would send data out based on context. So if Shana is waiting on the red line, we might send the screen that she's looking at, a notification of the latest red line arrival times. If Moe is waiting on the blue line, as there are updates to the blue line screen, we might send those out. Let's say that there's flooding on the blue line. Maybe we could aggregate some information from social media and send that out to the blue line screen so Moe knows what different things he can do. Now there's problems with data in real time. The first one is caching. There are two major problems in computer science. One is how do you name things? The other is caching. It is always a problem. So if, you know, typically on a website, we recommend minimum 10 minutes, a 10-minute TTL for cache. Generally, often it's an hour, six hours. With Drupal 8, often we're even going higher to 24 hours to a year. It gets in the way if you're trying to do real time. So that means you can't cache. You can't buffer the data from your web server as much. Ultimately, you want the science to be current. So you want them to get the latest stuff. There's no interactivity. So they have to just work. If something goes wrong, you can't be there to hit the refresh button. It has to just happen without you, reliably. If it doesn't, people aren't going to be very happy with you. So what's our solution? So we are looking at... So we use web sockets to solve this problem. So what is a web socket? So a web socket really is just like a TCP connection. It's a two-way connection. It allows this near-instantaneous communication between some server and some client. It can be shared between devices. So if multiple devices are subscribed to a web socket network, you can broadcast data out to them. And it can be bidirectional, too. So some services like Slack, for example, can apply on web sockets to send data back and forth. So... And the most important part is it scales really, really well. And there are a lot of providers that do this. So there's PubNub, there's Pusher, there's Amazon, there's Firebase, which is Google. And, you know, you can use Drupal to send data through one of these web socket channels. And it can then multiply that data and send it out to all the relevant devices it needs to. So it's really important to get data to all those people out there. So let's take a look at this in a more practical example. And we're going to happen here to talk about the Amazon Internet of Things service. We've got some experience with that. It may not be the perfect fit for everybody, but this model is pretty consistent with web sockets overall. So let's say we have some number of devices that we want to install over a geographically large area, maybe a big city. Each of these devices will form a connection back to that web socket. And in the Amazon speak, they actually connect into something called a topic. Topics are that sort of bi-directional communication mechanism that Adam was just talking about. And you can actually have multiple signs connected to a single topic. So if you have a couple of signs that are right next to each other or close together, they can all be on the same topic. So when you push the data to one place, it instantly propagates to all those signs. Or you can have each device on its own entirely separate topic. That way you have more specific targeting for your messaging. We already talked a little bit about Drupal going out and getting data. So in this model, Drupal is going to subscribe to several different data APIs where it's going to be doing that constant parsing and receiving of data that we were talking about a moment ago. And then Drupal is actually going to push JSON data out to the web socket. At this point, Drupal is sort of your gatekeeper of where it's going to go because we've got our content model that tells us what's relevant. And we will push out to the topics from Drupal and the topics themselves will distribute to the signs. So that actually might look something kind of like this. So in this case, we have five devices. Those five devices have subscribed to three different topics. One is that we have some overlap, some devices are sharing topics. And then we have two different data sources. And you can imagine as time goes on, these data sources are going to begin spitting out new data. And depending on what you're subscribed to, depending on how frequently you need it, that could be once every couple of days, that could be once every five seconds. So obviously your data is going to dictate how often this changes. But when it does change, so sorry for the red on black here, the source one in this example has actually updated. Drupal is going to get that data. It's going to parse the data. In this case, it's going to realize that topics one and two care about that update, but topic three doesn't. So that new data will be pushed out to devices one, two, and three via the first two topics. And devices four and five won't change. Whatever was showing up on them would stay the same. And as these two sources start propagating additional data, as Drupal parses the data, now Drupal sort of dictates which tube the mail gets shoved down. And as soon as it goes down that tube, it shows up on the screen. And you might also think about it as like a television channel or a radio channel. This is a tool that we can use to broadcast to different places. So maybe one message gets sent across channel five, and maybe a different one gets sent across channel seven. So depending on who you're tuned into, you get a different story. The important thing about this model is that it's really fast. Like really fast. You can get data from the source to the device in a couple of seconds usually. So granted, not all data needs to happen that quickly. That's fine. But it can if you want it to. This also gives you a very high degree of granularity. So let's imagine we all get up right now and walk out into this main hallway and somebody's installed 15 signs down the hall. In this model, we could actually have an evacuation plan for the convention center that is highly granular. First sign says emergency evacuate. Second sign says go 100 yards. Next one says go 75 yards. Next one says go left. Next one says go down the stairs. Like you can actually target each individual screen for that messaging. And remember, you've got Drupal behind it so that you don't need a developer to do that, right? You turn it over to your communications team. You turn it over to your emergency management team. You turn it over to somebody that's never used Drupal before. That's fine. They still have that really simple user interface where they can go through and use this incredibly complicated model. The downside to this is we actually have a new problem. And that problem is that for this to work, Drupal actually doesn't just need a content model. Drupal also needs a device model. For Drupal to know which of those signs gets which of those messages, Drupal needs to know about all those signs. So in our content metal so far, we've talked a lot about Drupal parsing and receiving the data. We've talked about actually setting up the content model itself. But we haven't really talked at all about what these devices can do or where they should go. So there's three big questions that we need to think about with the devices. So the first one obviously is which devices actually care about which data. The second one is what data should that format be presented in, right? If we're talking about this cell phone versus that, a big screen, that format's going to change pretty significantly based on the device size. And finally, we've got to make sure that the right devices are getting the right data. We're going to talk about each of these in a moment. So let's think about which devices care about which data. A few things to think about here, right? So if we're still working with our DC Metro example, if you're talking about a screen on a platform and there's trains going by, what direction are those trains going? That's important, right? We need to have context for that data. What platform is the device on? We don't want to tell somebody about green-line trains when the platform you're on only services red-line trains. That's kind of a no-brainer, but it's important. And what station is the platform in? You know, maybe certain stations are going to shut down. I mean, this has been happening for what, the last year and a half now? Is there actually fixing tracks and trying to stop things from catching on fire? Where they will actually shut down stations and do maintenance for a period of time, and then they'll start it back up again. So you might actually need to send a message to an entire station. Or, you know, perhaps an entire route. And, you know, when you're thinking about an architecture, when you're thinking about content modeling, a lot of that relationship between the different content components is really critical. And here you can see how, you know, different pieces of that content hierarchy actually can fit together. We've started talking about something new here very recently, messaging, right? How do you target messages to these different screens? And when your messages are contextually aware of the rest of your content model, you can target any individual screen, or a platform, or anything else in the model. So the way we handle this is we actually create additional content types for not only the content model, but for our devices. So when we actually go in and add a new device into this system, we're creating a node in Drupal for that device, so we can capture additional metadata. The entity references that we're using to create the hierarchy in Drupal anyway can not only be used for parsing the data, but now it can be used to actually target and place messages on devices. We can use the metadata that we've got for reporting or dashboards for our content team. And as I said, we can use this to help target messaging. Another word of caution, fully bootstrapping Drupal to actually take advantage of all this content is really slow, right? I mean, if you actually fully engage the full stack to get the database and everything, that will slow you down. So this is actually a great place to cache. What we do is we actually take a snapshot at any given time of our entire screen inventory. That way we don't have to query 3,000-plus nodes every time we want to use it. The only time we ever invalidate and update that cache is when a screen changes or a screen gets added or a screen gets removed. So our parsing can go in and just grab that complete cache once done, doesn't have to query for each thing every time. And again, that's how we get a lot of our optimization and speed. So we talked about format a little bit. A few things here. So how big is the device? If you're lucky, your signs are all the same size. If you're not lucky, they may be the same width, but different heights. Maybe some of them are vertical. Maybe you have... One of our case studies we'll talk about here in a little bit is for a retail store, and they literally have every size of cell phone, every size tablet, video walls, and TV screens that they have to maintain using a system like this. So I mean, at that point, you're basically talking responsive design anyway. What direction is the screen facing? So my emergency evacuation example is only valid if the screen is telling you to walk the right direction. So we have to know where the screen is and what direction the sign is facing. And again, language. Drupal handles multilingual really well, understanding where that sign is placed geographically can help make sure that you're targeting the right language to the right screen at the right time. This last one is an interesting problem. So the advantage of using some of these services that we mentioned earlier, like Amazon, Google, et cetera, is that they just provide all this technology for you. The downside is you do have to pay for it. And most of them charge you based on message volume. So having a system that's wide open to the world, well, cool, can end up costing you a lot of money. So making sure that the right devices are getting data and making sure that only your devices are getting data are both really important. We actually can generate a whitelist using the metadata that we capture in Drupal. You can, or should, make sure that there's additional authentication, not only for your devices, but for your users. And we strongly, strongly recommend two-factor authentication to make sure that nobody's able to compromise your Drupal system to get in and deface signs. I'll let Adam talk a little bit more about security. So you just created a device in the real world. It's pretty exciting. Most of the time we only get to do websites, like we're banging away on a computer. I mean, it's kind of a very, very particular way of looking at things. So it's in the real world, and now everyone wants to hack it. Like, you have just put a giant bull's eye on your back. So you need to be a little bit wary of all of that. So, give your hardware some love. This is a great place to start. Embedded devices have extremely poor security records. Every single week you hear about different IoT devices being hacked, you hear about botnets. So make sure you don't take them for granted. Devices, routership with default passwords. Change them. There are voting devices that people have used where I think the password is one, two, three, four, five. This is voting. So make sure you look at those. Restrict the physical access. If you have a sign somewhere, you know, lock it down. Make sure someone can't go and access the USB ports or the ethernet connectors. It seems obvious, but you're dealing with a whole lot more than you usually have to worry about with a laptop. The other thing you need to remember is your hardware isn't just hardware. Hardware needs patches. Make sure that if you are buying hardware that your vendor actually releases patches on a regular basis. If they don't, that may be a red flag that you're buying hardware that may not be supported or that you may not have confidence in its long-term security. Make sure you stay on top of those updates and have a plan to make sure you can deploy them quickly when they come out. The minute those suckers come out, you know, you're vulnerable. And if you have a high-profile sign, people are going to want to go for it. So, you know, you have some breathing room, but, you know, be careful. And always remember, if you can connect it to the interwebs, you can hack it. Even a TV, even if you go to Best Buy and go and buy a brand new Samsung Vizio Sony TV, that's, you know, if it has an ethernet port or it has Wi-Fi, it's running software. That software can be compromised. So, no matter what it is, you need to think about it. So, it's... You need to not just worry about that, but you also need to worry about the source data feeding your sign. So, you know, we've talked a little about securing the hardware. We've talked a little about securing Drupal, but now we're talking about the data that's going into Drupal. That's a really good backdoor to get data onto your sign. So, it needs to be at least as secure as the rest of your stack. So, validate your data and audit it, too. So, imagine that you are feeding data from Twitter or Facebook, and someone compromises your Twitter account. Well, now they have the ability to get data onto your sign. Congratulations. You just said something really unfortunate, and you had a venue to publish it to everyone. 2017, SSL everywhere. No one wants a man in the middle of attack. It's a great way to vandalize your site. It's a go-live blocker. You will not pass. Yeah. We feel strongly about that one. And at the end of the day, just remember, it's not paranoia if they're really out to get you. And in case you were wondering, they are. So, we've alluded a lot today to the fact that this is really backwards from a normal website, right? There's nobody actually sitting here driving this thing. So, if you have a problem, what do you do? Right? I live on the other side of the country from the signs I'm working on. I mean, do I get on a plane to go reboot a sign every time it goes down? I mean, that's not super achievable. So, what you really want to think about with your, you know, building your sign, some of this can be done in your front end, some of this can be done on your hardware itself, some of it can be done multiple places, is how to get out of trouble when you get into trouble. So, a really easy one is what happens if you pull one of your data providers and it doesn't have data to give you? Does your whole sign go down? Do you show an error screen? Do you show the normal user interface without data? I mean, those are all valid, I suppose, but you need to think through what happens when there is no data, because sooner or later, there's not going to be any data. What happens if the internet goes down and your device relies on the internet, right? These are just the sorts of things that you have to think through that you don't usually have to think about with a normal website. And, you know, I think the most important thing here is that if your screen shows a lovely blue screen of death, somebody's going to notice, somebody's going to take pictures of it, and it's going to be on social media, and you're going to look silly. So, you know, fault tolerance is definitely one of those things that treads very closely to security, because, well, I don't know if you consider that a security incident, it might be. But making sure that your sign can autonomously recover from as many things as possible is really critical. And whoever's testing your sign should absolutely be running through as many scenarios as you could possibly think of to try and handle these sorts of situations. We actually ran into an issue some time ago where we did not do this. So, Chrome ships with this really cool aw snap picture. And when Chrome crashes, Chrome's way of handling that gracefully is by showing you this, instead of, you know, a broken web page. And when that is within view of thousands of people, that's not awesome. So, in this example, what we actually did is we used an open source plugin for Chrome, and we basically overrode the default behavior in the browser. So if Chrome tried to crash, we would instead just reload the page to update the connection. And if it didn't recover after the reload, then it would basically just go to a standby page with a clock. And even though no, this isn't Drupal, this is still an open source solution to a problem that you can certainly go in and begin overriding some of those behaviors. So we definitely encourage you to think about that fault recovery. Sometimes you can find a solution that's 90% of the way there before you write a line of code that will absolutely help you if it happens. So one of the next things that's really important to think about when you're doing these things is support. We've just outlined a lot of fairly complex stacks and feeds and pieces that are coming together. And troubleshooting, that's really, really hard. I mean, you know, what do you do when everything is on fire? Like, people are freaking out, you know, it's stressful. And, you know, what's the problem? Who knows? It could be, is it your web sockets? Is it your web servers? Is it the feeds? Where's the problem? So let's pretend you didn't see the presentation and you built a sign that uses Angular to pull APIs in real time. And you didn't load test any servers before launch. You added 20 more screens into production. Lots of screens, all polling, all knocking on the door, lots of kids. Kids are loud. All of your screens then lose data simultaneously. They just, they all go down. So what happened? How do you troubleshoot that? What do you do? Did the server crash? Did one of the APIs crash? Who knows? So, you know, make sure that you understand the architecture. Make sure that support understands the architecture. And make sure that they know what the warning signs are when a problem happens. You know, they need to be able to understand it and draw conclusions. You know, and have an escalation plan ahead of time. Know who owns which API. Know who to call when something, you know, totally goes wrong. And it will. And know what you can and can't fix. And figure that out before something really, really horrible happens. You know, maybe there's a feed somewhere that you can't touch. And if it goes down, you need to be prepared for it. You don't have control over everything. The world's a complicated place. And make sure you load test. Load testing is critical. You know, make sure you can handle it. Mirror the production hardware. Including any of your dependent APIs if possible. You want to make sure that anything that you're relying on can handle it. I actually did load testing a few years ago on a system. And the system itself could handle 500 plus concurrent connections. It wasn't going to be used that much. But one of our critical APIs could only handle five. So thankfully we figured that out before we went live. And you just remember that, you know, your source APIs, like Mike said, are not, may not be as robust as your main site. And make sure that they know that they should not make changes without notifying your team. Yeah. So we have a couple of quick case studies to run through to talk about. So, yeah, and we want to make sure and leave some time for questions as well. So the two sides of this coin we're going to talk about actually are interesting. The first one is lots of devices. Like tens of thousands of devices. But in this model actually they're mostly static data. So we're going to download some data on them and it's not going to change that frequently. The other one is not as many devices, only a few thousand devices, but all of those devices have to get data in real time. And thankfully the methodologies and the approach that we talked about today can actually get you most of the way to handling both of these scenarios. So let's talk about lots of devices with mostly static data. So in this case let's imagine that we're talking about a retail store. Maybe you walk into a Best Buy, or they always have media playing everywhere, wall to wall media. And historically many of these places literally have like a DVD player sitting behind a TV someplace running all of these. Let's imagine we're going to do it our way. All of this content is media for promotion. They want to show off the devices. Would you buy a TV if three quarters of the thing was like sideways and off the side of the screen? No. I mean would you buy a cell phone if it couldn't even play demo content? Probably not. So it's really critical, especially in a retail environment, that this stuff is perfectly tailored to each of the devices that it's going to play on. There's some challenges in this model for sure. In this case stores are actually really far apart. We could be talking about every Verizon store in North America. That's a lot of stores and they're really far apart. We could be talking about every Saturn store in Europe. And if you're not familiar with Saturn, it's kind of like Best Buy. They're all over the place. And it's not just a lot of devices in one location. Each of these locations has lots of devices. And each of those devices is going to be drastically different. I mean just think about the difference in like the iPhone 5, 6, 7, 7 plus. I mean there's even in that one line there's a significant difference, let alone everything else they're going to be displaying. So in this case the approach is not that different, right? So Drupal manages everything. In fact there's no outside APIs for data. Everything comes from Drupal because it's all video, media, image content. Drupal also happens to manage all the devices. So anytime a new phone, a new tablet gets put into a store, Drupal's made aware of that. They're actually on this project using the Google model instead of the Amazon model. That's fine. It's still push. That's good. And one of the coolest things about this is they actually have an offline content model. All of their devices in this case are actually running Android. So they have a local app that's running that syncs up with the Drupal server, downloads all of the content and then plays it back. This is incredibly useful because the devices don't have to be online at all times. And just like any other Drupal website they can actually schedule when this material starts and stops. They can pre-provision the devices so they can download data over time several days early. And then it all just works. The real advantages of this is that you do have a central place to manage and create everything. And I do mean everything. There's no additional APIs to worry about. You can rapidly reuse content whether that's per session or over time across devices. Internet connectivity, as I said, isn't constantly required. And this system really is quite massively scalable. Again, they're already running tens of thousands of devices and there's nothing to stop them from going even larger. And again, as we've talked about, there is a significant amount of user experience for content managers here because they don't have to worry about what else might be interfering with what they're trying to put into Drupal because it's all in Drupal. All right, so let's talk about Case Study 2 which was the real-time issue. And in this one we're actually going to be talking about mass transit displays again. So in this case, all of the devices are digital screens. And the content here is going to be a healthy mix of arrival data, messaging data, and emergency communications. In this case, we still have some challenges. We do have to accomplish near real-time communication across thousands of devices and it's going to be fairly obvious if you can see two signs at the same time if they're not the same. So the public will know if you're hand-waving too much. You do have to have constant internet and web socket connectivity for this model. And that is a non-trivial problem that you shouldn't overlook, right? I mean, your fault tolerance has to be near perfect in this scenario because, I mean, real-time isn't real-time if it just stops for eight hours every day. Error handling and recovery, as I said, and it does provide for on-demand messaging, right? If something's wrong, you can go in right now. You can publish it right now and it'll be on the signs right now. Performance is definitely a challenge here. As we said, you have to be able to go in and parse this data very, very quickly, get it to the right places very, very quickly without relying too much on caching. Our approach here is that all of the data is getting parsed by Drupal. This is not surprising. Drupal provides some of the data. We use the Amazon WebSocket push model. We actually used React for the front-end in this case, but again, pick your poison. It could be done in just about any of the JavaScript frameworks. The unique thing about this is that as soon as a screen bootstraps, Drupal loads once and it's done. That's it. React takes over entirely. At that point, even though Drupal is technically providing the data, all of that data is coming via the WebSocket, not Drupal. What we've essentially created is a React front-end that is informed by Drupal, not a normal Drupal theme. As I said earlier, we can cache and create an inventory of Drupal Assets to speed up the parsing model. There are a number of advantages here. You can really do what I call right-sized messaging. You don't have to send it everywhere. You don't have to just send it one place. You can pick, or the content creator can pick, how much they want. Again, this system is massively scalable. I think now we're about ready to wrap it up. Hopefully, we've given you an idea of, number one, why use Drupal? Why is Drupal important? How do you use Drupal to do it? Giving you a few architectural techniques and approaches for how to do it and how to justify it. And a few case studies for how we use them. Hopefully, we'll see some digital signs and more and more of them being powered by Drupal over the next year. If anyone has any really cool projects, we really look forward to seeing them. And please let us know what they are in the next year. And finally, questions? Please do come up and use the microphone for the recording. What hardware did you test this with? I've been tasked with looking at redoing our digital signage and we'd like to do with Drupal because the WordPress site that we use sucks big time. So I was looking at maybe doing a couple of Raspberry Pi's because it's a closed network at my institution so I was just kind of curious what hardware setups that you guys use. So we can't talk too much about the specific setup. I mean, it is a device-running Linux. It is a closed system. It's enclosed in a box. It's not a Raspberry Pi, but a Raspberry Pi is 100% legitimate. We can maybe talk a little bit more about that offline. But I mean, basically at this point, the important thing is that, like you said, it is a closed system and it is something that will actually power the screen. Sorry, I know that's kind of a vague answer, but... It's understandable. Thank you. Hi, this was great. But somebody has sort of propositioned me to make a mobile sign. So I'm wondering if there's precedent for getting data from, say, a GPS collection, you know, a unit going back into the servers to, you know, solicit the content. Do you have any experience with that? Yeah, I mean, theoretically, I mean, you... You would... I mean, the question is, is how much hardware would you throw at it? I mean, if you're doing stuff that's personalized to GPS coordinates, so real-time, based on someone's personalized data, you're going to need a lot of hardware, so it gets expensive. So, you know, like, let's say that you subscribe to a certain GPS coordinate on a push model, you might... You know, every user may require processing for that. So that can be a little bit tricky. If you can make it less granular, so you have a large, vast geographical region, so maybe you have, like, a borough in New York or a neighborhood, and based on that, you create channels. If you have that level of granularity, then it becomes really, really easy, and you're not putting too much of a burden on your system. You know, the more that you can think of it as a broadcast system with WebSockets, the better off things are. You can always fake it with some polling, but you have to decide how real-time do you need versus, you know, how static is the data? It's a balance. Does that answer it? Pretty much. It's a work in progress, but thanks again. Maybe I'll consult you offline. Yeah, look us up. You mentioned in the second case study you used an Amazon push model. Have you built your own, or have you used any other services for push models? Yeah, we used the Amazon IoT library to do it. We did not want to build our own. That's a very, very bad idea. You know, often you build your own, and you have not thought through things, you can't support it, and it's more expensive. They were a little bit perplexed, I think, by our usage of it, because normally they think about it for devices. But they give us a tool that has, you know, the typical level of scaling that you would expect from an Amazon product. So the thing that it's not that hard to set up a WebSocket, right? That's not the problem. The scalability is the problem. So the whole reason we went with Amazon was we wanted something that we didn't have to worry about throwing servers at it. We just wanted it to work, no matter how many devices or how much traffic we needed. So, no, it's the short answer. We didn't play around with rolling our own because we didn't want to try and handle that variable scalability. Okay, thank you. How do you guys handle transient data because, like, how long the delay for a train is from four days ago isn't relevant anymore today? Right. So do you just have another Drush script out there or how does that work? So the nice thing about this system is that we aren't doing a whole lot of contextual manipulation of the data. So when we go out and ask our weather API is it raining in Baltimore right now, we don't do hand-waving to try and figure out if they're telling us the truth or not. So in terms of, like, transit data, if we're getting a feed of transit data from a source, we assume that that source knows that, yes, that train is actually five minutes away or it's gone. So we're not doing any hand-waving to try and figure out if the data we're getting is accurate. We are taking it on authority that the data we're getting is accurate. And also just that it's extremely, the data is always in flight. So we're constantly sending updates and if there are no updates, then the sign would stay staffed. So when you pull that into the Drupal content types, that becomes a node at some point. Well. So what we're doing about the real-time data, no, Drupal is actually taking that much data and shoving it into Drupal would be really slow. So what we do is we take that inventory of our content model and store that in a cache and then as we parse that data, it never touches Drupal. So it's all just object-oriented PHP that gets it, parses it, and shoots it out and we archive it for auditing and like health historical purposes. But no, we actually don't take, like, the data and put it into Drupal itself. Thank you. Any other questions? Well, thank you very much. The only other thing is we just wanted to remind you that there are sprints on Friday. So if you can, please attend. The DA asked us to include this. So thank them for organizing this. And thank you guys for attending. Thank you. Thank you. So basically, the way we did this, we sort of came out of computer-oriented PHP that we typically run with a Drupal browser. So we built all the data parses on the object-oriented PHP. So we built all the data parses on the object-oriented PHP. So we built all the data parses on the object-oriented PHP. So we built