 Okay, it's 1045 people are still trickling in so I'm going to give them about another minute everybody's here for the presentation on how to Migrate your Drupal 7 website to psych or correct You don't joke about things like that Okay, let's let's go ahead and get started. We're not talking about psych or today We are talking about how to scale Authenticated traffic in Drupal. We're using case study based on Whole Foods Market comm which is where I'm from and a project that Adam's been working heavily on and How we did that with stateless authentication So I'm dr. Jay. I'm the tech lead for Whole Foods Market. I work on the web teams mostly Got some contact information up there if you guys have any questions or just want to connect with me afterwards and This handsome fellow here My name is Adam Weingarten. I'm a technical architect with Acquia So I work at our professional services group so helping to Come together with requirements look at you know crazy client asks And then try to you know pull a bunny out of my hat and you know make sure that we pull it off and You know keep out all the things working running and keeping our clients smiling So it's always an interesting challenge and we are that crazy client. So we brought you by Acquia, which hearts Whole Foods So today we're gonna talk about Everything you see on the board here why scaling authenticated traffic is hard. Some of you may have already felt the pain on that We'll be talking about what session this authentication is Sometimes we'll call it stateless authentication, but they're the same thing We're talking about how we use it at Whole Foods Market.com to do a single login to multiple sites under the same domain How we did that through an external authentication provider? We use GenRain And then as we get towards the end, we'll start talking about some other nuances of that project PS which is storing PII as a service in an API Talk about some of the pain points that we had in in this process as well Particularly proxying web service calls. Don't do that And then how we delegated a lot tokens in that process And we're gonna kind of go through all the basics about Sessions how sessions work You know and the different kind of these different nuances that will help you To ultimately scale to handle some you know interesting User traffic patterns. So if you don't if you're not like the greatest expert on user sessions, that's totally okay We got you covered Cool So introducing Whole Foods Market.com Hope right now Whole Foods Market.com is primarily a dribble seven website. It launched in July 2007 sorry 2012 It was based around Much more static world the the overriding goals of that project were to build out a Highly localized experience the idea being that every store could have a unique sort of flavor Trying to represent the way that each of our stores is slightly different But we didn't have quite the technology back then six years ago for Doing this highly personalized events based on users. So this is more controlled at the front end and so because of that we weren't really focused on on a lot of authenticated experience back then we did have the ability for users to log in but the The value proposition for that was was more on the nuanced level You could sign in and store a recipe box for example or maybe make a shopping list But these weren't really primary functions of the website So as we're building this out really everything was based around, you know, full page refreshes. It wasn't service-based It was a brochure website on steroids, right? and so on that first year of launch we had a lot of problems with scaling that just even at the static level on any given day we may or may not have had the capacity to stay up And so it was um, you know, it took a while and a lot of help from AQUI to get that under control and tame that beast And that's where we've been kind of coasting for the last four years is trying to keep this Sort of I don't want to call it an act to create a technology, but this sort of you know crazy build up and running I see a whole bunch of you on the back If there are people if people can move in towards the walls and make some room for the folks in the back So they can sit down that would be awesome of you Thanks so much hate to see people just like you know with no place to go So please come in We will defrag the other room nice Somewhere I'm Thank you so much. Thanks everybody Yeah, so D7 as we mentioned had a lot of different issues We have some very very primitive Jan rain social integration So we were using Jan rain to authenticate users. They provide some really great social API tools But our integration wasn't really well thought out. So it was extremely heavy on the pages The the libraries were using were quite heavy. They were contributing to page bloat They were slowing things down It just it wasn't thought out the right way our DB's Became extremely bloated One of the problems that we had was Because we were doing things In kind of this organic way. We were also using kind of the most off-the-shelf approaches in Drupal Every time that someone created an account using Jan rain It also created a user record in our Drupal database So if let's say we had a million customers creating accounts, we had a million users in our Drupal database And it wasn't necessary. We weren't actually really doing anything with that data in Drupal. I want to clean up. We were Using that data They were we were getting them logged in and then we were basically making calls to some other service to store some recipe Stuff or other information. So we had all this PII that was a huge risk That we didn't need so we're making this dangerous storage That and incurring this risk and cost and there was no purpose to it And then as we mentioned these these authenticated experiences were requiring Full page loads so we couldn't leverage cash So he mentioned how like the site would go down Because we weren't using reverse proxies properly. We weren't using a CDN You know and on the side of that magnitude. It just it doesn't work. It doesn't scale You have to you have to think about these things. You have to think about how do I cash as much as humanly possible? So we had some issues and you know to give a sense of scale on that whole with market comm gets about a half million visitors a Day and that can swell up and down in the holidays But that's that's roughly the size we're talking about which is a big website But not in the scheme of things people do much much larger traffic patterns and sort of those things well so just to give you a sense of where At what levels we're falling down So you want your food to be grown organically but not your website? So yeah, I mean, you know and this is something that we all have problems with you start out with something really simple You start bolting things on You know, you're all under a time crunch. It happens to all of us and You know you get something out and then you never get time to come back and do it right or you know So sometimes if you don't do it right in the first place, you're stuck with it for a very long time And so as We went through several iterations of planning around how we're going to fix this You know as I mentioned this first couple years were a patchwork of just trying to get big brains in the room To give us some guidance on how to tame the beast A couple years ago, we started going in a different direction and thinking you know What can we what can we build out as a fresh experience and we had a brief and dark for a into a psych or build? Which lasted about a year and we were going about that in a Big Bang fashion and one of the lessons we learned from that process was that We had our own institutional issues on planning and how to figure out how we're going to do large-scale Full-spectrum mode deployments and so when we backed out of that process and decided hey, we still have this dribble seven site We'd like to fix up We double down on saying okay, well, let's get to dribble eight But instead of going at it with the big bang new build fresh start fashion Let's see how we can nuance that and have a parallel track of development. I'm sort of upgrading certain things So from the D8 perspective, we knew what our high-level goals were we still had a basic north star idea of where we're heading and I'm not going to go into all those details but the big takeaway for this talk is that the there was a much much greater emphasis on Authenticated experience we really wanted to target to give targeted personal Individualized offers to customers on a one-by-one basis and we had back-end systems that could serve this out But the web stack that we had would never be able to handle that kind of traffic. So from that standpoint where We were looking at supporting 10% authenticated traffic, which is probably about 40x where we were Prior to that. So that was our that was our target metric for that. So it's a significant burst there Because we're splitting the stack up into dribble seven and dribble eight We needed to have some ability for people to authenticate in one place And not give them the journey experience of having to bounce back and forth and have to separate logins Personalized digital experience extended beyond just the coupons. We also wanted to make sure that they're getting some Unique value beyond the offers in that experience and then Mobile responsive was a big facet as well because our current experience was was very static and also, you know, we had a We had I think we were running three or four different themes on the site. We had an old-fashioned m dot site. We had a desktop theme it was really really difficult to test and You know regressions came up And it was difficult to scale those things and also make sure that we were then caching both things And the desktop properly and we had to then double the things that we were caching which also was unpleasant So back to the basics So we're going to shift now into talking about just scaling traffic in general and what the payments are around authentication So scaling anonymous traffic is easy. I think most people that were probably familiar with that We can you see we can lever CDNs to do the heavy lifting get people closer to the edge try to cut down the latency We've got varnish to lean on You know so caching is the quick quick show hands. How many people here have worked with Akamai Okay, how many people have worked with cloudflare? Okay, and how many people have worked with it fastly Okay, so so we have a lot of people who are raised their hands for one of those So there's a lot of experience in this room with with CDNs. This isn't you know cutting edge. This isn't you know Insane to people you're doing it You know basically, you know and for those who didn't raise your hands You know what the way it works is you know you a request is made It the CDN provides a caching layer says do I have this and is it recently valid if it is it serves a request The request never makes it back to the servers And your webs are happy So that could be with the CDN often times people also throw a varnish into the mix as an intermediate layer You know, but they hit the origin didn't store the information inside their caches for the next person So the next time someone comes along It's warmed up for them and in the modern web You know a lot of these things end up being served out as basically a static page And if we want to do personalization Destiny we can handle client side or otherwise. So so these things can get very performant at that level when we get the sessions though That's a different beast and yeah, so so, you know HTTP is a sessionless protocol. So sometimes it's important to take a step back and just say what the heck is a session? How does this work? So by default you send a request to HTTP. It doesn't have any idea about who you are what you are It just says okay. It's a request. I'm going to service it So we have to overlay this concept of a session and there are some interesting implications of that So what is the session? So it is like this those sequence of interactions between a specific person and the server So, you know, you add something to a cart Well, we want to make sure that you know, it's add to your cart and not to you know Someone else's cart or if you enter your credit card information It's you know, it's yours and it's not mixed up with someone else's And when we start to deal with caching where we are, you know storing things for everyone We need to make sure that we don't mix it up So those two can kind of hit each other a little bit So what does the PHP session look like? You know, we have this session super global Often it starts with a session start early in your PHP script Drupal wraps this up and it stores all that data across those requests in the DB So a lot of that is handled at a pretty low level the bootstrap And it then uses a cookie value to ID you so there's some sort of a GUID it's stored in a random cookie So what does it look like? So, you know, typically what you'll see is with a with Drupal you'll see these cookies that start with a an SCSS And then they have a value. What is the value? It doesn't really matter. It's just a random identifier for you And you can see these, you know, really really easily in your browser So it's something to you know, that can be useful to poke at But we kind of mentioned sessions can be problematic. So why are they a problem? So Isn't she cute So this is a curl of an HTTP request Typically what happens is We want to make sure again that we don't mix, you know, the different user sessions So when when most CDNs or varnish caching layers do encounter a session cookie as part of your HTTP request They will respond with a no cache response So they will make sure that Everything is bypassed that You know what you get is unique to you and it takes a bit of an aggressive stance to make sure that things aren't being mixed And this is really really good from a security perspective. It's simple. It's easy We have a clear cut set of rules that keeps everyone somewhat unique But the sad part is that on this X cache line. We see lots of misses. We don't want to see misses We want to see hits So yeah, we're under a lot of pressure here So we've got To some conflicting requirements here that are really really tricky to address So we've got these new experiences that want high degrees of personalization But our infrastructure sucks at it. So You know, it makes things tricky We can, you know, maybe throw lots and lots of servers at it be expensive probably burn a lot of You know cycles. I guess we could put some Bitcoin miners on them while they're idle You know do something with it with it when we aren't using them, but we can't cash it So we have to come up with some other clever ideas to figure things out and Oh, yeah, we need to also figure out how to make this work with D7 and D8 because we're not doing this in a shotgun We're doing it slowly gradually So users are gonna log in on one site and then they're going to move over and manage the recipes on another site So it makes things really interesting So in order to do that, we're gonna figure out some way to set a token that both sites can somehow read and do something with So solutions You know, we're good at crushing you, but let's let's try to do something constructive now So with traditional authentication Here's how it works. You you have a user Look, they're still happy We'll try to keep them that way it doesn't always happen They hit some sort of an identity provider Maybe it's Drupal traditionally It'll you know, they'll send their username password over SSL It looks it says yeah, they're cool. They're great. Whatever and then what it will do is you know Drupal will Go and query a session table to get any data associated with that session So it's typically hitting my sequel it will return that data And then it will manipulate that a little bit and send some personalized data back to that user That's our traditional model overall The way that sessionless authentication works is we break this up a little bit so first From the browser we hit we go through an authentic an identity provider. So in our case We weren't using Drupal. We were using Jan rain So Jan rain provided this great abstraction layer for us where our users could authenticate using I think it was Facebook and Google As their providers we had some legacy systems But those were the the new ones that we were allowing for new registrations as well as traditional user password combinations Yeah, so they they also allow that as well So we didn't have to be in the business of managing that inside our Drupal database our Drupal database could focus on other stuff What we would get back a token from this identity provider saying that they have authenticated and they are indeed who they say they are So then Drupal Instead of going and querying My sequel for that session information It would take those various IDs that we would get from this Identity provider so they're you know unique ID information their Email address some of those basic things that we needed to interact with them it would take that information bundle it up and Put it into an array and then encrypt that array and send it back as a cookie and That is actually the crux of what we're talking about today is basically getting rid of that database call and instead Delegating that to the user Itself on the browser so we don't have to deal with that There's a little bit of hand waving here and we'll we'll call it out But but that's the core idea Drupal doesn't really have any concept of the user. They are not a Drupal user There are implications for that and you're gonna have questions about that too So there are definitely some limitations in this approach And some of you are gonna see it some of you will ask about it. I respect nothing less So then what happens is let's say that I want to store some dietary preferences so I send that API call with those dietary preference changes Along with mine cryptic cookie Drupal then has a way to decrypt the cookie and decide is this a valid session So it decrypts it says first off doesn't be correct Do does it look right? Does the data seem well formed if it that happens it then will send the data along to a Personalization API that we had It would write the data and then it would respond back That yep, we have updated your dietary preferences And update the on the page So this is the general model that we have So from Drupal's point of view What happens here is it gets a request with this encrypted cookie? It first says like I said does a decrypt if it doesn't we just for three the request You are not authenticated. You do not pass go. You do not collect $200. Sorry If it does decrypt properly, then we say is we put a timestamp in that cookie as well So typically with a session a session has a duration So we're able to encode that along with the rest of that encrypted information So we say is the timestamp valid If it's not again, we forward through the request. It's not valid. Sorry If it is then you are now Authenticated we know who you are you do have a valid session and you can move along. So what is in this exactly so So it's anything that might live in the PHP session or user table So whatever you can imagine you can shove in this encrypted cookie and send it over to them So that way you're not storing it in your database So in our case we were using I think mostly API user IDs That expiration time was in there It was pretty lightweight that was pretty light Yeah, I mean, this is really a for us to prove a concept of what we could do to scale and We had big plans, but we will eventually shove in there But you know really it's anything it is the magic It's anything that you need to have to be able to sort that experience out can go into this thing And in our case we were using this external personalization API So we didn't need to store the actual preferences themselves We could just send that along to that, but if we didn't have that API We could have stored it there and just you know, let it get stored on the client side if we want it potentially Or if we absolutely had to we could still store that in a Drupal table somewhere We have those those different options available to us So how does this let me do D7 and D8? So assuming that the two sites are on the same domain or subdomain then both sites can read this cookie And as long as they both know how to decrypt it They can both read it. No problem. So they both get the same key They both have the same keyhole you turn the key and you're in so the way that we implemented this was We built this on D8. So we built our Authentication system over there you would lock anytime you try to log in on D7 it would actually kick you over to D8 And You would log in and then you would get bounced back to your original page on D7 to complete whatever action you wanted like Changing those dietary preferences or adding a recipe favorite. So there's there's code on I'm just wondering about multiple devices Obviously if they do this at work and they go home at night No cookie and on their phone. No cookie. How do you handle that? They would have to log in again. I mean just like any other device You know, you're gonna log in but once you're logged in it would behave exactly the same way So in this well, yeah, I mean in this case we have the that's abstracted out into a An internal service that we have we call it up. It's universal profile manager And that's what is attached to your What we call a UPM ID, but this is like your personal user ID So if you're using a mobile a native mobile app, it's attached to that ID And then as well as your login as well because we use a unique identifier by email address And so we know who you are We know that you share that ID and anywhere that you authenticate will call out to that service to grab what it needs About you and you know, hey, you're a vegan and you live in Seattle or whatever, right? And pass that back in terms of logging in though. Yeah, it's it's per Her device so just because you're authenticated in your native mobile app doesn't mean you will be also on your you know web Mobile app or your web mobile experience and and you really I think that that's a great point You really only in this case want unless you have that microservice Or even when you have that microservice you want to make sure that you are storing ephemeral data in this So think of it like a memcache Where you're not storing permanent data there you're storing stuff that can be destroyed at any point and can be recreated So in this case, we are storing your basic authentication credentials You know some stuff that we can get every time that we authenticate you We're not storing things that we can't recreate If we did then to your point that we would those would be lost when we destroy the cookies and that seems like a shame Speaking you're destroying cookies. Yeah, sometimes we need to oh, yeah, you have another question Yeah, I can repeat the question for you. That's fine Okay, so the question was since we're not storing data in the user table As these users command does Drupal see them as anonymous users and the answer is yes Yeah, so there are two ways that you can do this one is you can tie into the Drupal authentication system and you can treat them as Assuming that that the request comes through and that You know, they they pass all of our gates. We can treat them as we can assign them to a role And treat them as kind of a user of a certain class They're not an individual user from Drupal's point of view, but we can do that We can also just simply treat them as an anonymous user depending on how we're working. Yeah Does it go into the off-map table? We can't we could potentially do that in general We're avoiding that because then we start to get into Different database API calls in our case. We had a very very narrow use case So by doing this approach we we actually don't make any database calls That's the value proposition for us. It lets us scale very much if you start You know making all those database calls then you negate the benefit of this approach other than The the fact that you could potentially share the cookie between Drupal 7 and Drupal 8 So you have to balance that so you are making some sacrifices in order to avoid the database calls and To be able to move between the sites Yeah, I think that the takeaway is that it's a choice that you can make with this approach So you could in theory go the route where you're doing some hybrid of authentication and anonymous as far as Drupal is concerned Depending on what your use case is and that's that's going to be up to you to handle The you know the pros and cons of that approach and I mean and to that point I mean, you know, it's kind of where the slides going I mean, you know when we did have a session in Drupal we could log them out So like let's say that we there's a Drupal There's a vulnerability that attacks the the password table I feel like there was something a few years back where we actually actively logged out We had to log out all users doing that's really really easy You go you truncate this the session table boom everyone's kicked out go log in again I think I had to do actually with a there was a vulnerability in SSL Hardly yes, that was it. Thank you. Yeah, so in the case of Park lead You know, we just we couldn't trust that the on these SSL sessions were valid We didn't know if user accounts have been compromised. So we needed to log everyone out in a hurry And now that we have this kind of decentralized approach So so yeah, so, you know, we can get rid of these zombie sessions You know, you can look at the table here, but the gist is that You know at the top you look at our normal flow Joe signs in We said a cookie we talked about storing time stamps in those cookies so that we can set our own parameters around when they expire Looks like 30 days in this case And then you're off to the races as happy walrus says So in this in the scenario that Adam's describing is like, you know, what do you do when you need to say get off my lawn, right? So we've got this cookie set for September. It's expiring in October and Then we flushed them out and then saw logs back in and he's good to go for the 30 days, right? So mechanically how that's going to work. Well, it's not really on the table. Is it We're missing a slide here, but the you know mechanically how it's gonna work is like we've got code on the back end that I'm gonna let you actually handle that Getting a little a little heavy there, but the basically it's gonna kill all the stuff is off So what we do is we actually store a timestamp on the server that says, okay If anyone has logged in, you know before this time Or if their expiration is before this time plus 30 days If it was before that kick them out if it's after that they're now valid So we're able to kind of have this global identifier to say it's okay And that goes into that authentication flow diagram that we showed you before now The messy part though is we need to make sure that we we hit that button or make that timestamp on All the servers that are doing this so if we're doing it on Drupal 8 we also need to make sure we do it on Drupal 7 Otherwise, it won't work. So by having that one little global value and the timestamp in the cookie We're able to make sure that people, you know are relatively new So we're basically taking advantage of the check that we already do To manipulate that and kick them off And P. Yes. So so I'm Pete personalized personal personally identifiable information Storing that as a service in an API. I went into that briefly a minute ago with our our UPM our universal profile manager And this is our attempt at Whole Foods to abstract some of that outside of our web accessible properties So the way that that works on our end is, you know, that can store things like dietary preferences. It could store your email address In the past we have stored things like your date of birth I don't know if we're still doing that But you know, whatever whatever really You want to share across properties. That's the goal that we're trying to solve with this But it has the added benefit of taking that PII out of the sort of the hackable demands, right? And so in in our case Drupal is a consumer of that API just like any other service And so, you know almost anything in the company can hit that through a private API But it's not hitting from Drupal's perspective any database calls. We don't store that information We don't want the information and so we've got a really clear separation of concerns on that front And I mean from a security and compliance perspective. This is really really big Drupal's awesome but it would be very very easy for someone to accidentally create a view that maybe shows users and Maybe put forget to put like views permissions on it And all of a sudden with the best of intentions, maybe someone created an admin page They might have actually accidentally opened up a vulnerability that exposes all your customers PII and we got access bypass Issues all the time, right? I think there was one last week You might have heard of it, you know, so in the normal course of events I think it's becoming bad practice in general to store these things in your database So this is part of the process of ripping that out I think we're all kind of filling the pan on right now and you know it in this case It's a black box API. You could imagine it as another Drupal site potentially You know, there are really some great, you know web service, you know API first approaches in D8 But often it's a good idea to separate your personalized information from your main web stack There's obviously an overhead, but it's generally worth that and the other thing is that You know, it also allows you through that separation of concerns to load test things much more effectively You know, we mentioned personalization is really really hard Well, if we can make all of that simply, you know a single monolithic API that we can hit We can, you know slam it really really hard. We know what the tolerances are You know, and then we can attack those problems which are a little bit different than the problems of building a brochure site So we'll talk about another 10 minutes Just on some of the problems that we faced or really the biggest problems that we faced in that And then that should leave us 15 minutes for questions after So The biggest one is is proxying those web service calls I think that's the road that we went down at first and it turned out to be a mistake in some ways And I'm gonna let Adam go over the details on that but from a performance and stability perspective I think it was a big lesson learned. Yeah, so we were working with this API You know, it was black box to us. There were limits in what we could get from it and You know, we had to work around those so in our case we had to Get users authenticated they would then take it an action and then what they would do is you know Drupal would then make the call to the to the service and then get a response back and then respond to the user So a lot of bouncing Why is this bad? Well all those hops are adding a lot of latency. So it's slowing down our user experience You know, we have intermediaries That's not ideal Your web servers kind of suck And I don't care who you are or what your web servers are just web servers are finite resources And every single time you want to make one of these hops through your web server You have a finite amount of memory, which means you have a finite number of concurrent PHP processes that you can run on your web servers So every time that you make one of these hops You're you're basically taking one of those prox out of rotation and you're making it wait Until this other call finishes and HTTP calls take a lot of time So this limits how many transactions per second you can do on your servers Which means now you have to throw more hardware at the problem in order to handle it And since we're specifically having a project goal of scaling out traffic and scaling up authentic data traffic This was a big issue for us So what do we do to fix it? So in the model that we just discussed You know, let's say Wally. He's asking for some dietary preferences. He goes to Drupal. Drupal goes to the API API goes to some storage somewhere to get the PII it goes You know ideally what we want is, you know Once it gets authenticated, it'll go back to Drupal. Drupal goes back to Wally not ideal hops hops are bad what we want is ideally we want the Browser to be able to hit this API directly and bypass Drupal altogether Drupal has no benefit in this model right now So in order to do that though We need something like OAuth where we can delegate and create a token that the user can use to access this API directly So what we would do is we would have Wally go talk to Drupal Drupal would talk to OAuth get a token back for this API that they can use that Limits their access that has access controls in place. It would then go back to Drupal Wally has this token now and with this token Wally can then make Requests directly to the API. They are safe. They are secure their firewall to the things that only he has access to ideally This would be great Unfortunately, we didn't have this We begged for it. We wanted it, but we couldn't get it. So that's why we were stuck Unfortunately with the first approach where we did after proxy through Drupal and we did scale up our hardware a bit to compensate for that You know, it's something that we are still working on that we do want But you know, they're trade-offs and ultimately you have to deal with the reality of you know resources and other teams I love giving a talk at a Drupal conference about a Drupal site and the entire gist of it is bypass Drupal So yeah, so ideally if the browser can access that API layer directly without the intermediary then we're winning Now some of you guys may ask about like how does all of the scale we've talked about performance We've talked about scalability So we ran some performance tests And one of the things that was really really interesting was as we started to slam all of these authenticated experiences over at Drupal We saw our CPUs spike And that makes sense based on kind of the story that we just told You know the the prox and the CPU are waiting on web service calls And and that's kind of the the behavior that we would expect The other thing though that was interesting was our memory was quite low As well as Some other benchmarks, so I mean our CPU is we're busy waiting for these things, but we're not really we're not spiking our CPU our memory So this starts to suggest different trends on how we can scale if we're CPU bound being CPU bound is a good thing Actually in this case You know and the other thing that was interesting is if you look And our database Our database is cold. We're not doing anything. We have almost no database access for any of this And the database is the part that's really painful to scale That's the part that we don't want to get into because we have to basically just throw One big piece of hardware to handle the whole thing. It's horrible to do. We don't like doing it So the fact that we have these lines that are pretty much steady says It tells a story Which is that we can scale linearly with this approach by webs And not have to worry about scaling our database Webs are cheap and easy to scale we can throw webs into rotation behind a load balancer. That's simple It, you know, we don't really have to worry too much. We don't have to have any down time We can put them in we can take them out. It's great for scalability And the fact that we don't worry about our databases or memcache or other things is also a huge huge win for us in terms of scalability So at the end of the day, we ended up with something that actually scaled pretty well. It wasn't perfect We've worked around those limitations that we did have We got that separation of concerns and now each part of our sec and focus on its single concern It's single responsibility and you can do it well So we have the API which we've low tested independently We know how much traffic we can throw it at how many transactions per second we throw at it We know where our web layer is doing And you know, we've controlled what everyone is doing. We've done security audits on each layer So, you know, we're not trying to redo every single test every analysis at every single point You know, and ultimately it worked pretty well. We Since we introduced it, we haven't really had to scale up the hardware at all. It's been really quiet It's been surfacing a lot of traffic And it's been much more robust and had a much better launch than our original Drupal 7 code base Yeah, I wasn't around for that one, but I was it was painful This one's been relatively quiet and boring So, you know, that's what we're going for So, I think that's all that we have. Yeah, that's it. So we've left about Little more than 15 minutes for questions If you can please step to the mic If you can't I'll try to repeat it for you I'm looking at this from a cost-benefit analysis of the amount of effort that you put into this Given the fact that it's supporting a site that's on 24 7 with that many users Would you I don't know the question is how long did it take for you to go through this? Exercise and come up with this the solution I've got a similar situation where we've got you know thousands of authenticated users coming on At at one time and this approach would really really be beneficial, but it happens twice a year For only one hour per time So I'm not sure if I want to spend you know a whole year of a development effort For two hours out of the year I'm gonna take the first part of that if you don't mind Thank you. Yeah, and if you want to jump in a bit too, but the you know from our perspective and I Let's rewind that I think from what I'm hearing from your perspective is that there's two hours out of a year That are affected by this but those some are two very critical hours so from our perspective our expectations around our Marketing campaigns that were targeting certain dates and how we're advertising this We're expected to bring in so much traffic all at once that it would have been disastrous if we were to have downtime at those moments and so you know There's there's a lot of effort you can put into front-loading that in terms of stress testing and low testing and everything else and trying To figure out where where your payments are, but it's a different ballgame really always when you get out of the synthetic game so for us there was a huge there's a lot of dollars riding on getting it right and Compared to the technical investments, you know, we were this is really so central in our case to the The problems we were trying to solve that it was you know it added scope But it wasn't wrapped into its own unique base where you had to invest a ton of hours into it I think the I'm gonna let Adam ask that question, but I think that the biggest lift was Ideating it more than actually now. Okay. That was that was actually the easiest part I think So in this case we we would have campaigns running throughout the entire year So I think at that point that was one business driver that justified the cost of some of that development If you're doing it once a year, it probably would make more sense to scale up your hardware as my gut In our case, we also had a bunch of additional things that that we had to spend money on no matter what we had to You know do Jan rain integration. We had to be able to log in to multiple Drupal instances so once we we had those Requirements the session list authentication piece was actually fairly minor It really was was not a heavy lift and even thinking through it We hadn't spent we didn't spend that much time on it But yeah in your case, I would probably I would throw hardware at the problem. I think it's probably cheaper for you Sometimes that's an answer Yeah, I saw that in the brown shirt first, I think Okay, so the question was a little more detail on the PII API that we discussed Whether that was built-in house and then whether we're experiencing any kind of Lower performance issues on that front Okay So the yes the PII API actually does a lot more than just PII But it's we have an internal Whole Foods API that we hit for a lot of these services This is built and wrapped into that as additional functionality to handle that profile management and all of that is Has a Apigee in front of it as well. And so really the stress on that system is handled by that team internally It's not necessarily to say database calls on the Drupal side So on this talk we're focused just on the that session handling But yes, we do introduce load based on the calls that we make to that service I don't recall that we had particular load issues on that I think we had programmatic and logic issues, but yeah, so basically I mean from our point of view It's it's a pure black box. We don't know what it is. We don't really care what it is What we did do was we worked with that team to make sure that it could handle a certain number of transactions per second And we tried to think through what are some of the worst? Calls that could potentially happen Aim for those and then we had that kind of contract with that team to make sure that it worked now You know, what could that look like? It could be a Drupal black box Drupal does some great stuff there. It could be something powered by a MongoDB by a DynamoDB It depends but it's black box as long as you have that kind of contractual Agreement with them that they will be able to handle a certain amount of throughput and you have those API You know contracts as well, then you'll be fine Yeah in front yeah It's an excellent question. So the question is how do we handle our admin users in this system? And the answer is we don't we we haven't we abstracted them out into a separate area of the site Where we replace their existing login page with a different path and so there's we They go to the same domain, but they actually have a standard Drupal login in our case We're talking about, you know, maybe 500 people total So they might have loaded that our habits put on the system is quite low But but it's a good call out Yeah, so I'm just trying to wrap my head around kind of where the Drupal the actual Drupal piece of this is that's specific to Drupal So it sounds like all the personalization stuff is ideally happening being kind of rendered on the client side on top of a kind of static brochure where Pete right yep, and You're having sending the authentication tokens through to your API pass through so that's all independent of Drupal really So are you actually doing anything in Drupal? Yes, or is it all kind of just stuff on client side stuff on top of Drupal? In broad strokes there. There's a custom Drupal it module that handles the authentication piece And so that's what we'll take in provide the form Encrypted and pass it out to where it needs to go on the Drupal 7 side There's also code that you just be able to read that cookie decrypted and parse it out to where it needs to go So overall Drupal really is functioning in this case as a gatekeeper So it's it's it's working with those authentication providers to make sure that you're actually logged in once it does that it really just Is the moderator to make sure that you're getting routed to the right place? You know ideally once you've gone through authentication. We would love to be able to bypass Drupal And go directly to those API's okay So if you had another OAuth provider for instance then Drupal wouldn't even really be in the mix. It would all be client side It's direct. It would be the same. I mean, you know Jan Rain versus You know, I don't know some other external system. I mean really what we're looking for from them is a thumbs up or thumbs down And since we're trying to delegate those tokens to the client anyway, I doubt that well that we would ever want to go fully away Mostly because with all the different coordination points that we would need we probably are going to need an API To kind of make all those connections for since Drupal is really what we see being that coordinator And then once it makes all those connections and sends the data back to the browser Then the browser can go and start to function Autonomously, but that's really our goal. Thanks. Yeah Hi, Pat Miller from Scott's America. Oh, we have a very similar kind of history as you guys Usually had our Drupal site where site users who wanted to do personalized data would get a Drupal login Which calls all kinds of issues But we're still fighting through some we've separated our those users to a separate authentication system as well But we're still fighting caching issues. I mean so to go back to that prior question So you are do still have Drupal involved in not only pass the off But when somebody asked for personalized content that is still going to Drupal and then you're calling the PII system still from Drupal Yes, how do you deal with cat? I mean that makes so many more requests go to Drupal You don't have load or caching issues. You can't cash anything that we can so what we've done is we basically We so for this site we had about maybe five or six pages overall Everyone got the exact same five or six pages the place where things were different where we did you know kind of bus based on your session was On all the API calls, so we wrote a small Vue.js Application on the client side so anytime that you did have those personalized pieces of information Vue.js would You know check do you have a cookie? It can't really tell much about that cookie. It can't decrypt it But it can at least say okay. I'm gonna make it you seem like you're authenticated I'm going to make a request to get your dietary preferences and then display them So I'm going to make you know an Ajax call over to the server server is going to respond with another call I can't cash those because they're personalized as you mentioned But we found that You know they were lightweight enough And with the right balance of hardware we were able to handle that and the nice part is I mean full page renders take time We didn't have to do those so we weren't rendering the pages on a per user basis So we found that you know because we weren't doing that we weren't invoking You know all the different render layers on a per user basis We we scaled quite nicely Yeah, so Excuse me. Yeah, you mentioned in the beginning about Managing carts like it sounded like you had to deal with some commerce issues possibly we actually didn't for this service But we can let's pretend we did yeah, okay Yeah, so I guess I was just wondering like if you took that out of Drupal. Uh-huh. How did you manage? You know, how would you manage the cart in this in this case and it sounds like you didn't have that problem? So but but we didn't we may have had that yeah, I mean we would handle it just like any other You know this this PI is a service. It would be another API endpoint to us So just like we have at this PI I You know service we would have our cart on API and we would proxy those calls ideally We wouldn't be proxying we would you know get a token and then go direct Right, I guess and I think those are some of the implications that you have going this round Because you know Drupal commerce can manage that stuff for you, you know, there's like the cart on file module There's a lot of that stuff that's baked into Drupal commerce that I guess she would lose with this approach And you would have to go and build like a custom solution for it's basically what you're saying right and it depends on you know, which What what's right for you? Um in our case, you know, we were not We didn't have any plans to do anything with that date with the user data inside Drupal as a Drupal user That it was mostly a liability in our case There are other use cases where it'll be far more useful And in those cases this might not be the right solution because you do lose the richness of that Drupal ecosystem, right? So definitely with like, you know, if if you're a Drupal commerce user then this is probably not the right approach, right? Okay, I just want to make sure that that wasn't a piece of this Hi, I'm just wondering how and when do you update cookie exploration timestamp? Only so we do it when someone logs in we set that that time stamp Eventually it will expire and they'll get kicked out and they'll log in again And they'll get an updated cookie with a new time stamp Otherwise, you know, the only other time that it becomes an issue is when we have like a heart bleed scenario Like we mentioned we need to take everyone out. We set the global value and then the authentication system says, okay You're not valid. I'm gonna 403 you and then in that case they reauthenticate Okay, so if I'm not again today and my cookie expired a month for today If I am on the site like a second before the expiration, he would just automatically land me out then correct. Okay. Yeah Had it been a just a d8 on the site Would you have considered using the big pipe module for personalized information on the site? We had it definitely was something I had considered I don't think we would have I think be mostly because we still wanted to avoid those full page refreshes We still that there still would have been implications with all the different caching layers And it the we we still would have been doing full page refreshes Ultimately, I mean it's it would make them better But you know ultimately it's you know, Vue.js allowed us to create a much much more, you know Responsible I hate to say responsive, but you know reactive You know user experience So they would click something and then something else would happen So it was much more dynamic for the end user, which I don't think you're gonna get just from big pipe Yeah, so how applicable would a solution like this be to an ecosystem of websites? Which all share SSO and some of which are very locked down by roles So you can't go to a lot of past like 100,000 nodes that you only certain people can see It does you think this actually would provide a lot of advantage because it seems like we still have the same problem like Drupal still has to wake up like you know, just get to varnish, you know, just peel a layer of varnish It depends on what what exactly they're trying to do with the site because I mean you're still going to You know are these are people who all have different roles, I'm assuming Yeah, there are maybe like five roles across a bunch of many many people. Is it the same roles across the sites? Yes, they would they're unified now So what you could potentially do then is you could you know have them log in on a primary site And then encode the roles You know roles and any other attributes into the cookie Mm-hmm, and then they could take that with them You could tie in to some of the authentication systems inside Drupal to then You know create kind of an ad hoc user for them that you know has that role give them those permissions And that could work the only issue that you might have is in some cases when you edit content You may need an actual like database user Yeah, so what you could what it could do is it could on the fly just create a user for them to then Map that data to like when they authenticate. I guess I'm just not seeing where the savings would come in for that case There might not be there's not in the scenarios that that would lean on that user Yeah, I mean when I think where this starts to not work is when you actually need to have a proper user Like a Drupal user itself right and that's where where this loses its value I think so what on one of our sites is just like publicly accessible and Getting hello user makes the render ten times longer, right? Yeah So that's a case where we could clearly change it But if there are more complicated sites where you have access to forums and things like that I think once you once you go down the route of creating a row in that user table organic groups You're back to where you are right so right you haven't gained anything so I think in your case the question I would be asking is more of Is it is it acceptable to have a disruption in that user flow to where when they need to get that taken care of They're asked for additional Yeah, you know and then from your standpoint is that worth is that worth the effort to to roll it out But yeah, it probably isn't You know if you need the user table this might not be the right approach if you don't need the user table And this is awesome in our case We had we had gigs and gigs of users that were just weighing us down like an anchor our database right now It's like 25 megs. We can drag it between environments. It's awesome. Like it was like six I mean like it's it's like hopefully this is a case study about large client databases It's Yeah, and it's helped our developer velocity. It's helped You know, we can just pull the database down We don't really have to worry about sanitizing too much now because of that So we're in a much better place than we were We probably have time for one more quick one and then Adam and I'll say afterwards for another 10 minutes If you have any one-on-one questions Anyone go on one twice cool Thank you What did you guys use for the token encryption? Secret sauce Something good. Thanks everyone for the GDPR stuff you mean or just in general like We internationalized through it actually that's doing all the detection and that's how we know You know, you come from the UK. We need to change our form to do a fundamental prefix And have additional terms of service and all Yeah, so so in order to register you have to their terms of conditions on the page and they are changed for those international users so You know fast we're using fastly. They can do the geo detection on the edge. They then You know add that to the HTTP headers. We vary by those HTTP headers. So everyone gets a different version Their implications for so we use that sparingly. We don't want to do that on all pages Yeah, but on the ones that we care about we do that Yeah Hey You've worked with Sakurada and to halt. Yeah We did yeah, we use the API Yeah, they're the the built-in forms and stuff For what it's called, but they're like form builders. Definitely a little it's a little heavy So what are the things that are challenging you? Okay, so so you so you're having jam