 Okay guys, we're going to start now. Welcome Nick to the stage. Hey, how's it going? Everyone still alive, two o'clock on a Friday afternoon? Cool. Okay, so yeah, let's get started. So my name's Nick Shoe. I'm SysOps lead at Previous Next. I get to do some pretty cool stuff in my day job. Just around containers and kubernetes and as you can see by the shirt and yeah and then joining it together with Drupal. So yeah, let's just go through the slide first. So I like to give everyone a bit of a warning on what's going to happen. So the first thing is a bit of theory around what microservices are and then we'll go into and see where does Drupal actually fit into a microservices architecture? Where do we fit into this sort of new paradigm of things? And that'll take up about half the talk and then the second half is a pretty ambitious demo. So demo gods, please be with me today. And then plenty of questions after, so cool. So like all good talks and like all good things that spur people to action, this talk was born out of at the end of the Drupal 8 cycle, we've had a lot of examples around like extending the front end. There's been a lot of stuff around like how can we throw something in front of Drupal 8 to consume the API, but we haven't really had like we've put all this work into like re-plumbing it in Drupal 8 to be OO and to be extendable, but we haven't really talked about what we can do under the hood. So this is kind of, so I took that and then I took microservices and sort of formed a bit of an experiment to undergo up till today. So first we'll get into what are microservices. So I'll read it up, the term microservice architecture sprung up over the last few years to describe a particular way of designing software applications as suites of independently deployable services. While there is no precise definition of this architectural style, there is a certain common characteristics around organization, characteristics around organization, around business capability, related deployment, intelligence and end points, and decentralized controls of languages. So I think this is a pretty extensive quote. So I'll break it down into the things, into three. So business capability, so small deployable units that are designed around a specific domain, intelligence, so a well-designed API. So we can plug these things together and chain them into bigger applications. And my favorite, which is control of languages and data, which just means that I can do a little bit of go-lang in this talk. So if you want to read a little bit more, I highly recommend this book. This definitely took me through a bit of a journey, and it might be a little bit of a bane in the existence of a few previous nexters, just asking questions and saying why did you go with this, how about this, check out this book. But it's an awesome read. But for me, I like to think of them as interchangeable, so that's the well-defined API part. We can swap it out with whatever we want. Simple, so small programs, not massive, massive programs, lots and lots of logic, very hard to read. Keep them simple. And the final one is do one thing and do it well. And I think we've all heard this last one somewhere else, and that's in the UNIX philosophy. This is my credo. This is the thing that we should all be reading out every morning when we go to work. I think, so write programs that do one thing, do it well. Write programs that work together. Write programs to handle text streams, because that is a universal interface. It's a little bit dated, but I think we can swap that one out for, yeah. But yeah, and that's also worth a read, too. So I didn't realize that there was also an older edition, which even that is like, this is very generic. The version prior was even deeper around tooling and things like that, which is really, really cool. So definitely worth a read. But a good example of this is the command line, right? So here's a really cool example in my research. So essentially a reporter put out the word to Mr. Donald and said, take this file, tell me how many occurrences of a word exist in this file, and then order it. And he came back with a massive, massive program written in a language, written in Pascal. And Doug McIlroy, who invented the pipe, or was the founder of the pipe in UNIX, came back and basically just slapped him in the face with like a couple of lines of bash or of shell. But at the same time, I think we can say like these programs, you know, much more than just a couple of lines here, there's, you know, there's a fair bit to make up, sort and unique. But the key difference here is it's simple to read and you can swap it out and play with it however you want. So it was, yeah, very, very easy to compose new and fun things, basically. Cool. So, kind of the elephant in the room. So we're going to get this done. So Drupal is a monolith, right? There's no two ways about it. Drupal is a monolith. And that's all right. That's fine. I think the key way, like the key point here, or the key thing that points it out is the fact that we center everything around our Drupal site. We build this Drupal site and then we extend it and the common cases here are things like cache search database, we push logging, we integrate with other services but at the end of the day, like the canonical source is Drupal and that's where all the code goes. That's where everyone generally interacts. And now we're talking about throwing a whole bunch of front-ends in front of it as well, which is fine. That's all good. But when I see this diagram, I can't help but think about maybe we've just got like the Death Star. Or maybe it's more like Independence Day, like the mothership and kill the mothership and the rest of it anyway. But so where do we fit? So in a microservices architecture, we split everything out, right? We split everything out into deployable units. This is a very simple example of how somebody might compose it. So you might have multiple front-ends and then you'll have multiple APIs, these can be all different teams and then you'll have some data stores in the back. But you might have say front-end one that just serves the blog and then you've got maybe front-end two that's for like subscriptions and payments and then it's got like a little thing at the top that loads a recent blog, like this is how they all intertwine but they're able to do that through these APIs in the middle. So going into this talk initially, I was thinking about just extending a back-end service and essentially giving us the Death Star. But something dawned on me, which is kind of our bread and butter of this whole thing, which is Drupal is a great content editor experience, or at least that's what we strive for it to be. But when we build our sites, that's what we focus on where a CMS. So why can't we be another front-end basically? Why can't we be an administrator front-end that's a tool almost an IDE for content editors that connects into all these services and then provides them with a really cool, really nice content editing experience. And that way you still have your APIs in the back. You still have your front-ends and we can play nice. We can move into these areas at the same time. We don't need to make people change to us. We can come in, connect, and show them what we can do. So this is where I see Drupal. So at the same time, I banged on content editor experience, but there's a whole bunch more. So when we're talking about all these different services here, you have to have standards around these things, like authentication, logging. The list goes down and down and it would be a checklist basically and you have to make sure everyone's doing the same thing. Everyone's pushing logs to central logging. Yada, yada, yada. And I think we have a lot of items that lend well to this as well. We can push logging to central logging servers. We can bake into other people's authentication. Mail and search are pretty easy too. So I think we can still extend the backend while we also work together with this architecture and embed ourselves in there. Cool. So this got me to thinking that I would write a pretty ambitious demo. And before I get into that, an example that we do a previous next on an existing project, or the closest thing that we have to this in the real world is something like Migrate or Feeds. So we're running a batch process. We're syncing data into us, but at the same time, we're still treating ourselves as the canonical source of truth. We're syncing data in. We can't really change it and that's kind of it. So this is kind of, to me, this is our offering in Drupal 7 right now. But with an OO architecture in Drupal 8, we can do so much more. We have so much more available to us. So we have content entities which are really, really fun and I'm going to show you through that. But then you can also override the storage. So you don't need a table in your database anymore. You can just point at another bit of storage in your way and it's also contrib-free baked into core. Cool. So let's get to it. So this is the architecture. I've removed the storage layer. So the second I kill my instances, there goes all data. That's all right. So we're going to have a front-end. We're just going to have a little JavaScript front-end. We're going to have a blog API in the middle and these are going to be interacting independent and then we'll have Drupal come in and be a content editor experience or a basic content editing experience. Cool. No more slides? Cool. Cool. So let's clear this first. So I've composed all my endpoints or all my microservices into containers. So don't worry. You don't need to know too much about containers. It's fine. But that's just what I'm using under the hood and if you want to go into it, this entire presentation with all the components are going to be available or are available online. So there's no worry there. But I'll just quickly show you how this is wired together to resemble that diagram. So we have our front-end, port 80. That's our JavaScript. We have our blog API. And then we've got Drupal. And we also have MySQL. We still need to install Drupal. And then I also have some custom modules just mounted in. So key takeaways, port 80, 8080, and 81. 80, 81. Cool. So let's just go ahead and spin that up, shall we? So the first thing, actually, let's do one thing first. So we'll go to our Drupal. Let's install our Drupal first, right? So these are just some basic credentials for the MySQL side of things. Cool. So we can leave that go off. Sweet. Okay. So let's get into an existing microservices architecture or a demo existing microservices architecture. So I've started up my components. And I have this API. And as you can see, I've got like a very basic sort of interface for creating, deleting, updating my blogs and listing them as well. It's very, very simple. It's written in Go. I'm not going to subject anyone to Go, but check it out there. This is to prove that we can play well with other languages. But at the same time, we have a JavaScript front-end pulling from there. And it's actually quite simple. That's the index.html. And then we just do a JSONP and just pull from the listings, right? Super, super basic. This is the blog. Pretty, pretty basic. And this is the API. But we have no data. So let's throw in some data, shall we? Like all good blogs. We need some data. That's not very pretty. So let's go to the page. So I have some top-notch data here. Sweet. We're ready. Ship it. Put it live. It's all good, right? Nobody's ever going to want to change that. It's there forever. I will donate it to Drupal South. They can use it next year. We're ready. Cool. So these are the two components. We've covered those off pretty well for all in all. They're working together. But now I want to change some of these titles. I want to get in there, right? I want, yeah, it's not very accurate right now. So let's just quickly install the site. I'm doing this because I just want to make sure that, you know, there's no aces up my sleeve or anything. So that's just nervous typing. Cool. So we're just going to get the standard profile, right? That's not very cool. That's not very pretty. So let's have a quick look at the Drupal component itself. So what I've deployed here. This is what's closest to us and what we will be implementing. So I have a custom blog module. It was bootstrapped from Drupal console. And then it was extended from there. It's a content entity. And let's just scroll down a bit just to show you the fields to get you a bit of an idea of what's going on. So we have an ID. Ah, we have a title and we have a body. So it mirrors the content on the other side. So what this would mean is if we were coming into and consuming someone else's API, we would have to have a knowledge of their API, understand their API, and then we can start to build these fields and pick and choose what we want. But there's no table created when this is created, right? So we need to do some storage handling. So for now, just a quick under the hood, we're actually overriding key value storage for this content entity. And this gets passed through and all this code's online, but essentially what you end up with is this class here, right? And if we scroll down a bit, this starts to feel a little bit like storage. So get multiple, get all, set, rename. Some of it's not supported. Delete multiple, so it's all there, right? So let's enable the module and see how we go, right? So once kicking off, just some shortcomings here that I want to flesh out a bit more is when extending this and creating this, Drupal still has a pretty first-class idea of what a content entity should be. So there's things like owner ID. There was, like, is published. So you can mock some of that out, essentially. So owner is nope. There's no... But yeah, you can definitely get there. So let's go back to our site. The blog's enabled. So I'll just quickly jump down to the menu. So we're actually deploying this like it's content on the site. So we're chucking it under admin content. So if we go there, we'll see now that there's a tab across the top here, right? So if I've done everything right, we'll load this tab and we should see some content. Sweet. There we go. We did it. Now here's the fun bit. So it took maybe an hour or two for Leonite to wire this up. But then the update was the interesting bit. So let's make a change. You know what? This isn't very accurate. I believe there's 44 cutest pastries. So updated. Let's just make sure that Drupal's not lying to us. 44. Cool. And on that note, I will conclude. And we can go to questions. Yeah, it was, but there's going to be tons of... Yeah, I'll take a sip of water and... Okay. Who would like to ask the first question? Drupal? Question index zero. Have you thought about adding console for service discovery of the microservices? So I've contemplated from here we could... So is everyone familiar with Swagger or OpenAPI? So basically it's a standard for creating an API. And then you can bootstrap servers and clients off it. And it gives you this shell that then you can fill in. So I think a really good extension from this would be to consume that API and create these content entities off that, essentially. So I think that's a great way to do it. And we can model it off that. So that could definitely be something as an extension on the console. So we saw there that you weren't satisfying the whole content entity interface. So returning nope and other kind of weirdness to work around that. The shortcomings of that particular API. So did you notice Drupal breaking in any weird ways where you were doing weird things around just not having fully fleshed out the implementation of that interface? Yeah, it was definitely around those flags there. So I basically shelled out all those fields one night and then spent the next couple debugging until the early hours. So yeah, it's definitely around the... So we can go back and have a bit of a play. So, whoops, wrong one. So yeah, published owner. These are all that I had to keep set and name. Set and get name are locked in. So my initial reaction was to do get is and set is and set is a title. But yeah, didn't like that. So the parent class actually goes through and goes get title. So that blew up in interesting and extreme ways. So these are definitely things that I want to flesh out and dig in a little bit more to get into. Because yeah, I honestly think that maybe it's me and not Drupal. And that's pretty optimistic. So I'm feeling optimistic. So we'll see how we go. But yeah, owner published and get is and set is on name. There was I guess at the same time, then once you fleshed that out, it's all about just wrangling the storage essentially. So you can pass in guzzle and then essentially start to do just straight guzzle calls and have a bit of fun there. Some of the other wrangling in the similar vein were around languages. So as you can see, my API back here is passing out just like a straight. That's just because let's go over here. That's easier. I was wondering why that didn't load. So as you can see, it's very structured in the way it's coming out. But it has no concept of languages and multiple values. So in here, there was a bunch more wrangling to do to handle that at the same time, which is fine for better or for worse. Yeah, if I was to extend this out, I would probably make those helpers or something like that. But yeah, for now, it's just sort of handling that at the same time. And yeah, we also had to make sure the ID was set. And so it was actually pretty hard to conform to that. And as Lee would attest, I needed to use a debugger. That was the only way to work it out. So maybe that's some other work that we need to do around documenting this kind of thing. And that may just fall to me when we release this as a blog post or something like that too. That's what everyone knows. Can we make views? Yeah, so I believe it's exposed as view starter, but I haven't had a play yet. Actually, I think we shelled that out. Yeah, I think views got ditched. But it is MVP. So my question is that are there any kind of modules, real life modules out there in the wild that are doing the same thing that someone could actually look at and see the same approach in play? Not in what I found. So I did a bit of a looker. Okay, loaded. Sorry, it's not exactly the same thing, but I wrote a system for Drip or Seven. It's just on GitHub and the idea behind it was that you could plug in any API, be it like the Facebook API, the Twitter API, you build the API structure in your Drupal site and then it would expose to certain systems within Drupal. So one of them was views. So you could make a view of a Twitter API call without ever actually storing the data. So it's just doing the get via the API directly, that sort of thing. It's not exactly the same, but it's similar. Yeah, yeah. There was also in Drupal 8, there was like, I just went on a massive Google rampage and there was external entities module, but there wasn't, I couldn't really find any good documentation and it didn't really lead me in the right direction, to be honest. And yeah, bootstrapping this and then going through the steps. Yeah, actually, and I got to say like a big shout out to Lee for helping with the storage because I had the content entity, but then actually wiring through the storage was pretty tricky, but a lot of fun. So I think a lot of this stuff, it comes down to having some good use cases for it, which is kind of what spurred this whole thing on. There's lots of use cases of things consuming the Drupal API, but we just don't want to let our data go. Like we want things to consume us, but we don't want to go and consume somebody else's API and pull it and use it as the canonical. So that's, yeah, I think we just need some solid use cases around that. And that's definitely something I'm going to keep pushing for from here. Well, we see an extended example with views in the next Drupal South. Hey, I'm down. Yeah, let's do it. Anyone? Thank you. I saw that we cannot support search right now with this content search. We cannot support content search. Yeah, so do we need a new search plugin for that, which will curie this back end or something? What is the goal? Like how can we tackle this problem searching of content in the Drupal? Yeah, yeah. Honestly, I'm not sure to be frank, but yeah, I got to this point, but I haven't sort of gone the next step. So there's, I guess, yeah, definitely some of the gaps are around, say like admin content, how it doesn't show up in there. It's its own content, which is fine if you can educate people in that direction. But what about a microservice for the search and send all that data off into an index? Yeah, absolutely. It'll be nice to pull it all through and then send it through Search API or something like that. You don't want to join anything, you'll be fine. Any more? What's your thoughts on transactions that span multiple microservices? Okay, yep. Sorry, what are my thoughts on it? Yeah, what are your thoughts? So say you have two microservices, they both make a database change in some way, and the second one fails. How do you roll back? Yeah, yeah, fair call. I think a lot of that logic should definitely fall into the third, like the other API and to the third party API. I'm not entirely, I wouldn't be entirely happy to talk to sort of two APIs at the same time to fulfill a transaction of that nature because that would mean that, like depending on what it is, but it would start to throw a whole bunch of flags that maybe the model around that external API isn't fulfilling that need. But it depends. Like some, yeah, some linking of data could just be straight through the body, like someone puts a link to something, like the blog and sends it off to an article, API or something like that. But yeah, I think the first transaction should still go through regardless and then let the second one fail and log it and make sure the end user knows that that happened. And it just depends on the nature of that second transaction and how it's linked together, I'd say. Cool. Do you have music like they do? I got some sweet jams on this thing. I can plug it in. Okay, so if there's a number of questions, please give Nick such a round.