 Thank you everyone for coming to the session. We can kick off. So the topic me and my colleague will be presenting is modernizing Drupal in the enterprise. This session was originally to be presented by our colleague Ashish Thakur. Unfortunately he is not here. Don't be sad it's his visa that's still processing at the immigration department. But we are working in the same team, working on the same project. So we thought we'll just come in and on his behalf present what we have. Now one thing that we have been seeing in the enterprises, we work with a lot of enterprises, the SMEs and also the bigger ones. There is a trend that everyone is going through a massive digital transformation, the kind of legacy systems they had. They're moving out of that and getting into more cloud sort of architectures, more distributed architectures. And if I have to categorize that on a very high level, one is the hybrid infrastructure where you are using a lot of terraforms, a lot of scripting to build infrastructure that's agnostic of the platform that you're using. Either you are hosting it on AWS Azure or GCP. Containerization, Kubernetes, Docker, they have become bigger than ever. One of the most important aspects in the digital transformation nowadays. Microservices, the distributed architecture, that's definitely one of the high things. And a couple of more things that have become even more mainstream in the last three or four years. That has been there forever. But API firsts firsts. So rather than API being an kind of an add-on to your existing systems or the auxiliary kind of thing, they have become like the primary thing when you define your architecture and a lot of reactive or the event-driven architectures. So any event that is happening at one place, they trigger something in a different system. And what we are seeing more and more now and it's common saying that data is the new petrol or new gas. So a lot of data products coming in the market. And why this modernization is happening or why we are seeing so much of money being pumped into digital transformation. They are primarily just three things. One is to make money. People want to diversify. They need to identify more avenues to make money. Second one is to manage the risk. With the distributed architecture, there is more risk involved. There could be a system that goes down, very difficult to monitor and ensure that everything is up at all times and business does not suffer any losses. And the last one, as it is said, money saved is money made. So saving money in the operations, making it even more robust in the systems and reducing the costs of hosting stale infrastructures and all of that. So on a very high level, if I have to speak about and this architecture may resonate with all of you. The modern architecture when we are talking about distributed systems, there are multiple components to it. And every component is handled by a specialized service or a specialized tool for API management. We have got Kong or Apigee or WSO2, those kind of things for event streaming. We have got Kafka for live reporting. We have got Graphen and these things or these tools, they work in a specific niche and they do what they do in the best way possible. So either we are making API products or data products, API management, there are like specific commerce things, networking, security storage, all these components, they make up an enterprise architecture. They interact with each other through APIs or different protocols. And that is all what we see. The components could be different, but on a high level, 30,000 feet, this is what we have. Now that poses a very important question. And that is, is Drupal ready for the enterprises? Is it a monolithic stack as we have known it from Drupal 5 or Drupal 6 or it has evolved into something that is more robust than ever that can compete in the enterprise level or have the capabilities that we can have the separation of concerns? With that thought, I'll hand it over to my colleague who is more technical than me and can actually take you through some use cases. Okay, hi everyone. So I think this is my first presentation Drupal South. So I hope I don't have any other stage for something like that. So okay. So I'll just proceed from here. So continuing on the previous slide, like the constraints that we talked about, like whether this is a monolithic stack and how we can separate out all the concerns. So if we move ahead, so what are the kinds of constraints that we are facing in here? So some of the constraints which exist in the Drupal ecosystem, which most of us are familiar with like difficult to scale subsystems. This is a problem with most of the monolithic systems. For example, while we value would require a new functionality in Drupal, there is a model for that, right? Or maybe we use Drupal APIs and and like build a custom model around it. So this leads to challenging and scaling, for example. So basically just like, so I'll be just going through all the constraints and then I'll be going deeper into that. Okay, due to the above challenges, our team might not prefer Drupal to build high velocity and low latency systems in Drupal. Thirdly, there is a lack of documentation on well defined integration architectures around Drupal. The reason being is let us consider the case of the API concerns, for example. So API is something like suppose we have cars, or suppose we have like rate limiting, we have authentication, we have authorization. So we have all the modules available in Drupal. And we are basically giving all the responsibility to the application code itself. So instead of giving all that API concerns to the application code itself, we can basically delegate or like offload all those responsibilities to like API gateways or proxy APIs. So like that is one of the like, like that is one of the things that we can do. And probably we can use Drupal just as like, like we can use as content as a service. Basically, it's a basically, we can just consider Drupal as a platform for just content. So let us look at one of the cases, for example, let us suppose we want to build a news site. Okay, so on a news side, we might be having different, like different categories like sports, politics, technology, and on like every on like on a daily basis, we are actually publishing a lot of content. So there can be a million of content altogether. And on a daily basis, let us suppose we can consider one of the publication houses, they might be having different publications across different regions. So there might be a lot of content. They might be getting published into the different categories. So suppose if a new content is being published into the sports category, for example, so there are only 200,000 users we have which have already subscribed to that particular category. So they need to get a notification on their devices. Okay, this is the new content that has been published. So if we offload this particular or suppose if Drupal is supposed to do this thing, so it will be quite difficult for Drupal to scale. Because in this case, we are just seeing notification as a service. Now Drupal as a whole it is having like if we consider Drupal as a whole application, you can see the subsystem one that we have described there. So that is responsible for the notification system. But when we are scaling Drupal, so the other subsystems which is not frequently used, so they are also so like they are also getting scaled when we are scaling the whole application altogether. So like that is one of the constraints. Moving ahead. Yep. So then the other thing is like, so like that was one of the event driven system. Now coming on to the other concern that we had was the API concerns. So we already addressed all this, like all these issues currently we are doing everything into the application code itself, rather than rather than doing to the application code, we are trying to like move all these API concerns, like cars, authentication authorization, throttling, or like rate limiting to some API gateways. So despite these constraints that we have mentioned before, so or any of the constraints that might come ahead. So basically Drupal has always evolved. And during the times of we see like from Drupal 7 to Drupal 8 to Drupal 9 to 10. So it has always been providing us many opportunities by which we can extend Drupal, it can be like by giving us the headless architecture, or like suppose we can go with microservices or like other sorts of services into it. So how Drupal has evolved from being just a web content management system to additional experience platform. So I'll just give you some figures for example. So if we look at the top 10k sites that we have across the world, around 10% of this around 10.6% of the sites and it's already growing, it's like built in Drupal and around like 1.7 million websites around like they run on Drupal. And there is like one of like one of the Drupal pass providers named Aqua. So it has been recognized as a leader into the Gartner magic quadrant for additional experience platforms. So how Drupal has evolved through the ages. So like as we are moving towards more AI powered systems, so Drupal has also like in the previous session as we had seen, so Drupal has started to integrate itself with certain AI tools to like give us recommendation based engines or like or like text metry. So basically we so basically Drupal has always provided as an option to like extend those things. The other thing is like we like I think some guy basically to discuss about the headless and the decoupled architecture. So basically headless and the decoupled architecture is also quite a word and it can basically couple with different sorts of like the like the front end tools and it can basically act as a microservice in a federated system. Now and the other thing is like as I mentioned, like moving towards an API first approach. So Drupal can basically be a kind of an omnichannel CMS, via which we can embed the content to different devices. It can either be like for example, it can like it can either be kind of the variables or the mobile phones or like Alexa's or like kiosks or the smart TV is anything. So basically Drupal can act as just as a content delivery system. Now now moving towards the specific concerns. So in order of Drupal to converge with modern architecture and build a charter for the modern CMS platform, there are some of the things that we can do. First of all, we can externalize the API concerns. The API concerns like rate limiting, authentication authorization, etc. are best handled by the API management tools and then offloading the Drupal events into an external event stream. Most of the most of the Drupal modules are built using concepts of Drupal Symphony events or hooks. The model would execute a business logic when an event occurs. So suppose suppose when we are like creating an article, so basically events like it emits an event which seems like okay, an article has been created. So we can so we also so we just need to restrict that particular job to Drupal like okay, it can just emit that event okay, an article has created on that particular event. So we can have event-driven systems like suppose there might be a notification service, for example, so that particular job can be offloaded to a different system. So like how are we going to do that? So this is a kind of a modern architecture that we can look at. So if you see onto the Drupal side, so we are so like we are basically aspiring to like build two modules around it. That is first thing is a Kafka module and then and then the other thing is a Kong API Publisher module. So what the Kafka module do like it does is so suppose when we are having any events onto the Drupal system, so it is basically delegating that all those all those events via producers to the Kafka brokers and via the Kafka brokers, it is going on to different consumers and via the consumers different different like microservices, it can be microservice or there can be third party services who can consume all this data and then they can make something out of that. So the only responsibility of Drupal will be to just emit the events and similarly going on to the API concerns like ingress firewall, routing security, traffic management, caching analytics or transformation of the APIs. So everything needs to be handled by a specialized application. In this case, we are going on the com. There might be some other applications as well, some API proxy applications or the other API gateways that we can go ahead with. So now moving on to like what exactly is Kafka and the nutshell. So Kafka is basically an open source distributing event platform system. So mostly so mostly like Kafka is like like like like if we see Kafka in a nutshell, so basically there is a kind of a producer and then we have then we have a consumer and then suppose we have a source system and then we have a target system. So suppose if the source system wants to emit certain events to the target system, so it is basically emitting those events and those events are basically emitted via the help of the producers. So the producers are emitting all those events in the form of data to the different topics and those and like in topics are basically the pockets of data into Kafka and then they are transmitted to the consumers and by the consumers, these are being consumed by different third party applications. So this is what Kafka is in the very nutshell. So there are a lot of things into it. If I go if I go into the nitty gritty, it will take the whole session. So I'll just move ahead. And what we are basically aspiring to do in this particular module is that like as I mentioned, like we are creating a model wherein like we will be like suppose any of the events are happening onto the triple side. So we are basically offloading all this events to the lecture Kafka and from Kafka, we are basically delegating those events to the different consumers and from there on we can have any services. It can be a frontend application or it can be any SDKs or like it can be any other application that might need that particular data. And similarly for Kong as well. So, so suppose in Drupal we have different entities. It can be the user entity or it can be the node entity or a custom entity. So suppose when we want to expose that particular piece of information, so we are able to expose that why the rest APS or the JSON API is out of the box. So, so in order to provide security, so what we can do is we can basically offload all those APIs onto the call and call will be the single point of reference for all the external systems to like consume these APIs. So basically all these all these other like like all these other responsibilities of the API concerns will be offloaded to the API gateway and triple will only be concerned about like how like how they are going to like like they are just being treated as a content as a service and a content platform and nothing else. So I will be just giving you a certain use cases around that. Like I think you all might have used Google and Google we have something called the Google accounts. So therein you can go and manage our accounts for example. So for example if you are building up a system and you're building an account management system okay and suppose if you're having around like one million users or like on your application and suppose you have also initiated a kind of a subscription module for example and and like we know like we have different third-party IDM's like Azure AD or Octa which provide us the identity management solution. So there is something called the SCIM standards which says like the system for cross-domain identity management. So what we can do is we can expose the user entity like like by the rest APIs or by the JSON APIs that the Drupal code does out of the box. Now what we can do is on to the con gateway. So before we move on to the con gateway so we can define the open API specifications or the swagger specifications that we like. So firstly we need to firstly define the open API specifications as per the SCIM standards and then on to the API con gateway we can basically write a transformation layer. The transformation layer what they will do is. So suppose if there is a provisioning system like Azure AD for example. So Azure AD complies with the SCIM standards and then octa also complies with the SCIM standards. Suppose if they want to provision a user onto a system for example. So then then we have actually transformed our user APIs on the basis of the SCIM standards. So when so suppose at first the business has decided to go with the Azure AD for example. Suppose if Azure AD is provisioning that user onto a Drupal system. So since they both are complying with the SCIM standards it will be quite generic for them to like push or like provision that user onto our system. And later on suppose I suppose like two years down the line or three years down the line. If suppose if the selects like suppose if the business decides okay now Azure AD is not good we have to move away from this particular IDM to octa. So then in that case it'll be just like it'll be just like plug and play. We can just remove Azure AD and we can just plug in octa and then it'll like it is all good to go after that. So basically all that transformation and all that caching needs to happen onto the API gateway. So that is one of the use cases and then the other use cases is like suppose when you are building up like nowadays when we have certain large scale applications or mid-scale applications or the small scale applications. For suppose various sites the main source of income is advertising for example. So how we can implement this event of loading and using this API gateway to create a very robust solution around like how like like a very robust solution around creating a very dynamic environment for the advertising management. And suppose suppose onto the front end if if we have suppose we are building a decoupled system for example. Onto the decoupled system we are having the home page we are having the article pages we are the search pages. So firstly we have decided like okay these are the kinds of ads we need to deliver there. Suppose it might be the static ads it might be the programmatic ads or it can be the video ads it can be tees. And suppose we also need to implement the real-time bidding for example the header bidding that we call it as. So then in that case what we can do is onto like onto Drupal we can define certain entity types wherein like we can provide a kind of an entity design wherein the user can provide an advertisement like an ad start configuration. In the ad start configuration the user can provide okay this particular ad start configuration is quite specific for the desktop sites for the mobile applications or for the tablet applications. So they can basically define that ad start information there. After they define the ad start information all this information is being exposed via APIs by the rest APIs or the json APIs. And in this case firstly we'll be going with the API first approach. So what will happen with the API first approaches we will be first defining the the open API specifications like how that particular API will be delivered to the consumers from the source which is Drupal. So once we'll define the API the API the API specifications so then the end systems or the consumer systems they can start building in parallel even though when we have not built our particular APIs under Drupal by using the mock data because at the end of the day the the response that they are anticipating will be in the same format from the Drupal system. So they both can start building the application in like in parallel and that will basically speed up the whole process. So now we have now since we have exposed all the slot data so this slot data has been transferred to the front end applications and along with that we will be creating a kind of a ad SDK. The ad SDK will basically involve it can be built in next year's or like any other like languages that like that we like or like that script on like on a separation. So basically we defined an ad SDK so ad SDK is something what it does is it will basically implement the GPT libraries in case if we are going with the Google ads for example and then it will basically consume the hydrated response from our site and then it will basically render all the ads into the slots. So how are those slots going to be get created there? So the front end application will be basically be consuming a less hydrated response from our application and it will basically create the ad slot placeholders everywhere on the site. Now our SDKs included onto the site and it will basically include the static ads the video ads as well as the suppose then we are including the programmatic ads for example like as many of us have used Instagram or like any other applications wherein we are just swiping in all the photos and after we swipe like two or three photos we are getting an ad and suppose when we are scrolling through our feeds we are getting ads after certain interval. So those are basically the programmatic ads and we are basically defining in all these configurations onto the Drupal site. So Drupal will basically act as a kind of a source to like to like basically emit out all the all these configurations and then our API gateway that is the Kong layer the Kong layer will basically transform this API and will render to the consumer applications. So like in this way we can basically have a kind of a very dynamic experience to render all the ads and now the other question is like that we can address here is the concern of cache warming and like the cache invalidation. So these are the two major problems that we need to address. So what we can do here is suppose when an entity is created onto Drupal so in that case it will basically fire an event. So then we can basically use a Kafka producer that Kafka producer will basically send out that like send out that event to the consumer and after sending out that event to the consumer it will basically pre-warm the cache like suppose that that particular configuration is all is like already stored onto the API gateway that is the Kong. So suppose if the consumer applications are requesting for that ads or information so in no case the call will go to the Drupal application. So there will be a chance of 99% of the times the results will be served from the API gateway rather than going on to the Drupal application. Hence like most of us have this question like okay when we are when like when we are hosting and content from this cache then it's stale like what is the TTL there. So basically when we use this concept of cache warming using Kafka like using Kafka. So then basically we are always warming the cache and at all the times the content that is being rendered to the consumer applications is rendered from the API gateway itself. So like that is the another use case that we can think of. So with this I presumably end my session so if you have any questions you can take that out. Sure yeah please. All right. You may have said about the Kong thing. Yeah. And instead of handling the event can I handle like the Drupal event or is it like the party event? So the events are basically handled by the Kafka site for example. So from so from Drupal there's suppose when we are emitting an event for example. So then what we can do is we have the we have the event subscribers right. Why are the event subscribers? We will be we will be basically implementing the producer code of Kafka and we and why those producers will be creating certain topics okay. So this so when we create this topics this will be this will be sent out to the Kafka cluster. So on to the Kafka cluster we have this topics and it will be transmitted to the consumers. And the consumer here will be our API con gateway. Okay can there be another instance of Drupal I can find it and then just run some code that doesn't have to be denying? Yeah yeah so yeah so probably like it depends actually suppose it depends upon which like which concern are we going to scale. Suppose if I like I gave you the example of the notification system for example. Suppose if my users are increasing for example there are 200,000 users or like 500,000 users. So in that case Drupal is going to emit a lot of events right in that case since Kafka serverless and it can like it can scale horizontally okay and and and and like it can and in that particular Kafka cluster it can basically add the brokers on fly and all the and all the topic partitions will be distributed evenly across all the like all the brokers and then it will be sent to the required consumers. Yeah and the other question that you had was like when we are like whether we can replicate the Drupal instances. Yes we can replicate the Drupal instances in in and in that case there will be like depending upon the number of number of replicas that we are having in for the Drupal whether we are suppose if you're horizontally scaling for example like suppose we have three instances for example so then there will be three instances of producers as well. Yeah I had a I had a small like a kind of a demo so basically the model that we are going to write so basically it's still under process and so I've just set up a kind of a very like a kind of Kafka on my local. So I have I have started at the zoo the zookeeper and then the Kafka server onto my local so currently I'm just having on my cluster I'm just having a single broker and via this broker firstly I'll be creating so this so via this so like these are the so like these are the CLI commands but there is a library in PHP which is known as the RD Kafka for example okay so by using RD Kafka through our event subscribers we can publish all these topics to like to like to like like to work Kafka brokers and then our consumers so here I'll just create one of the topics suppose if I created this topic group a demo and onto this group a demo I'm creating a kind of a producer and suppose if I have actually started a producer here and now I will be just creating some consumers now if I publish anything for example so you can see like all my so like these are basically the the topics like the messages inside the topics that are being sent to the consumers and if you see here I have created one of the Kafka consume like one of the Kafka consumers which is belonging to one of the consumer groups and inside this consumer group I'm having two consumers so here so here whatever messages that I'm pushing in so also the one consumer is enough to like to like get in all the messages because here we don't require the horizontal scaling here and suppose if I start off with another like another consumer group so it will get all the messages from here so if you see both of my consumers are getting all the messages here so this is the like the concept that we are going to implement inside the Drupal module and then we and then we'll be contributing that shortly yeah I'm not sure if you're familiar with the Firebase yeah they have an SDK that you can install on your front-end JavaScript by the way somehow monitor the database on the Firebase and so anytime it gets updated it somehow pushes real time into your JavaScript front-end yep did you guys could now see doing a similar thing here was extra this is a software do you guys consider using Firebase? yeah so okay so before we move on to Firebase okay so if I understand you're like what like is Firebase better or Kafka better so basically you're asking that question right yeah I'm just wondering if you consider yeah yeah yeah because it's kind of similar yeah yeah basically we have we have considered both the options we have gone with Kafka because Kafka uses a zero-copy policy and then it is having like like using like it is using serialized input and like input and output system so what it does is suppose when suppose when we are writing in something to our like to our application like suppose if in this this like it's a producer for example so in a non-zero-copy policy what it does is so once you are writing it to like suppose we have the Kafka application for example like in your case it's Firebase for example so from the Firebase it is getting written to the OS cache okay from the OS cache then it is getting written to the disk okay and then and then and then it is getting written to the disk so that is the right process okay suppose when we want to read that data for example okay from your system if you like suppose if there is a server or anything so all the data is getting emitted via the NICs okay that is the network in the interchange cards okay so what it does is so once we are reading the data from the disk so then it is written to the OS cache okay and then from the OS cache it will be written to the Kafka application and then from the Kafka application it will be written to the socket buffer and then after that it will be written to the NIC buffer okay so that is the number of hops it will make okay but in the case of Kafka what it does is during the read process the data is being read from the disk from the disk it is written to the OS cache and then what Kafka does is it is sending a kind of the send message or the send notification to the OS Then from the OS cache, it is directly sent to the NIC, so it is avoiding that particular hop. For that reason, it is quite a performant. And then the other thing that I can see is like in the modern NICs, like the modern systems that we're having in. So it basically implements DMA, that is Direct Memory Access. So in that case, what it does is it is basically, so this load process that is happening, it is basically eliminating the CPU processes. So everything is happening in memory. So once the CPU cycles are excluded from that particular process, so then it makes it even more efficient. For that reason, we have gone to Kafka. So these are the mechanics that is happening behind the scenes. But yes, this has to be also considered when we are building a very enterprise-grade application. Just to add to that, so this comparison actually happened when we were designing the architecture for the buttons. Just that now the projects we are doing, they have more implementation of Kafka already in place. It is more to streamline that whole architecture. That's why these buttons. We can more use cases with the other tools. We'd be happy to work on them. Like it's a progressive approach. So if you have your front-end application, you don't talk to Kafka directly. You talk to the API gateway. So is there SDK for the API gateway that you use in your front-end? Yeah. On the API gateway, that will basically act as my proxy endpoints. Suppose if I have my Drupal application, for example, that is my content source. If I directly, if there is a third-party consumer application, if I'm directly exposing my APIs to the Drupal application, and if I'm not keeping that behind a private subnet, so then it is more prone to attacks. So in that case, I'm offloading this API concerns to in this case, it is Kong API gateway. So here what it is doing is, so Kong API gateway is the only application that is sitting in front of multiple application. So whatever internet traffic that is coming in, it is coming into Kong. And then Kong is basically seeing whether this is a valid request or not. Then Kong is basically, because they are assigning it to the same DPC. And then it is connecting to that particular Drupal application through their private subnet. And then it is sending the information to Drupal, okay, Drupal, I need this particular piece of information. Then Drupal transmits that information. And then on the Kong layer, we have written a transformation layer. What this transformation layer does is, whatever the open API specifications that we have, that we have written, for example, if it's the user entity. So in the user entity, if we are going by the JSON API standards or the rest API standards, okay, it will basically emit as the details in a very default format that Drupal does out of the box. Okay. Now on the transformation layer, we are basically transforming these particular specifications into the required specifications required by the third party consumers. Okay. Because if you see the response that is coming in from Drupal, it is having some metadata which will give the end user the information, okay, this information is coming in from Drupal, right. So we are avoiding all that metadata via the transformation thing. And then it is being sent to the consumers. Any other questions?