 Yeah, I think it's 1252. Let's go ahead and start this session. So yeah, hello everyone and welcome to the DevCon conference and I'm Hemant representing Red Hat and yeah, welcome back from the short break sessions and right now we would be taking over the data gateways, the legacy data for services session and we have with us the huge of Giruru who would be taking us this session for around half an hour and we would be having a live Q&A after the session is getting completed. So yeah, welcome again and yeah. Hi, hello. Welcome to this session. My name is Subu Girero and today I'm going to talk about data gateways and how to bridge between legacy data and monolithic databases and micro service architecture. So I hope you enjoy the following minutes. Thank you very much for joining and let's get started. So we will be covering in a couple of topics in the agenda for this session. We will be talking about the architecture evolution and how applications has been have been been changing in the way that they are using their components and how they're handling the clouding nature of distributed systems. Also, we will be talking about microservices data and what was the problem that the API gateways solve when they start to rise in a few years ago as well as how to handle data under the microservices pardon and the architecture that it's focuses on microservice. As well, we will be talking again about the data gateways capabilities, how they're working and what is the idea behind having a data gateway as well as some of the types that are currently available. And finally, we will be talking about different open source data gateways, what options do we have available and we will be focusing on one specific one that is called TIFF and a project that is sponsored by Red Hat. So I introduce myself. My name is Hugo. I'm Mexican. It's currently covering a couple of topics in the agenda for this session. We will be talking about the architecture evolution and how applications have been changing in the way that they are using their components and how they're handling the clouding nature of distributed systems. Also, we will be talking about microservices data and what was the problem that the API gateway solved when they start to rise in a few years ago as well as how to handle data under the microservices pardon and the architecture that it's focuses on microservice. As well, we will be talking again about the data gateways capabilities, how they're working and what is the idea behind having a data gateway as well as some of the types that are currently available. And finally, we will be talking about different open source data gateways, what options do we have available and we will be focusing on one specific one that is called TIFF and it's a project that is sponsored by Red Hat. So I introduce myself. My name is Hugo. I'm Mexican. It's currently based in the Boston area in Massachusetts here in the United States. I'm currently working with Red Hat as an API and messaging or even driven specialist. I also have been an open source advocate since I first started working with the Javos Enterprise application server around 2004. So it has been a long journey of working with open source software. I'm also a history and travel enthusiast. So I really enjoy traveling as well as new food and new snacks or street food around the world. So here's my trade handler. If you want to follow the conversation, you are welcome to just follow or tweet or continue to post more information on these topics. So we were talking about what is the evolution that applications have been coming around since we start moving away from the traditional type of architecture of three layers, one single application, one single deployment or multiple deployment files but just running the same infrastructure. So first we start with the 12-factor applications. We start to decouple certain characteristics and part of the design of your application and moving away from high-coded configurations to externalize ways to do injection as well as keeping different source repositories to have different paths and ways to handle our applications. Obviously that led us to microservices architecture with all the benefits of having these cloud resources that looks like they're infinite and allows us to have several deployments of our applications in different locations just asking or requesting resources to a cloud provider or in a cloud feature. And finally, the next step on these kind of solutions is what is coming right now on the topic of serverless as well as function as a service, right? Serverless being the capability of having all these microservices or containers or functions being able to scale to zero and just being called, just being activated while being idle when there's an event or when there's a call or request that it's addressed to that specific service container or function and then it gets started, it do all the processing that is required and then scale back to zero. Also, there's, you know, this part of this change and then this focus and having also function as service as the synonymous of serverless. But that's a specific way of how to handle the code and the packaging and who's actually managing those kind of components. But obviously they get the benefit of having this option of scale to zero as part of the function as a service infrastructure. So this is how we have been moving away from the traditional one single block of code in one single deployment as an application. However, this has been only a trend or a change at the processing network layer because most of the times this kind of development, this kind of microservices and serverless and functions, most of them are very dynamic. That's the way they require this scalability, this portability. But sometimes we are actually getting away from the, from this taking nature of one of the components of the architecture that we cannot neglect. And that is data. So most of the times you perhaps have been facing this kind of problems when we're dealing with your application, right? So you have your this beautiful microservices architecture that it's pristine and it's ideal and it's ready to be deployed. However, you have behind you this monolithic database, this single source of data that it has not been refactored has not worked around to make it work with your microservices architecture. So you think you're already, there is, you think you have already saved your just a few moments of crossing the finish line. However, there's this big monster behind you that is just waiting for you to be distracted to, you know, take that stuff on your back. So I'm totally sure you have seen this kind of scenario before when you have microservices that are actually sharing the database or are being dragged back by the limitations on how the services are limited and coupled in the same moment when they're sharing data with other services. So what can we do? Well, we have seen that the traditional architecture has some challenges, also benefits when you have all these three layers all together, but they're actually accessing just one single data source or one single type of storage. You certainly have some consistency, right? All the logic is in the same language of the all the access to the database. Most of the times goes to one single storage. So it's something that it's going to be there forever when we take a decision around the storage. It's going to be, we're going to be using that story for a long time. So there's obviously the challenges of how frequently you're going to release. If you make change in one of the layers, you certainly will need to be sure that that change does not affect the rest of the architecture and how to make these resource isolations if you want to focus on problems in a specific component of your architecture. Well, those are the things that we are trying to get away from. So that's why we have this distributed architecture. And there's there's new challenges that we need to deal with when you were switching from the previous monolithic type of diplomas, traditional architecture to the distributed one. So the first has to do with microservices being a distributed system, a system that it's not running in one single place, but it needs to be deployed across different resources called it could be servers or it could be different clouds even. And that makes communication between those services critical and vital because we need to we need to face this. There's no service that lives in the vacuum and that can exist alone. So most of the times your services, your microservices will connect to communicate or have this kind of connectivity between them or with other services. So that's where that's why we still see some challenges on this kind of architecture, but now they're moving from these previous monolithic challenges to two kind of challenges, two new challenges. So the first ones are related to the network. We mentioned that we are having a distributed architecture that needs to communicate, right? It needs to connect. So that's the network challenge where you need to know where your services deploy. So you need to do discovery. You need to balance the load between the call to your services and what happens when suddenly one of your services is not available. And how do you share the information you can use? Just simple connectivity or you can use the call of services directly or you can use something like APIs to have their governance of your communication. And obviously monitoring and tracing to know what is happening in your communication that year. The second type of challenge that you get when you're talking about distributed architecture goes in the data layer. So we talk about when we're seeing these most behind you that the data access also becomes challenged when we are talking about microservices. Why? Because there's this focus on having these decoupled services that can be also managing independently. And the idea is obviously to have each one of those microservices enabled to just own one of the storage and be able to manage that, take property of that kind of storage. So you still need to be able to share that information, share that data between your services, right? If you have a customer that is doing orders and you want to have the items and the payments, you need to communicate with all these systems and share that information. So you will certainly need to have this kind of abstraction layer to be able to move that data in a simple way between all these different services. Also, you want to have the services available in different points of your system. You also want to handle them in a homogenous way where you can communicate them easily and everybody is able to understand them. So those are the kind of challenges that you can face where doing a distributed architecture like doing microservices. So when we were talking about the network challenge, one of the things that we came on the idea when we discovered that we need to have to handle this kind of network connectivity and way to connect your services is the race of API getaways, right? So suddenly the load balancer is just not enough for having a system that is a security device that it's in the perimeter of your application or your network that is just shielding you from outside. That's certainly not enough. You need to take more decisions on how to apply certain controls, certain governance, or the way you share information between your systems and how your applications suddenly start to consume those services. So this is how the getaway helps us to cover this type of access into this new layer. So the API getaway first gives us an abstraction layer where you don't need to know exactly what system needs to answer what request you can also apply in a single point of access certain policies. For example, access control, or rate limiting, or even adding some security layer that perhaps you don't actually need to do or implement in each one of your services. When you are taking your applications to access for the getaway, you are able to have this point to be able to enforce those policies, and then because you are actually now in the trusted zone of the network or your application, you are able to call those services and the getaway has the enough intelligence to know which service needs to solve or access one of those requests. So in the data in the data layer or in the data type of challenges, as we mentioned, when we are talking about microservice, we are talking about that we have services that require an independent database for each one of those services. Because this will help us to tackle that problem with the coupling. If we have a shared storage and we have services working on the sac-same storage of data, and we change one, obviously it's going to hit or it's going to pass some of your improvements to the other systems. So the idea behind how microservice should address data is one having their own database. And by owning this database, it could mean two different things. The first one is be literal and have two different instances of database totally independent each one from another. So we have an instance running one server for microservices A or having a second instance with the data and the access just for microservices B. The other option to have this kind of independent that is just having the sac-same instance but having this separation between schemas if the database is able to handle the schemas or grouping of tables. So a set of tables are owned by one microservice and the only way to communicate to that or to manage or access that data is through that service that it's owning that information and other service being able to have the information from the other perspective and just accessing their own table. So there's different ways to handle that independent how really you want them to be independent. But the focus is that just one microservice owns the data. You are not sharing that information at the start level. The second thing that the microservices tend to enforce or shape the way we handle data is the heterogeneity of the services. So most of the times when we start to build or develop our services in specific languages or different languages and different tooling also it represents that we certainly will end with a different persistent layer for each one of those services because actually the data is different. There's data that needs to be treated differently and also it needs to be stored in a different way. So it's over those times where we can put in one single type of store just to make it compatible with the rest of the data available. So in this case we certainly will need to have a type of information and data that is stored in a relational database where we need to have high consistent data or perhaps we will need to store that information in a document database type or you will be able to access that easily and you will be able to have a more flexible way to store that information or even sometimes you will need to do just a in-memory type of key value type of storage for your information because that is different. microservices allow us to have this kind of difference between how to handle data and that obviously represents a challenge when you need to deal with this polyglot persistence persistent failure. So as we can see we have different type of solutions but in kind of similar way addressing the challenges we mentioned before. So in one side we have the API gateway where we can have this implementation or the abstraction of the communication with services where we are focusing on the contract through API first design and development and this charge of the load balancing and the network resiliency on how to access our services and allows you to implement certain access control on how your applications are calling the microservices. So in a certain way we will have something similar when we are talking about data gateways right so we still want to have this abstraction layer on top of the implementation details under that store and also we want to have this type of federated approach where we can access and address data in in the same way independently in how it is stored right from the application perspective I just don't want to deal with the all these implementation details of how the database is being managed and how it's implemented. Also allows us to you know boost the performance when suddenly we need to scale allowing us to have this type of data gateways can help us to be able to implement some policies or some mechanisms like caching or materialized views to allow us to you know have more as services or more applications being able to access that that perhaps is being it's being kept on an infrastructure that perhaps is not able to scale saying we're targeting a very old database or perhaps a mainframe that cannot scale beyond the current infrastructure. So basically the data gateway is just like an API gateway but instead of you know focusing on how to access the network layer to just reach the microservice that you want to do and being able to talk with that services under the same circumstances it focuses on the data layer implementation and and it's been able to actually access the data implementation details and being able to connect and talk to the specific implementation resource and then being able to offer that information or that data in an homogenous way to any one of those consumers. So that's basically how data gateways as a resemble the API gateways. So what are these data gateways capabilities we're talking about? Well we mentioned a couple of them in in previous slide right we actually were looking for a piece of software that can do this abstraction layer that could help us do the coupling between the service that it's coming to data and how is the actual data start being implemented. So it hides these implementation details I just don't need to know if the data is coming from a document database or in memory database or even an API service I can just access it in the exact same way so it hides implementation and attracts the physical source of the information or also allows us to add this security layer where we can take some control on the access of the data through the modeling of the actual data. So instead of thinking about just accessing resources like we do in the API gateway we can focus then on implementing some policies around specifics on the data like for example row or table level so you are able to have a more fine-grained control of the database without having to implement all those policies all that information in the actual data. So you will have this layer or at the data gateway level where you can implement all these policies without being to go to the actual implementation. And obviously this with the mechanisms like catching and materials use you are able to scale your infrastructure to be able to handle more loads right instead of you know hitting a mainframe that perhaps it's limited in capacity you can just do it once then populate the cache and then it's being used to return all their responses or their responses that are being requested to do that data gateway. And with the usage of some other components like for example or patterns through tools like change data capture for example through the BISM you're able to well be able to invalidate the cache or just update the caching when suddenly there's an update on versus layer and you're able to continue to have these scalability features on your gateway. We talked about the federation and being able to access the data in a single way even thought the implementation details are different on the different sources. And one of the approach that is very useful for applications is to you know keep a very known language and how to how to query the information and that can be very useful to try to keep the schema first or the design of the contract through standards like SQL. So we have talked about the different capabilities of the data gateway and now let's go over some of these examples of what that is because there's not one single solution there are different shapes of data getaways going from for example the classic data visualization layer where you can find solutions like a composite or the nodal where they still have like the same centralized single deployment approach that we used to have in the ESB when we were talking about networking or SOA for example to do this kind of visualization where everything goes under the same infrastructure or you can have also the federation within the database where you can have these databases like portraits that can implement some kind of the genes that allows you to offer access to the details of the database through standardized connectors. There's also a rise on the use of GraphQL bridges to be able to access data sources in a simple unified way so it's pretty common to be found in mobile devices or front-end applications that need to do several movements or several changes in the data store at the same time and so using a bridge like GraphQL allows you to apply all these different types of changes on a single store. There's also things like cloud hosted data getaways like for example AWS Athena has some examples there's AWS Redshift where you can have a single access to a data store that can be backed by S3 buckets or other different type of components that are more proprietary to the cloud provider but still you can have a single point of access that it's been able to serve you through standards like SQL and others and there's other ways where you can rely more on the network level and have these kind of data proxies that are just doing a secure tunneling to access the backend services so you can use something like Google's cloud SQL proxy or even there's a project based on the QP dispatch router called SCOPR that allows you to access certain resources on your network, certain services like your Kubernetes cluster going through a proxy that can apply certain policies and being able to let you forward the connection to your to your action database but you're missing some of the capabilities that we talk about what we are expecting into that I would get away with in the past and finally there's some open source data getaways that are also available out there so Apache Drill is one of the projects that has this schema-free SQL querying gene for no SQL databases there's other solutions like rest of the project started by Facebook that allows you to have this gene being targeting mainly big data use cases but still something that that it's open source and it's available in the field and finally there's a project that it's sponsored by Red Hat as I mentioned at the beginning that's called TEED it's a project that has been around for a while but currently it's now being focused on being able to deploy this kind of getaways at the Kubernetes native level when talking about TEED so TEED has different components the new focus or the new report of this project is to be able to have based on the operator pattern the Kubernetes operator is main focus on on Kubernetes to be able to deploy these getaways as microservices that act through connectors to be able to you know connect and establish a communication with the that source they have different type of connectors to different type of of of that stores that can talk to relational databases non-relational databases API's or even object storage and then can connect or interact with policy policy control planes control planes where you can define things like the type of security that you want to apply the roll access control and then expose that information through different type of Facebook you can just use a or implement or start a JDBC endpoint or driver where you can connect with with the with the JDBC driver and being able to create information using traditional NC SQL or you can access that information through a REST endpoint or for other applications that perhaps require different type of access there's an endpoint that could be used through ODBC so you can have different ways to interact with the getaway in a way that most of the applications are ready to do and then apply the policies as part of the getaway pattern and then being able to access the the data store so this is a way where people like data engineers can design the access to their to their data to data pipelines or data lakes or data ponds where they've been able to share that information in a secure abstract and homogeneous way to all their applications. So just to finish this is a great comment from a colleague of us in Red Hat, Bill Guinea, he is having this first part of the of the article he wrote about data getaways so data has gravity also requires control and it's hard to scale if you are using a traditional approach obviously and it's it's sometimes difficult to move between cloud infrastructure so the getaway is one of the components that it makes even more clear that we need to have this kind of of companies to be able to deploy not just in a single way not just in one in data center or just in the cloud but is in is the requirement of this kind of components at the hybrid cloud level that actually it's becoming a necessity and the data getways is it's a way it's a pattern that you can use to be able to overcome the problems and the challenges of the data layer as we did with the IPA getways at the network level. So I really appreciate these moments with me I hope you like this session it has just a brief information how data getaways work what are we expecting about capabilities and there is different projects that we are you are following on how to implement this kind of of software components you have some examples there you want to take a look visit the tth project it's something that's still working and they're still updating on the different options you can see some samples in your data page in their website so I hope you like this session I really appreciate the opportunity to be with you and thank you very much see you bye yeah thanks a lot for lighting talk on the topic and yeah we are open for the Q&A sessions if anyone has any questions about the session or about the microservices they can just pop in in the chat box I think usually you had a pretty exciting session I don't see any of the questions lying in the chat box so yeah we'll be ending out this session right now and yeah we'll be