 So today we are going to talk about time and about digital technology and about how those two things correlate. But to keep expectation in check, this doesn't mean unfortunately that we are going to talk about the time travel or about the flux capacitor or back to the future. We are not quite there yet, although I hope that we get there in one of the next big things. What we are going to talk today about is real-time digital platforms. And talking again about the famous flux capacitor or back to the future, back to the future is only one of many movies about time travel. The last of them is Tenet, as far as I know, which I hope you had the chance to see before the confinement because it's really worth it. But the thing is that there are almost 400 movies and TV shows that talk about time travel, time bending, time slowing, time recovery. Why? Because time is an obsession for humanity. Time is the only asset in fact that money cannot buy. And maybe that is the reason why we live obsessed to save every second that we can get. Our life is in many ways a competition to save seconds. We compete to be the first in line, we get angry if somebody tries to skip our line. We chase the bus arriving at the bus stop, as is our life depending on catching it. It's true. And get hugely frustrated if we lose it. And we are willing to pay hundreds of euros in fines or even to risk our lives just to save a few minutes driving our cars. So we live in constant competition to save time. The delivery time is the most important value after cost of almost any product or service that we buy. That's correct. But the funny thing is that for some reason we have accepted, we have come to terms during years that these rules don't apply to the digital world for some reasons. During years both providers of digital services and we as users have gotten used to the fact that online services are slow, are asynchronous and there is no way to expect from an online service the same kind of response in terms of time that we get in real life. There is no way to expect real time. And when you think about it, in fact, the opposite should be true. We should demand from digital services more efficiency than in real life because if you take out the human interaction factor, everything should be more efficient. But as I say, the reality, in fact, is different or has been different during the last few years. For example, take online shopping. For years we have been used to waiting days for an order to arrive. This was the normal thing. And you had also complex payment procedures which you have to fill up a lot of forms, your name, billing address, billing information, everything which made the whole experience a whole slower than in person in almost every way. And when you go into a physical shop, you select your things, you carry them to the checkout, you will fight bravely for your place in the line, you will unleash your rage to anybody who tries to skip it, you will fight for seconds. So the patients that you don't show, you don't have to wait a few more minutes in line in physical shops turn to days in the digital world and we accepted that for some reason. So the same thing in online banking, we accept already today as a reality of life that bank transfers take days. Even bank clerks are able to tell you with a straight face how many days your wire transfer will take. A wire transfer at the end of the day, if you take out all the security checks that of course are needed, is if you think about it, just a digital entry in a database and it takes days. Why have we gotten used to this? And what to say about the other bank services like signing up for a new product or getting a duplicate of your credit card, here we are no longer talking about days, we are talking about weeks. But again, we have come to terms with this fact of life and we accept it cheaply. Or that's something that has been true almost until a couple of years or a few months ago. And all of that is when everything goes well because we have also gotten used to the fact that in big times when everybody is trying to get tickets for a concert that just opened or to get plane tickets with a special discount that you can get only at a particular time, it may well be impossible to get this and this was until now a reality accepted in the digital world. You could not be in a hurry in an online services. Online services were not dependable. If you really need something, you call by phone or by the jet, go in person to get it. And the problem here is that most companies have adopted this culture in their digital transformation and they have built their IT ecosystems and digital platforms around this level of expectations. This is what they think or they thought was good enough and this suddenly has changed. During the last months we have seen how suddenly this acceptance of a different time reality, so to speak, for the digital world has collapsed and suddenly all companies find themselves thrown into the middle of a race for time, a race for which many of them in fact are not prepared, they don't have the proper car. And this is the challenge that we are going to discuss today. We are going to share the experience of Paradigma, helping our clients to compete in this race. But before going into detail and sharing with you what we see and how they are doing, I think first it's important to understand the principles of what has changed, why this has changed, what has motivated this disruption of the user perception of digital services in terms of time. And I think there are two main reasons for it. The first one, the first reason as usual and this is something that happens again and again, is the arrival on the scene of new players that have simply changed the rules of the game. And I ensure that the first one that comes to everyone's mind right now is Amazon. Suddenly, when Amazon arrives, the whole shopping online process becomes a real-time experience. And digital does not only catch up with the in-person experience, but in many ways it surpasses it. Just picture the process when you are buying at Amazon. First, we enter Amazon website and we have, at the beginning, just for a start test, instant recommendations with a huge rate of satisfaction, a huge rate of success. Because Amazon makes every effort to know us in real-time, according to how we are feeling, what we have done today, better than our own families. From analyzing our behavior to our conversation with Alexa. So we no longer have to look for what we want because it comes to us. That's one point in fate of the digital experience against the physical experience. And then we have one click order. If you like something, you click and you buy. No time lost writing your address, your card, your bill of information. That's another score for the digital experience because pain in person takes in fact longer. And we finally have the delivery itself. The delivery is so fast and so efficient that as they like to say, by the time you take an Amazon delivery off your ports, walk into your home, open your box, you've already spent nearly as much time handling the packets as all Amazon employees together. This is efficiency. And this is also happiness. I quote before delivering happiness because for the user, this is a sense of pure joy, pure happiness. In fact, recently this showed up in El Mundo today that as most of our many of you know, is a satiric newspaper, very famous here in Spain. The news is of course made up, but it could very well be true. And it said the phrase, your package is on the way already surpasses. I love you in the list of the most appreciated by Spanish people. And also over other mythical phrases like this year, your tax result is a refund, which now was the top of the podium. It causes segregation of endorphins. I mean, this is also a joke, but in fact, I wouldn't be surprised if all this was true because the online shopping experience when it turns to real time produces an instant satisfaction in the users that drive them even to buy more. Who hasn't experienced this Black Friday frenzy in which you see instant deals going through your screen and you start clicking almost by instant thinking, I will return that later if I don't need it, but I have to catch it now. This is a whole new experience and psychologically is a whole different thing when you have almost real time or you have this experience that assimilates to real time. And Amazon knows it, of course. And when discussing the Amazon's access with retail companies some time ago here in Spain, there were those who were not worried at the time at the beginning when Amazon was selling books or small items, thinking that Amazon would stay there. They would do very well with books and small items, but they could never enter in the food distribution because of the logistic challenges of food, the cold chain and so on and so forth. But then Amazon launches right now, delivering food in an amazing under two hours window and that's simply the last nail in the coffin of traditional online shopping experience. It suddenly throws every retailer in the world in a race for which they are simply not prepared because they have built their own systems with a different level of user expectations in mind and that's the drama, their challenges right now. And it's not only an issue of retail or only Amazon. Another player, you all know it, who has changed the rules of time, is Netflix. They know from studies that they have just 90 seconds to convince the users they have something for them to watch. So what they do is they analyze constantly your behavior and they recommend content in real time. Even when you are still watching the previous show and it's about to end they are already deciding what to recommend you and to give you an idea of the importance that this has for them more than 80% of the TV shows that people watch on Netflix are discovered through the platform's recommendation system and it generates for them 1 billion revenue. So this is something that real time has brought into the table and it's not only recommendations what they are generating in real time the streaming quality of the transmissions is constantly adapted to your personal conditions and they are even able to predict it. So they are not only being able to react in real time and adapting for each and every one of their users. They have raised the bar even more by moving into the future which is really mind blowing. They are predicting what your quality will be and they are able to even anticipate to that reality that has not yet happened. So we are entering in the tenet territory so to speak and this is all in the digital world which until just a few months ago was supposed to be a lot worse in terms of time than the real or in person world. More example there are in fact quite a lot of players that have changed the rules to put a last example where we can mention Uber or Cavify that are able to adapt in real time their prices according to the demand and again you can easily see the importance that this has for them. But I mean there are as I say several significant players that have changed the rules but I mentioned before that there were two main factors in this launch of the digital race for time. The first is this the arrival of these new players and the second is the arrival of something which actually has been and is being much worse which is COVID-19 in just a matter of weeks. We find ourselves suddenly confined in our homes and unprepared for it. Suddenly we are scared to get out of the house to run daily runs and our life is upside down. We can no longer do in person critical necessities for our lives or we prefer not to do that in person because we are risking a lot and we immediately turn to digital services instead. I mean they have been there our whole life and now we turn to them but with a critical difference. We turn to them with a different look. This is no longer a leisure thing, a trivial activity to make the most of a rainy day or a black Friday frenzy that nothing happens if it fails because you simply lost one buy or you lost one ticket. Suddenly we are talking about essential needs that we cannot address in person and we cannot longer afford to wait. What happens? Mostly failure. E-commerce collapses and it is no longer funny for the user. It impacts our life squarely. We find ourselves, and I don't know if it has been your experience, it has been mine, setting our alarm clocks for three or four in the morning in the hope that we can get an order through the system and avoid the competition of other users. As our frustration as users grows, our expectations have been changed forever because there are those who come up to the challenge and answer to those expectations and once you go to that road, there is no coming back. In fact, the result is that we no longer accept waiting and this has been a very quick change in a matter of months or at most one year or something like that. Years of acceptance of slow digital processes turn into complaints and into rejection. Take, for example, the US aggression. This is, I don't know, a historic process, a centenary process for years, decades. Votes have been counted during days and they have a fixed period of time for everything and there were no complaints. We have listened to complaints, but this year the whole world is wondering how a technological power such as the US can't have results in the same night of the election like we do here in Spain, for example, but the thing is that our expectations are completely different. We can't go back from this. The new expectations of the user have been set and it is an inexorable process that has thrown everybody in the race for time. So what to do about this? We, for starters, can take a look of how these new players are doing it. Some of them are digital natives. For example, Google has been doing this for years. I always tell the story through a story of how 15 years ago when I was trying or maybe I was trying to sell some application to a client, they would say that's all great, very good, but I want it like Google, that is able to autocomplete what you are trying to write in real time. It never goes slow and it never fails and it has the whole internet data indexed. So I guess that you will be able to do the same with your small application, won't you? And you know I had to change the subject back then because back then it was magic. We didn't know how they were doing it. And they have had a 15 years at the start and we cannot go back in time. So that is a dead end for everybody else. By the way, I was not so wrong back then because what Google did and what all these new players keep doing is a kind of magic trick and illusion, but a very bright, a very ingenious one because in distributed computing, as you know, there is no such thing as real time. But as users, we have a perception of real time. Behind the scenes, what we have are very well thought architectural patterns and design that provoke this illusion. For example, we were talking before about Amazon and there are one click orders. You click and you get your thank you, your order is on the way message and it gives you this end or fin kick. But the reality is that you have not the paid and the order has not been yet processed. You will pay eventually and if your card doesn't go through the system, then your order will not be processed. That's why you get sometimes the message that this order has been cancelled or whatever. But the illusion of real time means everything for the user and still near real time, but that should be your goal is a huge challenge. So one way to go, digital natives, but that's behind the time. Another guy going back 15 years, impossible. But not all these players were digital natives. Take Amazon, for example, again. They took a different path. Not so long ago, Amazon was a quite traditional company from an IT point of view, but then one day the business growth communication to all the company, which today is known as the API mandate, telling everybody in the technical area to start working in a new radical way. And he finished his communication saying, anyone who doesn't do this will be fired. Thank you. Have a nice day. This is another approach to change, but it is brutal, so it's not up to everybody, but you can't discuss the results. Amazon has today a full real-time AI power digital platform, which allows for every other automatic selection of warehouse, recognition of the effects, 15 seconds to assemble packages, and allows a click to chip time of something like 15 minutes, which is really amazing. So that's another way to go. But at the end of the day, the thing is that for 99% of the remaining companies, the solution is not as straightforward. They face a huge challenge. They are at the very beginning of a very long cloud technology adoption road, and the road ahead is hard. So what to do, some general tips, and then we will go into detail. First, we'll tell you what you shouldn't do. You shouldn't take an all-or-nothing approach building a new platform from scratch and trying to jump on it on the way. Senses are that you will fail. So as the saying goes, please don't try this at home. So that would be the first tip. But you also can jump to the other stream and settle for cosmetics just for some digital washing, rewriting a few front-end applications and hoping that the user won't notice that underneath you have the same back office. You will fail with this approach also. So you have to evolve and you have to assume that during a long time, possibly years, the old IT ecosystem will have to live together with your new real-time digital platform. You have to prepare for this scenario, which is not at all an easy one. And at Paradigma, we have been helping our clients in this process for quite some time now. We want to share with you our experience. So Ruben, our technical director in Paradigma Barcelona, is going to tell you, to tell all of us what we usually find in our clients that we usually are able to help them. So, over to you, Ruben. Thanks, Jose. I'm going to try to share my screen. Okay, so I suppose that... Yeah, yeah, we're watching your presentation. Okay, perfect. So, as Jose has said, before helping customers, we first need to have a look of how things are standing. So we are going to have a look of how things are standing and how did we get there. So, at the beginning, things for all systems are easy. So you start with one big monolith, maybe a mainframe, and all applications and processes are built against this system. Here, what is written into this single database is immediately available to read. So you put things in, you get things sound. Basically, this is real time. So, at this stage, users are happy. They're happy to have real time, and things are really simple. But as everything in time, as everything in life gets... things always get complicated. So, normally, you never can do with only one system. You need to grow. So you need to add more systems into the picture, maybe a CRM, and a specialized ERP, a data warehouse. Even maybe your company needs to integrate new services. So, basically, the more systems you integrate, the more complicated things get. For every system you need to integrate. You need to develop integrations for all the remaining systems. Basically, this means that the growth of integrations is exponential. So, one try-and-tested way to integrate these systems is batch processes. So, these batch processes basically are a process that runs at a scheduled time, normally at night, and you update systems, you move data from one system to the other. But, at that moment, something happens, and users begin to get data that is not real-time anymore. The data is outdated. So, processes are not real-time. Basically, what the user is reading from the system, what the user is getting out from the system, is yesterday's data. It's not data from today. It's not fresh data. As Jose said before, a very common example are bank transfers. It takes one or more days, at least in Spain. In Spain, it's like that. I used to live in the Netherlands, and it was the same day. But, in Spain, we need to wait a little bit more. To perform a bank transfer, you need to wait some days. One or two days. Basically, this is a limitation generated by the technology. This is not something that should be like that because there's some inherent limitation. It's just the technology we are using. The funny thing here is that even the customer support of banks is aware of this scheduled process. You are worried about a bank transfer that is not getting there. They know what time the process is going to run. Even at some banks, they know the strange name of this process. It's something that we've got used to. It's a little bit frustrating, but it's the way it is. A good measure of the prevalence of technology, in general, is the market of tool around this technology. If you look, for instance, if you do a Google search for ETL, you'll see that there's a whole lot of tools that are related to these ecosystems. Basically, all the big database builders and even cloud providers have some kind of tool related to ETL. IBM, Oracle, Amazon Azure, even the popular development framework, Spring, has this Spring batch library. With this, you can see that the prevalence of this kind of tools at company is really, really big. Also, one important thing is that most companies, the proportion of online versus batch processes, these processes that are hidden, that run in batch, that run normally at night, are not visible. Most of the logic of companies that a lot of time is integration between these systems is executed as batch processes that are scheduled to happen at a certain time of the day, normally at night. As an example, we had a customer who had more than 14k daily schedule processes to manage the integration. There is 14,000, this is a lot of processes. This not only affects the obsolescence of data, so this not only means that you are not going to get fresh data, but also this also has a lot of problems related to data governance. This data is really difficult to manage. All these processes, this 14,000 is just almost impossible to know what they are doing. Even with all these problems, all these delays and this problem with this governance of these processes, all these data just going in and out, companies have felt comfortable for years working with this kind of technology because basically if it's working, don't touch it. So the problem now, as Jose said, is that suddenly they need to fight for survival in this race of time and they are in a tough spot. Basically these big tech companies like Amazon, like Uber, they have raised the bar really, really high. But they are good news. The first important thing here is that technology is here, is ready and is readily available. So we don't need to worry about how to do this or about this black magic. So even though real-time platforms are something new and are a big part in change compared to the more static world that we are used to work with, it can be a little bit scary. Again, if you look at the market as an indicator, as a thermometer, we can see that the main tech players have introduced into some type of real-time platform. And even better news is that a lot of these platforms are self-managed. That is a fundamental step for the adoption of these type of platforms, because these platforms are really complex to manage. Unless you have these self-managed platforms, the adoption is going to be really, really tough. So in the first place, we have Confluent Cloud offering a self-managed Kafka platform. Also the main cloud providers have their own even streaming platform available as a service. Kinesis from Amazon, PoopSoup from Google Cloud, even Haps from Azure. We also have streaming frameworks that allow to change data in real-time, like Kafka Streams, Pulsar, Spark Streaming. And another really important piece in this puzzle is this open-source ecosystem of change data capture. We'll talk later about this, what change data capture is. But basically what change data capture allows or change data capture or CDC allows to connect your database system to an event platform. So you have these CDC connectors from Red Hat, it's a project called the Vessium. And this is a really, really amazing project that is a really is a facilitator of this kind of technologies. So I've only shown you a small sample of all the existing platforms and tools related to real-time platforms. But in general this is all these technologies are a really good indicator of where things are going in the next years regarding real-time. Just a quick note for the sake of this tag when I talk about real-time platforms I'm talking about systems that integrate of data in real-time. I also will refer during this presentation as streaming platforms or even platforms. It's the same. Another really important piece is the availability of high quality training resources related to the fundamentals of the design and development of these type of platforms. As I said before event-driven systems, real-time systems are a complete change of partying from the more static world of databases and batch processes. So not so long ago they were a kind of lack magic even scary. You have all these events moving around in real-time you need to handle errors you need to make sure that all message arrived to the target system. You need to care about scalability, replication, so it's a lot of things to take into account. So today we have a lot of literature based on real use cases and best practices on how to design this kind of architectures. Also these resources are really high quality and in a lot of cases are even free. The three resources that these three already books are free you can just search Google and you can download them for free. So it's really amazing that all these resources are available for free. So this aspect is for me is key for the wide adoption of event-driven architectures. Technology is necessary, but it's not sufficient. So things like Apache Kafka Confluent Cloud, Kinesis are not enough. You also need to know how to use them and you need to have all these training all these materials available. So we are going to have a really quick look. So this is even streaming platforms are really complex, but I'm going to simplify them a lot so that you can all get a picture of what they look like. So I'm going to focus on Apache Kafka as an example because it's the more material technology for implementing these kind of architectures but all the concepts you can apply the same for Kinesis, PubSub, etc. So the main concept you have in an event-driven real-time architecture is Event Log. Event Log, if you look at the picture is this kind of this kind of queue. So you have like this kind of this time picture. So you have on top of the time arrow, you have this kind of queue. And basically what Event Log is is like the text log that you have in every application but with structure information. So in the log that you have in application you have something like a date and then you have some kind of text message. This kind of log is a little bit more complex and you have some kind of structure data like an order, a user, etc. Basically something that comes from a database. A log is a really simple data structure and the thing is that it's really, really fast because the only thing you can do with it is to append data. You cannot delete, you cannot update data, the only thing that is that you can only add data at the end of it. And this makes this kind of structure really, really fast. For instance LinkedIn is managing more than 7 trillion messages per day with this catagap platform. It's 7 trillion. So this is a lot of messages. Okay, we know what a log is, so let's go with events. Events is basically something that has happened, like an order that has been created a user register, a movie watch it, etc. Something that has happened. And if you look at the middle basically, so you have the event log, you have events inside this event log but you need to get events inside this event log and then you also need to be able to get the events out. So for this, you have producers producers just write events in the log as they happen in real time and they can be applications, for instance application writing user events related to the activity of a user in a website database events, for instance insert, updates, etc. You can also have sensor events, for instance a GPS and in car positions, etc. So this is producers, this is just things, it's just putting events on this event log and then finally you need to get the events out. So basically you can have here, you can have some kind of processor as an example you could have a consumer getting things out of this event log a process checking great car fraud from payment events this could be an example of a consumer. So basically this is really, really the basics but this is the how an event driven architecture looks like. I'm going to just show you a little example of this. This is how an event is driven architecture looks in the with real, as a real example a really simple example with an order system so at the left you have a database of orders and line items basically orders, in the middle you have Apache Kafka and you have these events in this case events are orders so you have for the one or the two, etc. and on the right side you can have databases, you can have different systems, you can have a Splunk dashboard real-time dashboard, elastic search flake, etc. What you have this is that everything that gets inserted into the database in real-time will be inserted in all these systems and you will be able to search in real-time to see a dashboard in real-time to put your data in another system in real-time etc. So basically this is an example of the usage of an event driven system as I told before for creating events from database changes we could just update all of our applications but this could be really overkill, you need to update all the applications in our company so that when something happens in our database they just write an event into our event log but this would be overkill. So for this for solving this problem you have a technology called ChangeAttackApture that automatically creates events for every database event. So for instance you can see that if you do an insert you will have an insert event updated, inserted into the event log. The same for updates the same for delete. So this is a technology that enables the adoption of these kind of platforms because it makes things it makes life much easier. CLC is basically existing for almost all known databases and there are commercial products for all databases like DB2, Oracle, etc. You also have this Red Hat project called the Vessium for implementing the CDC for a lot of open source databases like Postgres, MySQL, etc. Basically what CDC does is enable to free our data from our systems and integrate them in real time with new ones. This is a really simple concept but it makes the life of people trying to implement this kind of systems much easier. Finally I will show the strategy how can you implement a strategy from legacy to real time. How can this happen? Basically this is a process that takes many years and must be implemented carefully and in different stages. The first phase of this process is just replicating data to another systems. Basically normally this happens to a system in the cloud to perform some of the queries that used to live in the legacy system. One standard way to start data from our legacy system is as we talk changed at a capture that allows the faster implementation. So this is the first phase is just getting some data and make this data available to query. The second stage would be by directional replication of data between legacy and new systems in the cloud. So meaning that the new system in the cloud is not only it's not only with only but it's also accepting data. So this would be just all systems and new systems living together. And the final stage basically is just getting rid of the legacy systems and having all the systems living in the cloud. Normally these systems would be developed using real time platforms using with architectures communicating through events. And these new systems also allow the implementation of some of the functionalities that we that Jose has been showing like in newware Netflix etc. That require the use of our real time platform like real time prices, real time recommendations etc. Jose If we if we have to summarize keep this message the world real time and it is time it is high time in fact that your digital platform follows. So at Paradima we will be more and unhappy to help you because that is what we do best and nothing else thank you very much for your time and feel free to contact me or Ruben if you have any questions. Thank you so much for this for this interesting presentation we have only one question trying to point in a question that we have already talked today that is biases, biases in business decisions so how these platforms, how these real time platforms can manage with these bias with these problems that can have an impact in daily basics in a business I don't know which one of you prefer take the question Ruben you want to take it Yeah Basically what we have taken into account that real time platforms are there just to transmit events they are like a pipeline so basically what they are doing is you are connecting a pipeline to all your systems basically you can use these systems among other things you can use for just integrating with new systems with you can also create dashboards services etc and of course one of the systems that you can integrate these platforms with artificial intelligence anyway these real time platforms basically they are let's say neutral so basically what they do is the data you are putting in one systems I'm going to put it in another one they are not the pieces that are analyzing data so basically we could say that they are just like a transport mechanism to just make data available to artificial intelligence systems but basically they are not the ones that should be careful with data biases Yeah guys it's only a tube it's only the definition so thanks both for this presentation we learned a lot about real time data platforms so thank you thank you