 Again, to another OpenShift Commons briefing, I'm Diane Neuler, and as we like to do on Fridays, we're going to talk to some of the thought leaders in this space. We normally talk about transformation, DevOps, all kinds of DevOps, DevSecOps, other topics on Fridays. But today, we have a friend and colleague who has got a book out the door recently at O'Reilly, Flow Architectures, major feet getting a book written and done and published. James Urquat is here from VMware, and he's going to teach us, talk to us about flow architectures in the future of streaming and event-driven integration. And then we're going to have some live Q&A and a conversation with some other folks from the Global Transformation Office along with James. So, type your questions in the chat, and we'll let James introduce himself a little bit more deeply and take it away. Well, thanks. It's a pleasure to be here, and it's a pleasure to see already a number of old friends on the call. This is great. And so my name is James Urquat. I've got about a 30-year career in distributed systems, development, operations, you know, team building, a number of those other topics, and a variety of different roles ranging from engineering early in my career to, you know, a lot of things around fields for various vendors that have been in the space, including Forte Software, Cisco, Casatsos, and now VMware from Pivotal. And then also, I've been, you know, a general manager at AWS, and I've built teams at companies like SOSTA and Stratus that have all been sort of either in distributed systems application development space or in sort of the supporting infrastructure space the early days of sort of data center automation and cloud computing included. And then many of you might know me from a blog that I wrote that was on CNET and then later GigaOM called the Wisdom of Clouds. And so for a good six, seven years or so, I had a lot of fun writing about and putting my observations out about the nascent at the time cloud computing industry and what it meant to the ways that we're changing the work that we do. So as is my way in the last two years or three years, I started sort of saying, well, cloud computing is reaching its maturity. What am I going to do here? And you know, what are the, what's the next big thing? And of course, there's a million next big things. There's not just one, but the one that struck me was event driven architectures because that flow of information and removing barriers to real time use of information in both customer experiences, data analysis and so on was really interesting to me. So that's where this book came from. I did some research and I had an observation and I'm going to step you through that in the course of the presentation and give you an idea from this sort of very high level overview of what the book's about. And I encourage you, if it seems interesting to you, if you'd like to know more of the details or how I came to certain conclusions, welcome you to pick up a copy. And I hope that you find it as useful as I meant it to be when I wrote it. So what are we going to talk about today? We're going to talk about the future. In reality, we're going to talk about something a little bit less than flying cars and all that kind of stuff. We are going to talk about the ways that organizations are looking to combine their real time information with that of others to take action. And so kind of the tagline of the flow architecture book, and we'll define flow here in a second, but the tagline of flow architecture book is basically like HTTP length the world's information, the worldwide flow is going to link the world's activity. And it already sort of is, but as you'll see from the flow definition, some things will come into fruition that will fundamentally change the math and make it much, much cheaper and therefore much easier to experiment and try new things and make things available to see if anybody's interested and so on, which will just create an explosion of available streams and uses of streaming. So what is flow? Every time I ask that question, this is what pops in my head. So I had to share it with you guys. But in reality, the definition is pretty straightforward. What I'm talking about when I talk about flow is event driven integration across organization boundaries. So not just within a single application, but across different organizations within a large corporation or even different corporate companies. And ideally in the long term, the real valuable thing will be when different companies can share real time data streams with each other through standard interfaces and protocols. And that's really the thing that's not there today. The thing that's not there today is that there's something like HTTP that's super simple, that everybody's code is written to consume, that everybody very quickly recognizes what a URI is for it, what a link is for it and knows how to take advantage of that, knows how to put stuff out there into a stream and so on. And so what we're going to do is talk through a little bit more in detail about what this simple definition does in terms of changing the game a little bit. I'll give you a little more detail, because that definition is really good at a high level to give kind of core things. But there are some specific traits to flow that are what makes it sort of uniquely different from other forms of API integration. And that's probably the number one question I get is, how is this different from APIs? And hopefully this will give you an idea about the specificity of that. So first, the idea is that consumers have control or in charge of making requests to connect to streams, right? So they can, the self-service interface can say, hey, I'd like to start receiving the events from StreamX. So they receive these weather events from the National Weather Service. And maybe even put some more information in there about specifically what they want to receive. Producers then get to choose who they allow to see those streams and which requests to accept or reject. Now this, I do say, or there are agents in here frequently. Just real quick note about that. In reality, it's probably not the actual data producer or the actual data consumer that will control that logical connection between the environments. There's very likely to be infrastructure involved in this. And there are a number of companies doing really interesting things in that space in terms of sort of decoupling you from the mechanics of establishing one of these connections that we're talking about. So keep that in mind when I say producers, it could be that a producer is using a technology that does this for them or a service that does this for them. Consumers or their agents don't have to actively request data. And this is really, this is what streaming's about, right? It's the push data. So when an event occurs, an event is sent and the consumer receives it and then can decide what they want to do with it at that point in time. It may get pushed to a queue on their behalf so that they can retrieve it when they're ready to retrieve it. All those kinds of things certainly come into play. But the logical idea is that consumers, once they subscribe, they don't have to actively request data. The data is coming to them in a mechanism that is pushed to them. And then finally, producers maintain control, or not finally, producers maintain control of what events are transmitted to whom and when. So they are basically as the party that is pushing data, they decide what data they want to push in a given stream and when they want to push that data in a given stream. And then the last, they are transmitted over the network using standard protocols. So this includes things like TCP, IP and TLS and all that kind of stuff. But it also includes to be determined protocols specifically designed for flow mechanics. And that includes not only the protocol for how we contain the package events and the metadata related to events, which we'll talk about a little bit more significantly here, but also things like flow control and requesting maybe a series of events from a certain time range. Those kinds of things will probably be also defined specifically for flow over time. So for many of you might go, okay, so it sounds like API's a little bit, but it's not just API's, I get that, but now it sounds like PubSub. And my answer is sure. That's probably the dominant model here. We are probably talking about most often concepts around publish and subscribe being applied into the way that streams are consumed across organizational boundaries. But in reality, if you look at that definition, you could do individual connections to a dedicated URI of some sort that says, hey, just start sending me events over this connection that doesn't really have the full semantics of publish and subscribe. There's not really topics. There's not really a subscription ID or a stream ID of some sort that is used to figure out who gets what, but that there's more of a direct decision to make connections. But just keep that in mind that I believe that, and I think everybody who's been looking at this believes absolutely publish and subscribe will be the dominant model, but it doesn't 100% have to be. And so we can't really predict that it's definitely 100% gonna be published and subscribed. So what do we enable with flow? So one thing is when we have standard interfaces and protocols, we're gonna lower the cost of integration, of real-time data integration by at least an order of magnitude. And I would argue there's a good chance that we lower it by orders of magnitude. We're gonna make the ability of say a, let's say a service company to understand what has been purchased from a retailer that they can service under some kind of service subscription, like maybe you have a subscription with an appliance repair company that will understand that you now have a new washer and then once that washer is connected, they can track the health of that washer through the standard interface. It's very cheap to figure this out, to experiment with ways of connecting things. And I believe that that experimentation, and we'll talk more about that in a second here, but I believe that that lower cost of integration drives a few things that are really valuable. You also get increased flexibility of solutions. So now, because the decoupled nature of this sort of publish and subscribe model or this flow connection model, it's much easier to compose solutions. If we have standard protocols where you can say, all right, I understand what's contained in the event and I understand what I need to know to interpret the payload of the event, you can begin to link things together in unexpected ways, which is the parallel I get to that is the Linux command line and piping. It's very similar, you know, because you know ASCII is being sent as output from most of these things to be consumed as input in the next thing, you can use piping to link things together in really, really interesting ways. And in fact, this, you know, the entire industry is the operation side of it is largely built on scripts that use piping fairly regularly. But, you know, you can see this kind of manifesting in terms of the interesting new ways of composing things like if I have a business process to what we've typically done up till now is we have outsourced entire business processes. So, you know, my payroll process, I outsource the entire thing to ADP. But if I have a unique situation, it might be really nice to outsource specific steps of the process to an external party. And then for me to be able to manage and maintain the process itself in a little bit more detail, either on my own infrastructure or at a core vendor that then has an ecosystem around it that can fulfill different needs. And so that because it becomes a much more proposal, it's much more easy for a startup to create something that can be consumed in those ecosystems because they know that if they implement a certain interface and protocol that it's likely to work. And the last thing is that ecosystem story, right? So as we get more and more technologies that stream using flow interfaces, what we're gonna get is more and more technologies that can consume using those, the flows using those interfaces. And as there are more and more customers consuming, then there would be more opportunity for vendors to say, hey, there's a bigger market here and so on. And that flywheel effect really comes into play as we connect the world's activity. The expectation will be if you want to understand what's happening in real time from vendor A, then use flow to subscribe to their streams and it will grow that vendor's ecosystem that much more. And the overall ecosystem. Now, why would anybody even want this? Well, the first thing is, I looked at a bunch of surveys when I wrote the book and there's the data's in the book if you're interested in the specifics, but really comes down to three areas. The one I thought would be number one on the list was customer experience, right? Is making sure that you're creating a situation where if a customer interacts with two things that they think ought to know about each other, they think ought to share information with each other that it's really easy to do that and anticipate that and be in front of it. And in fact, there's one poll that I found that said 45% of customers, V2B customers that were interviewed said that they might choose to move to a different vendor if their current vendor fails to anticipate their needs. And so that ability to sort of say, hey, we can have this data in real time and be able to put the pieces together is really important. But what turned out to be the highest thing in the poll in terms of why people would want to basically, do digital transformation and build a more real-time experience is employee productivity, which is not super surprising, I guess, when you think about it, but what they're really thinking about is, as data is moved and as they're interacting within their own organization, they're thinking about how real-time movement of data, real-time digitization of their activities will reduce both lead time and processing time for their various process steps and basically remove constraints in their flows in their system, which makes sense, but again, what I would argue is, there's a potential market out there that's phenomenal in which some of those activities which are just not core to the business that you have to keep doing because you wanna own these flows and the only way to get that activity is for you to have an employee that does it or a computer application that does it. If you could basically say, hey, I wanna consume that, that sort of very, very standard commodity activity as a service, you're very likely to say, okay, that's great, if I can do it in a timely fashion, I would love to do that and not have to carry that labor on my books or carry that application capital on my books. And so doing it across organization boundaries is what I argued that flow will enable that will change in a big way, change the game in terms of how people put together the way they process data and move data and move activity throughout their system and the various ecosystems they decide to consume. And then the last one was innovation and if you lower the cost of integration significantly, you enable a significant chunk of experimentation. And so that's a critical piece of this is just imagine that you could just say, hey, I'm gonna play around with pulling this weather data in from the government and pulling this available trucking capacity information in from say an Uber cargo service or something along those lines. Those things are, you could say, hey, is there something I can add a value here or I got an idea for something of added value and it's like cloud computing has done in terms of getting infrastructure to try to do applications at scale, it's very cheap to just run an experiment. And so I believe that's gonna create kind of a can be an explosion of the use of streaming and the use of flow across the business environment as well. And it's really valuable to businesses to be able to innovate new capability, new value into the market as the market adapts and changes around it. Okay, so all right, this is all cool, this is all the value, but how does it work? How is this all gonna come together? Before I get into that too deeply, what I'd like to do is just kind of point out one thing which is when you talk about flow and you talk about the movement of data, just flow without interaction with that data is just moving data. It's just moving data around the internet for no good reason. So as we evaluate what flow is about and what the mechanisms are or at least what the components are that would come together, we've got to consider not only what moves events across the internet, but we also have to consider what's on either end, what's on the producer side, what's on the consumer side that enabled you to take that information and make it valuable in a new way. And so what I did in the book and I stepped through this piece by piece. So if you're curious as to how I came to the conclusions I came to and what the different aspects are is I started to do some things around a technique called wordly mapping. And hopefully many of you were aware of what wordly mapping is today. I chose not to define it in any deep way. Really, as I go through this, you'll see sort of what the final map looks like when I get to the end, but the first step is always to identify a need, which in this case is flow integration. The users for that need, that you know, this is a user need. So who are the users the need applies to, producers and consumers in this case. And then I said, what are the components that are needed to meet that user need? And after many, many iterations landed with this value chain that I think pretty succinctly identifies the core elements of what will make up the flow value chain in the long term. But how do I know it's right? Did I, you know, is it just a guess? Now, actually what I did was I used a little bit of something called, if it comes up here. Oh, there we go. Used a little bit of something called promise theory, which is another modeling technique. This one from Mark Bergus, wordly mappings from Simon Wardley, that allows you to say, hey, you know, what is the intended action, intended relationship between a promissor and a promissor? So if you look at the notation there, the arrow points from somebody making a promise to somebody receiving that promise and utilizing that promise. So this is just a sample of some of the promise theory work that I did to kind of say, what are the relationships between all those things in the value chain? And use that to value, in some cases, I had arrows in earlier iterations where I couldn't really come up with a promise that wasn't either a promise on behalf of another component which is not allowed in promise theory or which is itself was a duplicate promise or just not a very logical promise. So it really helped inform the simplicity of the map to really be very tough on myself in asking the question, what are the promises in the value chain? And it made a huge difference in terms of the quality of the value chain that I ended up with. And then the other part of wordly mapping then is, he has, you can go look it up on the internet if you don't know a lot about it, he talks a lot about the natural progression of a given technology as it becomes more defined and more certain how it also drives more consumption and more ubiquity and commonness in the environment. And he identified four main stages that any technology goes through in that journey from being a kind of unknown novel, we don't really know what it is, kind of a behavior that's interesting or a combination of something that's interesting to being an all out commodity, everybody knows exactly what they're getting and has expectations about the exact behavior of what they're consuming. And that's Genesis, which is that new thing, custom built components which come as we begin to understand the general shape of things, people build their own. Ultimately, a few ways of doing this work better than others. So people productize those and sell them or rent for physical things and they might rent instead of selling as a product. And then eventually though, we say, okay, there's a very specific way of doing this is it's sort of very standard way of consuming these things. And it becomes very easy for somebody to either offer it as a utility of some sort or to sell, basically mass produce this thing and sell it as a commodity. So those four steps then we can map our value chain into those four core components. And again, after many iterations and much feedback from some wonderful folks in the industry, Derrick Collison of one of the creators of Nats, I.O. was one of them, Paul Butterworth at Bantic who was also behind Amber Point and Forte Software and Simon Crosby, who now is leading the way with Swim.ai but previously is famous for the names escaping me but for the virtualization work that he'd done in the past. And so really good feedback from really good people in the industry. And we ended up with this and it's a reasonable way of looking at if this is kind of the current state of that value chain in the market today. So the users, where the users are positioned doesn't matter as much. Flow integration is really just starting to become something that's productized. I think Kafka, I think some of the Apache stuff are beginning to be seen as ways that you can integrate across organization boundaries a little bit more, but they're still not standard interfaces in the sense that there's one way of doing things or a very small number way of doing things. They have their way of doing it for their product sets or their technology. And I'm not gonna go through every single one but you can kind of get an idea like all this stuff around processing and queues and sinks, data sinks and visualization sinks and all that stuff, that's all cloud computing today. There's a million myriad of ways of interacting with flows already out there today. And it's just a question of taking that logical connection from being a little bit more of a productized thing and the interfaces will be a little bit more product specific things and beginning to commoditize those and standardize those over time. So note on the protocol. The protocol is really divided into two pieces. There's metadata which describes information about the event that you might need to know in terms of understanding how to process it not only what's the format of the payload, et cetera but also what time did the event occur, what type of event category does it come from? Maybe it's source, topic, those kinds of things. So, and then there's the actual payload which this is the one area that will standardize probably less so in terms of everybody in the internet will know how to read any payload but will more likely standardize in terms of, standardize in terms of individual industries or even individual customer ecosystems or whatever. So, and it's already happened, right? If you look at the way we do batch processing there are already payload standards for batch processing. There are already standards for payload processing of things like electronic data interchange, EGI. So, I expect that many of the payloads that formats that already exist for other uses will probably transfer very well into the event processing world but that will evolve. I think as the metadata usage becomes better understood how industries use it will evolve as those industries begin to consume it. So, and as I mentioned that today's technologies fall right into place. There's a lot of analysis I do in one chapter of the book where I kind of step through a number of these technologies that you see here and kind of talk through what they are and how they worked with an eye not to sort of teach about the technology but with an eye to identify how does it fit in the map? Does it fit in the map? Is the map in some way able to represent how these technologies that appear to be predecessors to flow or parts of the flow architecture overall that they actually would fit into the map in the way things they use. And so, from that perspective I, you know so it's really interesting if you're interested in sort of how the world today comes together. I give some really good solid examples of this and then there's an entire appendix in the back of the book which is actually about a quarter of the book that goes through each of the components in the value chain and gives a pretty thorough survey of what the technologies are in that component space above and beyond what I discussed in the chapter as well. But none of these are flow, right? They fit into the value chain but none of them have that one key thing which is the interfaces and the protocols and sort of that common publish, subscribe or whatever mechanism to deliver the data from producer to consumer. So it means we have to kind of evaluate how are we gonna get there? What are the different ways that we will begin to get to very standardized ways of doing all the components that I've talked about here? And so one of the great things about worldly mapping is things move from left to right on the map. In general, as we saw with that S-curve earlier, I think things move along that chain from Genesis to being a commodity utility. So you can begin to ask questions about, well, how are things gonna move? And so there's a chapter that goes through sort of gameplay, the idea that in Simon and worldly mapping about the use of certain techniques and ways of influencing how a technology begins to evolve along the way on that map. And so he's got an entire huge collection of ways of doing things. And I don't go through every single one. Some of this is left as an exercise to the reader, so to speak, but I do walk you through three very specific ones. And just as a highlight of it, like the standards game basically says, you can essentially create an ecosystem or encourage an ecosystem in a marketplace. If your technology is a good use of standards, a good opportunity for standards usage, if people agreed on a certain way of doing things, it would open up for customers a larger world of technologies or solutions to use for different purposes. And then standards might be a really good way to kind of move your technology forward. So standards game is kind of at the heart of what I'm said with Flow is, the entire event driven architecture ecosystem market really ought to be playing the standards game together. But there's also then that flywheel that I was talking about sort of more and more solutions come to play, therefore more streams come into play, which means more customers are consuming the streams and finding more needs and finding more uses, which creates a more opportunity for more streams to come in. And that network effect thing is a really, is a big part of it as well. Like being very conscious and aware of trying to build network effects is going to be a big part of the vendors that are successful in the flow space in the long term. And then finally, co-creation is that idea of cooperating around building, the components for solving certain problems or building certain ecosystems. So things like maybe the, there's a government agency that works with the healthcare industry to define data standards and around healthcare and patient data and so on. And so they could gather a number of companies around to sort of say, hey, let's bring our engineering talents together and figure out some certain solutions that we can put together that we can then share as open source technology or just common technology approaches towards certain problems of handling patient data and moving patient data around. And also sort of working together to, one group of vendors working together to compete against another group of vendors is another really great way of doing things. So just an idea there, but you get a lot more detail on that stuff. But I think this is a really great way of beginning to evaluate a technology that's considered the future. And there's a number of these other ones that apply really, really well. And I have a blog that I started writing around this. So you can Google it, I think probably at this point in time. You can certainly, if you follow me on Twitter where I'm just at James Urquhart, first name, last name, you can, I certainly will share all the blocks that come up there, but I will certainly be talking about more of these gameplay opportunities there as well as time goes on. Okay, terrific. That's awesome. That's wonderful. That's great. So what do I do with all of this? Like, what can I do right now to prepare for flow or help drive flow forward in some ways? And so in my opinion, and I've been talking to a lot of people about this, right? I've been talking to, I've been talking to lead architects in places like Microsoft and AWS and other places. I've been talking to folks that have been involved in messaging and open source community for 20 or 30 years now. And really, there's a couple of ways to look at it. First, there's some basic patterns. These are just very high level patterns, but they have different sets of needs in terms of what you consider in terms of how you use the processing technologies. They often use the same processing technologies just in slightly different ways. But, so I'll go through these, I go through the patterns a little bit, but I also, and more importantly, I go through a set of use cases that were shared with me by Derek Collison that he's evaluated messaging and eventing applications over time. These are basically the four categories of use cases that he continuously finds and very, very rarely finds something that doesn't fit well into one of these categories. Addressing and discovery being, discovering that there are elements out there to connect into the flow network and creating identifiers for them, which is great. And then you have command and control systems where basically that's using events to then coordinate activity and send instructions to take activity or events that trigger activity around the network. Where an observability is about the ability to look at sort of specific agents or specific small groups of agents, smaller groups of agents. In terms of what their specific behaviors are and specific states are. So the ability to ask questions of the system of flow that you have in all the agents involved in that flow and the ability to observe and even sort of receive events based on the emergent behavior or overall behavior of the system, of that small group of agents. And then telemedia and analytics are the same thing, but for the system as a whole, in terms of understanding then, without identifying specific agents, identifying a little bit more about how the system is behaving, what the overall emergent behaviors in the system are and whether it's generating the value it's supposed to generate and so on. So I go through those as well and talk a little bit about sort of the technologies and involved and the approaches involved in basically addressing those use cases today and that are well set up for the flow of future. And then there's also this really, this is to me was my big aha moment in writing the book, at least I'm writing the latter part of the book, which is essentially looking at how you make decisions about what technologies to use for different use cases and different aspects of what you're doing. And Clement Vasters, who's a lead architect of the messaging and eventing systems for Microsoft Azure kind of pointed me to talk he did, which laid out this wonderfully simple sort of call tree or decision tree that you can look at, which is essentially looking at the types of events and the types of actions you wanna take against those events and drilling down to very specific technology capabilities that you would be looking for to process those events. At least as a consumer and potentially as a producer as well. And so it ranges everything from, if you're gonna do a lot of interactive conversation between agents, you might wanna use a message queue. They won't scale as well, but then those kinds of conversations don't scale as well anyway. But you might then look for a log-based approach if you're looking at, hey, I need to be able to retrieve a series of events from a certain time range or a certain related larger scale, like an ID, like a customer ID or something. And or you might, if you're looking more at discrete things, you may say, hey, I just want, every time I get an event, I wanna take an action. Or I may say, you know what? Every time I get an event, I need to trigger a workflow that may trigger a series of other actions and either through events or through API calls in a number of steps to resolve whatever action it is that I wanna take. And you might actually choose to use a workflow automation engine, things like step functions and Azure Logic Apps fit really well in that. So this is actually a really useful little section of the chapter in sort of understanding these different pieces and how to make these decisions. And if you kind of follow some of these guidelines and use some of these technologies in the right way, then you're fairly well set up so that as these interfaces become standard, then it's just a change of interface to the way that you're processing things already. And so it's a good way to be kind of prepared and from a interaction, from a flow interaction perspective. And with that, I'll say, you know, the last thing that I talk a little bit about and I'd love to see more people do, the reason I wrote the book was to get the conversation around flow going, get the conversation around how we can make more of a broader ecosystem and more of a commodity ecosystem around event streams. There's many ways you can get involved. There's open source projects to get involved with out there. CNCF is doing the cloud event spec, which looks like it's very much as the inside track in terms of metadata protocol for the way events are being passed around. There's, I think all three cloud providers have been involved in it. There are a number of technologies involved. And then, excuse me. And then also you might be a part of a trade organization or something. It's good probably to have at least a little conversation to go with the data protocols that we're using today. Would they fit well into an event payload? What would we have to do to change them to make sure that they do, if that's the case? And maybe, you know, read up on cloud events and understand maybe how you would wrap your data with the metadata in your specific use cases, your specific industry use cases. And then lastly, just, you know, a friendly push to say, hey, the book's out there. It's available on Amazon. It's available on O'Reilly. It's available. You know, where books are sold, more or less. And I highly recommend that you pick up a copy. Would be grateful for your feedback if you have any. Once you've read it, I write to learn in a lot of ways, and that's what I hope to do. And with that, I think kind of open it up if there are any questions or comments people want to make, I'm open to hearing. Awesome. Thanks, James. So back to the Wardley map, the first, the black, before you got to the red. This one. Yeah, sure. So I think this is an interesting play, like, first of all, meta analysis of Wardley maps is one of my hobbies, so bear with me. I'm curious how some of these got put where they are, and we could talk about that. And then I also think it's worth noting, like, when you think about what Wardley map implies, right, is like, there's a current state, and then there's the things that happened before. And lots of what we're talking about was sort of like the way the big web was built, right? Like Kafka was built inside of LinkedIn in the late 2000s, some things, right? So it's open source in 2011. The very first Amazon web service was SQS, right? So that's basically the sort of proto or whatever you want to call integration, streaming, queuing type of thing, right? And so lots of the big web was built with these sort of asynchronous flow. One of the big topics at velocity conference, like 2010 was conflicts of event processing. And that kind of stuff, right? So like this is all interesting. One thing I'd be curious, how you're thinking about it in these conversations, and maybe you could argue it's kind of buried in discovery is governance and who has access to, like I think it gets, this is something that's on my mind because of some of the projects we're working on. But when you start talking about like data and streaming and having that be available, that's fantastic, but you don't necessarily want everyone to have all of the data. So I wonder or have you thought much about how to handle governance? Yeah, so to me, governance is a part of either the interface or a combination of the protocol and the processors depending on what you're talking about, right? So there's a lot of things about, and I do talk a fair bit in the book about the challenges as we move forward, like some of the problems that need to be solved in order for us to really kind of get to a flow that works. And one of them really is things like data provenance. Like how do you guarantee that the data that you received is the data that was intended to be sent by the source or by the next processor back in the process, right? That's a big thing. That's a big problem to be solved. There are technologies that are working in the right direction. No, not blockchain, but there's other things that come into play in terms of that. So there are some technologies that begin to look at elements of that, and there are some definitely custom solutions in that governance space. I put it in logical connection in part because a lot of the identity stuff is being handled as kind of productized pieces of the interfaces that are out there today, right? So the flow, the access to a stream, role-based access control, or individual ID-based access control kind of stuff. The discovery piece is more about how you figure out what flows are out there and what the requirements are to consume that flow. And the reason it's very much in the genesis side for me is there are a couple of companies trying to already move into the product space on this, but there really isn't a lot of understanding when you talk to people who do event streaming about ways to catalog the streams they have available in their organization, right? That's still very early. There's people still trying to figure out basics, and there's definitely not anything out there that has a standard way of describing what's the data rate that I can expect from this flow? What's the encryption method used on the payload? One of the things I need to know in order to be able to subscribe to the stream, well, what kind of ID do I have to have in order to be able to be granted permission to access the stream? And so I put it very much in the genesis side. It can be argued to be in custom built at this point in time, maybe approaching product, and I'm open to that argument, but I think it's such an unknown and undiscovered space that I even argue that it's 50-50 that it could just end up being Google and webpages. Discovery could just be like APIs have been to date. There's not really a registry of APIs. You just go Google, hey, what's the API for Twilio? What's the API for OpenShift, right? And so, or for Tanzu, so from that perspective. The meta level is all manual, and then you maybe have a service catalog once you're kind of inside of the family. Yeah, there are some catalogs out there, like Bantic, a low code, no code, and then targeting event streaming has a really good interesting catalog of the event streams within your instance of Bantic that you might wanna consume. And there's a company called Solace that's doing some, actually has launched a product that is intended to be an event stream catalog standalone product that you can consume. That's also a part of their overall suite, but you could use it as a standalone piece. But I think so, you know, so, but I don't think they define anything about how discovery will work and flow. I think they just are kind of first stabs at that. So this might be nuanced. My initial impression when I first saw this is I kinda wanted to push a lot of it back towards the left. I'm not sure some of it's as deep into commodity as it's represented here. I'm curious when you talk about infrastructure, like what you're putting in that bucket. And then in specific, I think there's an interesting nuance when you start talking about some of the things people wanna be able to do with, you know, what was IoT and now it's Edge or whatever you wanna call it because infrastructure in that sense is in different phases, right? I think like when you look at cloud computing, you're seeing the commodification of these sort of centralized pools of fungible resources. But when you start to look at the emergence of these Edge solutions, that's definitely not in commodity for infrastructure. Yeah, that was a little bit of a tough call. The call I mostly made is basically the hardware, even in these cases you're talking about, basically the hardware is at this point in time, a fairly commodity thing. There are advances in chips that are happening. It may, there may be another go round of another set of technologies like ARM that move into becoming the new commodity, but generally there's not a lot of sort of highly customized way. So I think this is actually an interesting conversation because it touches on what's happening with cloud computing. So I think you can argue that the hardware is commodifying or commodified, but the operation of that hardware, the deployment of that hardware, imagine that the life cycle of that hardware, that's not commodity. But I think that's out, I got it, you know, we could argue this again, this is what I'm doing. I'm just trying to understand. This is what maps do. I would say that's outside of the scope of the map, right, there's infrastructure. Okay, the infrastructure is available. I don't care how it's operated to make it available to me. There's infrastructure I can run processes on the keys on- That's why all these companies are bad at this because they think like what you just said. What's that? I said, that's why all these companies are bad at this is because they think like you just said. I would argue that the defining advantage of the cloud natives is this operational excellence. Okay. Someone better care. Somebody better care. I absolutely agree with that. If I were to break infrastructure down, that infrastructure bucket would spread out depending on what you're talking about significantly. I completely agree with that. I just think that for the technology need stack that we're talking about here, I think the scope stops at the fact that, you know, I just, I need the basic networking, compute and storage that I have to have in order to support my processors, queues, sinks and sources. That's kind of where, Yeah, that's fair. It's kind of where this goes from. I mean, this also for the listeners at home, like we have talked about this for years, right? Like this is in the first conversation. Yeah. Yeah, exactly. I thought about James, so. Yeah. I think like there's two things that I have questions about. One is like when we, we can say the infrastructure is out of scope, but like doesn't things like data gravity issues eventually mean that for instance, if you say infrastructure is out of scope because you're using AWS, they're managing my infrastructure, but they're also managing my storage. And therefore the map isn't actually as kind of free form as you might think, because you're more likely to use queues, processors, consumers, et cetera that are provided by AWS. Yeah. And you are to choose from random things. So you actually end up, you know, having more like regimes of maps where there's already kind of preconfigured, like the easy way of doing it that are distinctly related to whoever's operating your infrastructure. They're commoditizing, but they're not necessarily interchangeable and fungible. That's right. Right. And that's a complete, although all of them as far as I'm aware, are based on open source technologies that are at this point in time, something that you could go pick up and create your own service that mimics, right? So with the exception of like Kinesis is kind of a not, it's not Kafka, but it's meant to behave like Kafka, right? But you're right. You could do a map of these for each of the major cloud providers that would probably lay some things out a little bit differently, or at least you would, instead of having just a processor Q layer, you might actually have for Q, you might have four different, services that could serve as different forms of Q for different purposes that are maybe at different stages. But one of the things, I do talk significantly about edge computing in the system because one of the things is how does flow scale? And when you look at that question, Jeffrey West's book about how systems in which something flows through the system in nature, in economics and in cities, how they naturally form, how they seem to form the same basic structures or very, very metaphorically similar structures. And in that way, I fully expect that edge computing is actually a major component of it. And it's fair to argue that edge component infrastructure may not at this point in time be commodity. That's a fair argument. Totally open to that. But I would say that ultimately the capability, the compute network and storage capability that you're trying to get out there through the edge is at least commodifying if not at this point in time commodity. So the second thing I wanted to kind of poke at a little bit, like when I squinted this, the first thing it makes me think of is semantic web, right? Like you've got discovery, you've got producers, you've got different ways of thinking about data. So what went wrong with semantic web? Like how did semantic web fail to become this type of thing? And part of it is they started with OWL. They actually started with standards and that's what broke the semantic web because they couldn't agree on the standards, right? So I'm wondering like if standardization isn't something that actually keeps some of these things from ever becoming commodity. So that's certainly, before I even wrote the book, that was something I spent a lot of time thinking about because in the famous debate between Sam Johnson and Ben Black always kind of comes back to me, right? And ultimately Ben was right, right? But the, so what I would say is this, we're not talking about, we do have the strong, we do have a distinct possibility that different types of streams that that map that I showed that that decision chart might end up with different interfaces and protocols. Potentially, right? But in the end an event is an event. It's a encapsulated state change, right? That has a time signature and a couple other things assigned to it. So that's the same for all of them. So it's possible that the interfaces might be different. It's possible that there are aspects of what's in the metadata, for instance, that might be different for different use cases. A really good example is I talk about high frequency trading a lot and how the mechanisms work for that today. You couldn't, with the way I have defined flow right now and the technologies I've called out here, you couldn't do high frequency trading with flow, the way I've defined it right now. So maybe an interface protocol for me that does direct port-to-port, direct ethernet port-to-port connections to maximize speed or minimize speed. So it's an argument that you couldn't do it because of the latency or is there another? Yeah, the argument would be that compared to what they do to squeeze out every little millisecond out of the processing time, there's too much overhead and publish and subscribe. There's too much overhead really but to kind of make it work extremely well. But like, I mean, part of what I mean is like, even if you had all those APIs, et cetera, the relationship between like metadata that the producer is publishing and the consumer, the consumer isn't required in any way to absolutely interpret the data inside the event the same way another consumer might. No, that's true. And as a result, you get these relationships that are always dynamic and complex between producers and consumers, especially if you start making them relationships that are cross organizational boundaries or industry boundaries, again, because we've seen this before, you can't get enough rigor around the metadata to enforce the meaning of every field effectively at an industry level. I was really clear and then we'll reiterate, the protocols or the payloads are not going to be defined so that every event has the same payload for it. That cannot happen. And in fact, what cloud events did extremely well is they acknowledge that almost in the first paragraph of the standard rate, they're like, look, all we're trying to do is make sure that when an event is received by some entity that processes events. And by the way, the protocol isn't like, it's not a byte protocol, right? It's not something that says this is the way you lay out the data. It says these are the types of information you should pass with the event in whatever line protocol you're using. So it maps, there's a binding to MQTT, there's a binding to AMQP, there's a binding to HTTP. And so from that perspective, then all you're saying is this is the information that should be readily available and consumable about an event in a format that you know how you can pick it apart, right? And by doing that, it's not the, to me, the thing with the semantic web is they're trying to define the nature of information, right? So we're linking the world's information, let's define the nature of that, the meta-nature of that information. That's a failure, right? We know that at this point. Nothing about what the way I define flow tries to define the meta-nature of what events are and what the world's activity is. I'm not trying to say, hey, these are the categories of activity. These are, I agree that that won't work. All I can, and I can't even predict what the future is other than say, boy, there's a real strong push, like the specific mechanisms, the specific technologies that will win out, whether it'll be public cloud that becomes the dominant player or whether it'll be some new vendor that comes out of nowhere, I can't predict any of that. I can just say that these are the needs that get identified as you analyze the map and the movement on the map. And so from that perspective, the only thing I can hold dear and sort of say this is what I believe will be true is to say, if we can define the standards around sort of how we package an event and the interfaces that we use to say, hey, I'd like to subscribe to that topic. Yep. Right? If we just do that much, that's an amazing, amazing opportunity to create an entirely different math for integration. Cool. Yep. So I'm gonna, we're almost at the top of the hour or actually we are at the top of the hour. And one of the things I'm gonna say right off the bat is that this probably is the best used at least this week of wordly maps I've seen. So, and it really helped drive the discussion so nicely. And I'm also a big fan of that as well as the promise theory stuff that you've done. So if we had another half an hour, I'd talk about that. But I think for me, this has been really an eye opener and it really kind of, we were supposed to on Monday have the cloud events folks on but we had to postpone because somebody had one of the folks had an issue. So yeah, so look forward to like next week or the week after we'll be having a cloud events, the folks from Francesco Gianni who's at Red Hat and Scott Nichols from VMware. We'll be back on with Paul Maury and myself and we're gonna deep dive into cloud events. So if anybody's listening and wants that, look on the calendar and that's coming up soon. I love this, these concepts and everything they kind of pull together where we sort of see the market going anyways. This is almost a market driven technology and I think it's going to evolve. So we'll definitely have you back James and bring you back as this all evolves. And the one thing I'm gonna say about this what wordly map and right off the bat is I love that the discovery piece is in the heart shape part of the map right there. It's like, I think that's the key as we've all sort of teased out here in the conversation around building a rich and diverse and trusted ecosystem. So I think and in this map it's still in the discovery phase or the Genesis phase here. So I think a lot of work and conversations in this book hopefully will drive the conversations too around that aspect of it because James use of the word regimes to describe how different vendors or different groups of vendors will set themselves up. I'm always leery of the word standards or standardization as well and we could probably go on for days about other standardization efforts we've all been in but really having the market driven piece and aspect of this and using places like the CNCF cloud events community and these to drive these conversations and figure out where it is we're going is really incredibly useful. So James I am grateful for you have taken all the time and effort to write this book because we all know how phenomenally hard it is to get a book out the door and for coming today and sharing the conversation and Andrew and Jade and everybody else who's listening in this is really the beginning of a really good journey I think that we're all gonna have to take together. So thanks James and look forward to posting this up getting people's feedback and making hopefully the slides available to James if you send them along with me I'll post them up and I'll actually put them up on slide share but in addition to that I'll send you a link and you can post them up if you want to as well. All right. Well Andrew and Jade any last words of advice for James? I mean it's good to see you James thanks for coming. I think it's always fun to watch your thinking evolve. You know we're all trying to build this better future and I think that event driven architectures will be the big part of it that you know the flow architectures are emerging right now. Yeah I appreciate that it's always really great to talk with you Andrew and Jade it's a pleasure to actually get to spend some time with you. We don't I think we've met a couple of times but I don't think we've really had a chance to kind of talk through things very much but yeah this is gonna be a great this is the very very beginning it's kind of cloud circa maybe even you know 2007, 2008 maybe but really you won't see this be a thing that mainstream is worried about for I would argue seven to 10 years easily but it is something it's an opportunity I wrote the book I wrote the book to get people thinking and talking and perhaps somebody creating something incredibly innovative and crazy that none of us expected. Lots of them are trying to use Kafka so we'll see. Yeah I know and I think that that's why they need to look at that decision tree chart because there's other technologies out there that are good for other purposes but anyway. Absolutely. All right, thanks guys. Nice to meet you guys. Thanks.