 Live from San Francisco, it's theCUBE. Covering Flink Forward, brought to you by Data Artisans. Hi, this is George Gilbert. We're back at Flink Forward, the Flink Conference sponsored by Data Artisans, the company that commercializes Apache Flink and provides additional application management platforms that make it easy to take stream processing at scale for commercial organizations. We have Steven Wu from Netflix, always a company that is pushing the edge of what's possible and one of the early Flink users. Steven, welcome. Thank you. And tell us a little about the use case that was first applied to Flink. Sure, our first use case is a routing job for Keystone Data Pipeline. Our Keystone Data Pipeline process three trillion events per day. So we have a thousand routing jobs that we do some simple filter projection, but the thousand routing job is a challenge for us and we recently migrated our routing job to Apache Flink. And so is the function of a routing job, is it like an ETL pipeline? Not exactly ETL pipeline, but it's a data pipeline. It delivers data from the producers to the data sinks where people can consume those data, like in a search, Kafka or Hire. Oh, so almost like the source and sink with a hub in the middle? Yes, that is accurate. Okay. That's one of the biggest use case. And the other thing is our data engineer, they also need some stream processing to the data analytics. So their job can be stateless or can be stateful. If a stateful job can be as big as televised update state for a single job. So tell me what these stateful jobs, what are some of the things that you use state for? So for example, like a session like user activity, like if you have click the video on the online UI or those user activity, they need to be session like window, put a window session like those are the state typical. Okay. And what sort of calculations might you be doing and which of the Flink APIs are you using? So right now we're using the data stream API. So a little bit low level. Okay. We haven't used the Flink SQL yet, but it's in our roadmap. Yeah. Okay. So and what is the data stream down closer to the metal? What does that give you control over right now that is attractive? And what will you have as much control with the SQL API? Okay, yes. So the low level data stream API can give you the full flexibility, full feature set of everything. High level SQL is much easier to use, but obviously you have the feature set more limited. Okay. Yeah, so that's a trade-off layer. So tell me about for like a stateful application, is there sort of scaffolding about managing this distributed cluster that you had to build that you see coming down the pike from Flink and data artisans that might make it easier either for you or for mainstream customers? Sure. I think in terms of state management, I think that's where upper the Flink really shines compared to other stream parts and engine. So they do a lot of work underneath already. I think the main thing we need from Flink for the future, near future is regarding the job recovery performance. But in terms of the like a state management API is very mature, Flink is more, I think it's probably more mature than most of the other stream parts and engine. Meaning like Kafka Spark. Yes. So in the state management, can you, can a user, a business user or a business analyst issue a SQL query across the cluster and Flink figures out how to manage the, you know, essentially distribution of the query and then the filtering and presentation of the results across the, transparently across the cluster? I'm not expert on Flink SQL, but I think yes. Essentially, if Flink SQL will convert to a Flink job which we'll be using the data stream API. So they will manage the state, yes. So when you're using the lower level data stream API, you have to manage the distributed state and like, and sort of retrieving and filtering. But that's something at a higher level abstraction. Hopefully that'll be. No, I think in either case, I think the state management is handled by Flink. Okay, yeah. Distributed of the- All the state management you can just, yeah. Even if it's querying at the data stream level. Yeah, but if it querying at the SQL level, you won't be able to deal with the state API directly. You can say I'll do a windowing. Okay. Let's say you have SQL doing window with some sessionization by idle time. That will be transferred to be set for job and Flink will manage those window, manage those session state. So you do not need to wonder about it, but either way, you do not need to wonder about state management. How about if Flink take care of it? So tell me some of the other products you might have looked at. It's the issue that if they have a clean separation from the storage layer for large scale state management, you know, as opposed to in memory, is it that the large scale is almost treated like a second tier and therefore you almost have a separate set or a restricted set of operations at distributed state level versus at the compute level. Would that be a limitation of other streaming processors? No, I don't see that. I think it's different than you take a different approach. If I look like a Google Cloud platform, Google Cloud, they are thinking about using big table, for example, but those are external state management. Flink decided to take the approach of embedded state management inside Flink. And when it's external, what's the trade-off? A good question. I think if it's external, obviously your latency may be higher, but your throughput might be a little lower because you're going all the network. But the benefit of that external state management now your job becomes stateless. Your job is to make the recovery much faster for job failure. So it's a trade-off over there. Okay, got it. All right, Steven, we're going to have to end it on that, but that was most enlightening and thanks for joining. Sure, thank you. And this is George Gilbert for Wikibon and theCUBE, we're again at Flink Forward in San Francisco with Data Artisans, we'll be back after a short break.