 Good afternoon. My name is Mike Graham. This is Dan Carroll. We work for Comcast in the Denver offices. Dan and I have been working to modernize the monitoring system for Comcast's outside plant, leveraging Cloud Foundry and Kafka Streams, and we're excited to share those experiences with you today. We'll provide you with a roadmap showing you how to create a Spring Java app, how to add the Kafka Streams libraries to it, and how to deploy the application to Cloud Foundry application and container run times. With the goal of building a system that is distributed, scalable, reliable, performant, and reactive to events in real time, we'll discuss recipes for creating and deploying apps, for scaling apps, for managing app state, for modularizing apps, using streams to build microservices, and for implementing REST services that expose a streaming app state. By adopting these technologies, you'll see the same benefits that we've seen, including reduced developer friction, resulting in increased innovation and significantly reduced time to market, immediate successes from initial laptop-based development spikes to full-scale deployments, application modularity resulting in smaller team sizes and deployments, the ability to easily scale applications horizontally, and application deployment portability using the Cloud Foundry run times, which helps prevent Cloud Vendor lock-in. We'll talk about these things in more detail. But first, let's take a quick look at the high-level application domain. We monitor Comcast's outside plant, but what is it? Outside plant is the broadband infrastructure that connects Comcast's business and residential customers to manage data centers and onto the internet. This includes customer devices, cabling at the customer location, drop lines from the customer location to shared fiber and coax cabling that's either buried or elevated overhead that terminates at a nearby computing facility called a head end. Parts of this infrastructure are outside and are subject to extreme weather events and construction activity events. In addition, customers can be impacted by power grid events like power outages. As you might imagine, Comcast's customer satisfaction is directly tied to the reliability of the outside plant. We're constantly looking for ways to improve the reliability of this infrastructure and to improve our ability to monitor it. Here we have a schematic view of the outside plant. You can see the drop lines that connect the customers to the shared cabling that terminates at a computer system called a CMTS, or Cable Modem Termination System. You can see that we organize the customers that share common cabling into groups. These groups are located in geographic proximity to each other and are often impacted by the same events. We call these groups nodes, and each node has a unique identifier. The goal of our monitoring system is to analyze these nodes. To do that, we need an inventory of the devices and their corresponding CMTS, as well as reference data like thresholds for computing good, warning, and error levels of device properties. Once we have the inventory, we can pull the devices in the CMTS to collect attributes like signal to noise ratio, transmit and receive power, device online offline status, and more. We analyze this data to detect various event conditions on the node. Once we have the event conditions, we can compare it with the previous event conditions. This allows us to generate event status for the node in three lists, new events that have just started, continuing events, and events which are no longer present on the node and have just completed. From the event list, we can calculate a node score, which is a measure of the health of the node. From the event list and the node score, we can prioritize the work for the text in the field. We decided to use Kafka Streams and Cloud Foundry to implement this new system. Why is stream processing a good fit for building this type of system? Stream processing closely models our world of streams of asynchronous events that we want to process in real time. Stream processing is modular. You can see that we've identified several high level processing functions and the input and output messages for each. You can imagine that each of these can be further broken down into a network of smaller processing functions with input and output messages. In addition, we have a very nice high level graphical representation of our system that directly maps to the underlying implementation. Stream processing is built using distributed systems architecture so it can be scaled horizontally. Stream processing is resilient with built-in data redundancy, failure detection, and automatic failover in both the Kafka server and the Kafka Streams applications. We'll talk more about streaming in a minute, but first let's take a look at how we deploy an Angular and Spring Java application to the Cloud Foundry application runtime. Ultimately, the Cloud is just a collection of physical computers with IP addresses. What makes it powerful are the abstractions that have been created that allow developers to deploy and manage applications in this environment. We want to deploy our applications and scale them with multiple instances and have the instances be restarted automatically if they fail. Let's take a look at how we use the Cloud Foundry application runtime to accomplish this goal. We start with the Cloud Foundry command line interface, or CLI. The CLI can be run on a developer's laptop or as part of a CI CD pipeline. Using the CLI, we log in to the target instance of the Cloud Foundry application runtime. These instances could be running in multiple data centers around the world. Some of them may be running in private data centers within the company, and others may be running in one or more of the public cloud providers. After we log in, we can use the CLI to push the applications to the application runtime. The push command inspects the application folder to determine the type of application that is being pushed and selects the appropriate build pack, which creates a container image of the application and pushes it to the application runtime. In the case of the Angular web app, we have to help the push command identify the appropriate build pack by creating a file in the app folder called static file. The contents of this file pushed state enabled will allow deep linking into the Angular application. In recent years, it has become much easier to get started creating these types of applications. For example, to create the Angular application, we can follow the simple instructions at cli.angular.io. To create a spring Java application, we can visit the start.spring.io website. We can identify the project's dependencies, like web and actuator, and then generate and download the project. We really like using Swagger with our REST services. Swagger generates metadata about our controllers and pojos that we use in our REST API. This metadata can be used to automatically generate TypeScript proxy classes that the Angular client can use to call the REST API. As our REST API evolves, we can regenerate the proxy classes, and the TypeScript compiler will tell us if we have breaking changes. To build our Angular application, we use the Angular CLI's ngBuild command. To build the spring Java application, we use the GradleBuild command. Now let's focus in on just the spring Java application and look at how we add Kafka Streams processing to it. First, we add the Kafka client jar and the Kafka Streams jar to the application by referencing them in the build.gradle file. Using the APIs provided by these jars, our spring Java application can interact with the Kafka server. The client API provides the lowest level building blocks for sending and receiving messages to and from the Kafka server. A producer is used to send messages to the server, and a consumer is used to receive messages from the server. The higher level processor API introduces a container class called a topology. With the topology, we can define a network of processors and define how the messages flow between them. We can define a source and add it to the topology that the source will receive messages from the Kafka server and bring them into the topology. Another special processor called a sync can be added to the topology to send messages out to the Kafka server. In Kafka, we organize our message streams into topics. Let's look at an example of how we could receive messages from topic A and filter them and send the filtered messages to topic B. To do this, we first create an instance of the topology class. Then we use the topology's add source method to create a source in the topology that's connected to topic A. Next, we create a processor, a filter processor, to filter the messages. And then we use the topology's add processor method to add the filter to the topology, connecting it to the source. And finally, we call the topology's add sync method, connecting the sync to the processor and to topic B. At this point, we can call the topology's run method to begin processing messages. The Spring Java app will connect to the Kafka server, begin receiving messages from topic A, and sending the filtered messages to topic B. This is a stateless example. The processor can perform its operation based solely on the contents of the messages. We can also define stateful processors that interact with a state store using get and put methods on key value pairs. To add a stateful processor, we create a state store in the topology by calling the topology's add state store method. The Kafka Streams Jar contains another API called the Streams API. This API is built using the processor API. The Streams API introduces a container class called a Streams Builder. Using the Streams Builder, we can define kStreams and kTables, which are abstractions of the processor API's source in state store. The kStream and kTable classes provide fluent methods that can be chained together to define the desired processing. One of these methods, the two method, can be used to send messages out to a Kafka topic. Now let's look at how we would implement our filtering example using the Streams API. First, we create an instance of the Streams Builder class. Then we use that to define a kStream object that connects to topic A. Then we chain together the filter and two methods. The filter method accepts a lambda function that is evaluated to determine if a message should be output or not. The two method writes the messages out to the topic B. We use the Stream Builder's build command, which actually generates a processor API topology, that we can then run to begin processing messages. When we're building our stream processing application, we can mix and match the capabilities from all three of these APIs. So now let's dig into the details of the stream processing starting with the client API. Here we have a topic, an unbounded, continuous, real time flow of messages, each with a key and a value. Topics are broken into partitions. Partitions are a unit of scalability and replication. Producers are used to send messages to a Kafka topic. The producer maps a message's key to its associated partition. The same key will always map to the same partition. So in our implementation, we use the unique outside plant node identifier as the key in our messages. And we make sure that all of our topics have the same number of partitions. So what ends up happening is a particular partition will contain the same set of keys across all of our topics. The consumer is used to receive messages from a topic. Let's say we create a consumer application and deploy four instances of it. You can see that the consumer negotiates with the Kafka server to determine which consumers will receive messages from which partitions. This distributes the load across our consumers. If there are more consumers than there are partitions, some of them will be idle and won't receive messages. If a consumer fails, its failure will be detected automatically and rebalancing will occur. One of the other healthy consumers will pick up its processing responsibilities. Now let's say we deploy another consumer with just a single instance. Notice that all of the partitions are mapped to that consumer and all of the messages feed to that consumer for the topic. Our Kafka cluster is made up of multiple brokers. The brokers share the work of managing the partitions. The brokers create copies of the partitions called replicas and distribute them across the cluster. The cluster will automatically detect if one of the brokers fails and its work will be picked up by some of the other brokers in the cluster. Because the other APIs are built using the client API, they follow these same patterns for achieving scaling and fault tolerance. So next let's look at the processor API and a topology. So here we have a topology with sources, processors, syncs, and state stores. When we add a processor to the topology, we indicate the one or more processors that feed it messages. The sources and syncs connect our topology to the associated Kafka topics in the Kafka server. Now let's say that we deploy or we create topics with three partitions each. And then let's say that we deploy our application with three instances. We've effectively subdivided our processing into three independent systems, each with its own data. If we need more scale, we can just create more partitions and more instances. So in our application, we have close to 60 instances and 60 partitions in each of the topics. But you can scale that up to 100 or 200 or however many you need. So next let's actually go inside one of these instances and look at how the messages flow and how the state store works. So here we are inside a running instance of the topology that's been assigned the responsibility to process partition zero. The source on the left ties to an underlying Kafka topic. Next to that, we have a processor. And next to that, we have a state store that the processor will be using. Below the state store, we have a change log topic. When a message arrives in the underlying Kafka server, it is consumed by the source. The source emits the message. The topology is responsible for orchestrating transmission of that emitted message to the processor. The processor can perform a get method on the state store to read a value. In this case, the state store is empty and no value will be retrieved. The processor can then do a put on the state store and put that message into the state store. The state store stores this message in a local data structure. So it's very fast. It also sends the message to a backing change log topic. Let's repeat this for keys two and three. Now let's look at a message whose key already exists in the state store and has a different value. When this message is put into the state store, notice that it overwrites the existing value and again is written to the change log topic. Let's repeat this for key two and again for key one. Notice that the state store contains the most recent value for each key and the change log contains all of the messages including the historical ones. If we want to query for the most recent values, we can query the state store. If we're interested in changes to the state store, we can subscribe to the change log topic. In addition, we can rebuild the state store by replaying the messages in the change log topic. Over time, the change log will grow larger and larger. This means that it will take longer and longer to rebuild our state store. There are three ways in Kafka to limit a topic's size. The first in default is based on time frame. Messages older than the specified time frame will be deleted. The second is based on topic size. Older messages will be deleted when the topic exceeds the configured limit. Both of these methods could result in removing the most recent key value pair for one of our keys. The third way to limit topic size is by indicating that a topic is a compacted topic. Periodically, Kafka will run a compaction process, which will remove the key value pairs whose key is present in a more recent message. So this is how we rebuild our state stores from a backing change log topic. So just to review, state stores are fast because the get and put operations are acting on local data structures. They're durable because they can be rebuilt from the backing change log topic. And they're scalable because they're based on the same distributed partitioning scheme as other topics. At this point, I'd like to turn it over to Dan, who's responsible for driving the implementation of our system. Thanks, Mike. That's a real good look at the processor API. Once again, my name is Dan Carroll. I'm a DevOps engineer at Comcast. I'd like to present some of our processor API patterns. All right, so we use state stores for de-duplication of node requests, materialized view of table data, and rolling aggregates. We have a use case to not have duplicate node polls running concurrently. So when Mike may mention before, we use SNMP to go out and crawl our network. And we generally do our polls at the node level, kind of the neighborhood level, to find out the status of all the devices on that node. So we don't want more than one of those running at a time on a node, because we like to reduce resource stress and also database locking. We had a little issue last year with database locking when we're hitting the same node with two different requests. So for our UI, we work in conjunction with sockets to return the response. There'd be multiple requests, only one will run, but multiple responses to the UI. We'll also publish the results to a state store. Okay, the flow for this pattern is we have the UI to kick off a node request. There's a couple other methods too. We'll look at those in a few minutes. So we have a node analysis request processor that takes in the request. It'll read the state store to see if that node is running. If it's running, it'll ignore this request. If it's at rest, it'll set it to running and then forward that request to the node analysis. Here it's represented as a cloud because there's a whole lot of steps to a node analysis. The last step in the node analysis is the publish processor. So that processor will first set the value in the state store for that node to at rest and then we'll publish the results. Generally to a state store, we can also forward the results to other topics for other apps to use or for any other need. We also have a use case that we need to have the current list of device data, location, device type, bunch of other things about each node and device, and also ref data, pull data. There's a lot of other needs for current data. So we reshaped the raw data for the intended use and put it in a state store and we're finding that it is faster than continually querying for the data that we need at that point in time. So we use the UI, a timer or an external system push, as we call it, to load the data to a topic and then a state store. So that's where we inject into the processors. In the future, we would like to get any change to data pushed to us so we don't have to do a load. We only wanna get the deltas but we do have to read a database at some point and get the data. With this method, like Mike mentioned earlier, we can query the app and not the database when we're running. Okay, so if we start at the top left, we have currently most of our data still is in a database in our existing system. So we have a timer that can kick off a load. We have the UI app that can kick off a load and we also have a rest endpoint where you can kick off a load so other systems can actually kick off a load of data in our system. A general, it's the same pattern for each one though. You query the database, you get the raw data as we call it so that's all the data about the node. You stream that to a topic. Off of that topic, we'll have a processor that'll change the data to the needed use. So in this case, for some node data, we have a state store for a device and location. We have another state store to keep the stats about the node and then we have another state store for some performance characteristics about the node. So that processor will write all those records there and then that targeted data is there and available to the rest of the application inside of the application. So every time we need to find the devices for a node, we don't have to query the database. They're right there in the app. We also have a use case to keep track of the steps of a node request. With this, we can pull stats from all the node requests we do a couple million a day. It really gives us a good look into checking into the health of our application and it gives us a window into the node analysis step so we can see what's going on with any individual node at any time. So here's a partial look at our UI. The names of the steps have been changed for the demo but just from the elapsed time, I can tell that step 3A is the actual pull step. So this is where we have the list of devices. We're gonna go out and use SNMP, ping each device on that node and return the results. So what's helpful for this, especially for DevOps and various debugging is if we see a spike in that elapsed time, we know something's going on with the node pull. So whether it's through network latency or something going on with our system, it's also helpful to see every other step runs much quicker. So if we see elapsed times going longer and other steps, we can target operations right to that step and what it's doing so we can find out what's going on when we see events. Also, the lack of a step tells us something too. If we don't see step six running, here it runs multiple times, that's one of our analyzers that we call them. Then we know something is happening before that step or that step can't finish and then we could target operations right to that step too. Okay, now I'd like to present timers and streaming logic for our data pump and our plant validator. We have a use case to load data from other systems on a schedule. Generally we do a lot of things twice a day, it seems like. And this is because like I said earlier, most other systems don't push the data to us. They make it available through REST URIs and we have to, REST APIs, excuse me, and we have to go get the data twice a day. So it's a general pattern of streaming the data to a topic and a processor takes that data, shapes it to the needed use and populates a state store or topic. All right, so we are a spring application. So we can use the enabled scheduling annotation and then we use the scheduled annotation on the method that we wanna run on a timer. This one runs every 60 seconds and it looks like we're getting, it's definitely doing a web service call and we're getting I believe the inventory. So after, we're gonna look at response described in a minute, but after we get the answer, we walk through some of the no data and we use a producer record to get all that data back into a topic and back into our streams processing. Okay, now when we find events in our plant, some events have a soaking period. So we wait, I think it's generally about five minutes to see if that event is still happening. Some events go right to confirmed and we take actions on them. Other events, we wait a few minutes to see if it's still happening so we don't roll a truck if something is just temporary. Okay. Yeah, so in this case, we use a one minute timer, we check the status of an event and state stores have proven to be very useful for this because we don't have to query database, we have the data. So here's a look at that pattern and it looks a lot like the pattern from earlier. Instead of a database, we have the soaking events store. So every, I guess a minute I think in this case, the timer runs, it gets the soaking events that are expired and need to be checked again. It sends a request to a validation processor. The validation processor then runs a node analysis. It's actually a partial node analysis. We only run the steps and analyzers that look for the same problem. And then we have the validation result processor which will then take the result, figure out what status that event should go to, whether it's confirmed or an outbound or closed. It'll update the state store and forward some messages to other topics so we could take the appropriate actions based on that status. And we also have a use case to remove latency when we're using web services. There's one stream available, I mean, excuse me, one thread available per stream topology. So if we have a web service call, it will block the rest of the stream from running. So we switched to asynchronous calls and this was most evident when we were pulling a node and it takes 10 plus seconds. We noticed that nothing else was gonna get processed in that topology at that time. But then the second thing we found is when the async call returns, we're no longer in that stream's processing so we have to switch back to a producer at that point to stream the response or the answer back into the topology to get it running again. So from, you saw Mike's diagram earlier, the two red boxes are where we do the poll. So that, like I said, when we go out and crawl using SNMP to get all the device data, if those two boxes are blocking, you can just see that the rest of the topology would not be freed up. So that was our first clue that we needed to do all our web services via an asynchronous call. So I'd like to show the code pattern for that. This is just a straight on web service call. This is a post method. We're using a web client. We return a mono, which basically is one answer. We do a post in this one and it's just your other web service stuff, media type or context type and what we're accepting. So that code is the second line in this code example. And then the next, the response subscribe is the Java future that's waiting to get the answer back. So anything inside of the function inside of there will be run when we get the answer from the web service call. And then that code is the business logic on this line. So after the business logic runs, that's where we pull the data apart and get the record ready for the intended use down the line. We then have to switch to a producer to inject back into the topology because this function here, the thread didn't wait, it moved on. So when this runs, it's not in that topology anymore because you're really outside of the producer. You're inside of this function when it runs. So when we're looking for solutions to redo our current application, we found for these use cases that streaming is faster than a traditional database for these uses. We're finding it's not monolithic, like Mike mentioned before. We can add new things to the topology without breaking the other things because we just kind of stream over to that topic, see if it works, then we can bring that topic into the rest of the application. You can see from the programming model, we only have a couple of patterns and we were able to solve most of our issues. So that was a, it's a very straightforward programming model. And we're finding it's fundamentally different but it does mirror the real world of the events that we're looking for. All right, I'd like to turn it back over to Mike for some technical notes. Thanks, Dan. So let's take a quick look at some of the technical notes we've made along the way. It's easy to stand up a fully functioning system on a laptop. We've talked about how we can get the Angular and Spring Java applications deployed to the local environment. We can also set up a local Kafka server. We can visit the kafka.apache.org website, download Kafka and start the zookeeper and Kafka broker services. Or we can visit the confluent.io website and download the confluent distribution which includes the confluent CLI. We can use the confluent start command to start up all of the Kafka services. This is also nice because we can run confluent destroy to create a clean, empty Kafka instance and then run confluence start again. So in your development loop, that's really nice. When you can experiment by creating Kafka topics and experiment with the streaming applications, when you're ready, you can push these applications to Cloud Foundry application runtime, like we saw. We may wanna create stateful services for use with these applications. The Foundry website at cloudfoundry.org provides a gateway to the Cloud Foundry ecosystem. This website lists services that support the open service broker API. You can create these services with the CF create service command from the Cloud Foundry CLI. In this note, it's all about the naming. Suppose that Dan and I are sharing a Kafka server as we do our development. One day Dan and I are both running our development environments and Dan notices that he's not receiving the messages he's expecting to see. Do you know why this might be happening? We're using the same topic names, the same consumer group names, and the same topology names. So Kafka happily load balances the partitions between our two running development environment instances. This is obviously not what we want to happen, and we fix this by creating a prefix for our topic names, consumer group names, and topology names. We create these topics using a script that has the environment topic prefix, the number of partitions we wanna create, the target Kafka cluster on which to create the topics, whether we should delete the existing topics or just create the new ones. So this script will create a collection of topics with the specified prefix. This is an example of a monitoring system for a Kafka server environment. This system revolves around creating JMX properties for the Kafka brokers and exporting them, configuring Prometheus to scrape the properties at a specified interval and accessing the properties from a Grafana dashboard to display and alert on the data. Now let's look at a streaming application that we have here, which has a topology with a graph of connected processors in state stores. Let's say that we wanna break this application into two independently deployed applications. We can sever our topology into two topologies. Notice that the two processors are now separated across our application boundary. We can reconnect them by creating a sync in the first topology and sending its messages to a Kafka topic and then creating a source in the second topology that receives those messages and plugs them into the second processor. The topic that they share represents a contract between these two independently deployed applications. We can use a technology like Avro to help us define this contract and we can even disallow breaking changes to the contract. We can repeat this pattern for another independently deployed application and again as many times as we like. Notice that the dependent applications are receiving the messages and the producer does not need to be aware of how those messages are being used. This pattern called receiver-driven control flow facilitates a pluggable architecture where we can add features to our system with very little risk of breaking existing functionality. Let's look at receiver-driven control flow in a little bit more detail. Here we have another example with a topology that has an analyzer. Its internal implementation is shown as ellipses here and its inputs and outputs are connected to Kafka topics. Let's say that the business has ideas about how they would like to modify this analyzer and we wanna test those ideas out. We can deploy a second version of the analyzer and tap into the same message stream that feeds the first analyzer. This has very little risk of breaking or modifying the first analyzer. We can take the output of V2, send it to a state store and create a rest service so that we can look at what V2 is producing. We can extend this further to compare the V1 and V2 results and store the comparison in the state store. And we can build an Angular application to display the version comparison results. We could even email these results out on a periodic basis if we like. Let's say that we're happy with the results and we wanna slowly phase V2 into production. First step is to take the output of V1 and send it to a processor that we will use to suppress the V1 messages that we want to flow through the V2 analyzer. The next step is to create a mechanism for declaring which nodes we want to flow through V1 and which ones we wanna flow through V2. We do this by creating a rest service that we can call with the node inversion. That rest service will produce a message to a Kafka topic with the node inversion. We can then create a source, a processor and a state store to collect and store the node version information. And update the rest service to read from the state store. Now we can build an Angular application that we can view the node inversion information and we can also update it. The final step in the process, we suppress the V1 results for the nodes we want to run through V2 and we create a processor to suppress the V2 results for the nodes we want to run through V1. So notice that the sync that's sending the output to the Kafka topic is now receiving messages from both the V1 and V2 analyzer and the state store is determining which version gets output to that topic. So we can just set one node to be V2, see how it works. If we like it, we can start adding more nodes and then eventually we can push them all across. So we've found that we can actually keep an updated diagram of these topologies. The other thing that's nice about this is the diagram actually maps directly to the underlying implementation. So you can see a dash boundary around part of the topology. We call that a subsystem and it maps to a Java package in our project. The items within that dash boundary are actually classes in the Java package. So this actually solves one of the very difficult problems that we have when we bring new developers on where they can look at the diagram, understand what it's doing and then find the associated code. Finally, we can access the state stores through a REST API. We've talked about doing this, let's look at how it works. This is called interactive query in the Kafka Streams documentation. So here we have three instances of our application. It's been deployed to the application runtime. It contains a REST API and it's apology with a state store. The application runtime will automatically create a load balancer in front of our REST services. So when we call the REST API to ask for a particular key in value, it will randomly select one of the instances. If you look at the diagram, the instance that's selected may or may not contain the state store that has the key of interest. If the state store contains the key, we can just return the key and the value from the state store. If it doesn't, we can issue a second REST call directly to the instance that has the key and value and return that. The way we're able to do this is through metadata that's made available to us in the topology. We get the topology to capture this metadata by setting a property called the application.server property in the topology. The rest of the mechanism has to do with dependency injection, so we create a spring component class. We save the running topology as a property in that class. We inject that class into the constructor in our REST controller. We pick up the metadata from the running topology, and then we make the determination about whether we need to pick it up from the local state store or go to another instance to get the key and the value. If it is in the state store from the running topology, we can actually ask for the state store and do a get to get the value for the key. All right, now I'd like to turn it back over to Dan. Let's talk about running at scale and various logging techniques. Thanks, Mike. All right, so we use Kafka in our Cloud Foundry environment for application logging and Oracle Feed for event comparison. So we have a use case where we needed a logging solution that we can monitor. So when we were first developing and we had sometimes three app instances with three partitions, we can just kind of watch the logging and see what was going on. Once we went up to scale with 57 app instances and thousands of nodes, it became impossible to follow the logging. There was lots of solutions out there, but we just used the built-in capabilities of Kafka for our logging framework. It also works in our local instances because we use log4j there. We then use Kafka Connect to send the logging data to Elasticsearch, and from there we can visualize using Kibana. We also have use case to follow a node through all the steps of a node analysis. So we use the same logging framework we developed using Kafka and then we have a swagger endpoint where we can turn on and off the logging for that node. So here's, still has that stuff on it. So here's a look at the Stratos UI, provided by Cloud Foundry. It's an aggregate of all 57 logs. So if we're doing a couple thousand node analysis a minute, you can just picture, you can't read this, it goes by too quick. You can actually get two of the logs for each of the instances of the app, but you don't know what instance you're, if you're trying to look at an individual node, you won't know what instance that's gonna be hosted in. All right, so here's, you know, like Mike said, we are a spring application and swagger enabled. So we have this endpoint where we can turn on the logging for an individual node. So usually we turn it on temporarily, look at the node, see what's going on and compare it. We can also at that point compare it to existing app and then we turn it back off. So it's kind of the same pattern we saw before. When you hit try it out on this screen, it'll inject a message to a topic. And then the process around that topic will update the state store value for this particular node and say, yep, start logging this one. So during the node analysis, we'll write all the statements for that node, all the logging statements. Here's a look at our partial look at our UI. This is our application homepage. This is the UI where we can hit a node analysis. So the green button there for run. Once again, there's other ways we kick off node analysis is just one of them. And then here's a console consumer of the log, the Kafka topic we use for logging a single node. So this is good enough. Sometimes you can follow the node there. Being that we usually turn on one or two nodes at a time, you won't get overwhelmed with how many log messages are going by. But to make it even more useful is we send all those log messages up to Elastic. And then from Elastic, we can use a visualization in Kibana. This is just Kibana showing the messages as they came in. There's other visualizations we can use. In this display, there's not actually the message either. We had to blank out to see MTSN node, but usually you get the real message that's in the log. So with this, we can query over time for the logs and we can do aggregates like the average response time of a node request, how many times various nodes have been had a node analysis done. A whole lot of good information can come out of Elastic and Kibana. So we do have a use case also as we run parallel between the existing app and our Kafka app to compare the results of the two. The existing app stores its data in Oracle. So we need to compare the Oracle data to the Kafka data. So one solution we have is to trigger, have a trigger on the Oracle tables that write to a Delta table. So we'll have all the changes in that Delta Oracle table. What's good about that solution is it doesn't affect the existing app either. So we have our own set of tables for comparison and we can't get in any trouble with the current app writing to the main tables. And we use Kafka Connect for the JDBC publish of those Deltas to a Kafka topic. And then we have a streaming application that does the comparison for us. So a quick look at that is say the existing application is on the left, it finds an event and or it needs to update an event. So it updates the event tables. We'll have a trigger on those tables which will push that Delta to another table as we call them the Delta table. Kafka Connect will see that record and push it to a Kafka topic. So over in our Kafka application, we do have our Kafka topic in our state stores with other events. Those messages will also be pushed or read by the comparison app. And we have right now like a running total for instance of events. So any event that happens in the existing app and any event that happens in the Kafka app, we just keep a running total of them. And if we're about in sync then we know everything's running okay. There's other comparisons too but that's just kind of a main one. Let's go on. Oh, that's it now. Mike's up again. Okay, in this final section of the talk we're gonna look at containers and the cloud foundry container runtime. Let's return to the running instance of our application. Look closely at it state stores. State stores are fast because they use local data structures and they're durable because they can be rebuilt from backing change log topics. We've talked about how we can push these applications to the cloud foundry application runtime. When these instances restart and they come back up their disk storage will be blank. That's called ephemeral storage. In this case we will have to reload the state stores from the backing change log topic. If instead we deploy our application to the cloud foundry container runtime we can use Kubernetes stateful sets. These stateful sets have a number of advantages. Our instances will have stable persistent storage which means that when our instances are restarted and they come back up the disk storage from before the restart will be reattached to the new instance that came back up. In this case we will not have to rebuild our state stores from the backing change log topic. We get a number of other benefits from stateful sets including stable unique network identifiers, ordered graceful deployment and scaling of our streaming application and ordered automated rolling updates of our streaming application. So how do we deploy an application to the cloud foundry container runtime? Let's return to our deployment diagram for the cloud foundry application runtime and look at how we modify it to support the container runtime. The cloud foundry container runtime provides us with a Kubernetes cluster that we can use to run our applications. To deploy into a Kubernetes cluster we are responsible for building our own containers. We use Docker to create container images of our application. We push these images to a harbor image registry. Then we define a Helm chart that identifies the number of instances that we want for our stateful set and references our container image in the harbor image registry. Then we can run Helm install to deploy our application. The cloud foundry application and container run times can be run separately or together. When they're used together they can share cross cutting concerns like authentication, authorization and service access to the services that we created with the service broker API. So as a development team at Comcast we just request and are given a Kubernetes cluster but now next let's look behind the scenes at how these Kubernetes clusters are created by the cloud foundry container runtime. Here we have a Kubernetes cluster. Kubernetes is an open source, best of breed container orchestration engine that's provided by the cloud native computing foundation or CNCF. Our development team accesses the cluster through the Kubernetes API. We can deploy our applications through this API using Helm charts or directly using the kube control CLI. Kubernetes exposes a number of resource abstractions shown here. The most important for our purposes is the stateful set. Inside the Kubernetes cluster we have master, SCD and worker services running on multiple computers. Consider the operational effort required to provision these computers, to install and configure the operating systems on each and to install the master SCD and worker services. Consider the additional operational effort required to scale the cluster up or down to patch the operating systems or to upgrade the cluster as new versions of Kubernetes are released. Google also faced this problem and developed an internal system called Borg to address these needs. Later, VMware's Paul Moritz recruited several of these engineers to create a refactored version of Borg called Bosch, which is the Borg name with the R and the G incremented by one letter. So it's Borg plus plus. Bosch is an open source tool chain for deployment and life cycle management of large scale distributed systems like Kubernetes. We control Bosch by issuing commands to the Bosch CLI. One of the key benefits of Bosch is the ability for deployment portability through infrastructure abstractions of servers, storage and network provisioning and management. Bosch groups these IaaS abstractions into an interface called the cloud provider interface. Implementations of this CPI for private and public cloud are what allow Bosch to automate deployments to a large number of private and public cloud backends. Using the CPI, Bosch provisions the virtual machines and installs the operating system from an operating system image called a stem cell. In addition, Bosch installs an agent service on each of these VMs that Bosch director can use to monitor the health of the VMs. Bosch can automatically recover from a VM failure by either repairing the VM or provisioning a replacement. The cloud foundry container runtime is what's called a Bosch release. This is a set of jobs for installing the master, etcd and worker services. The CFCR Bosch release is guided by a deployment manifest which specifies the number of instances of each job that we would like to create. Using the CFCR release and the deployment manifest, Bosch can install the master, etcd and worker nodes in the cluster. The worker nodes actually contain the containers in our stateful set. If a new version of Kubernetes is released, Bosch can perform a rolling update at the cluster without bringing down the applications. Bosch can also deploy a rolling update of the stem cell OS as new versions of the operating system are released. If we wanna scale the cluster up or down, we can just edit the deployment manifest and redeploy. Here we're showing the use of Bosch for deploying the cloud foundry container runtime, Bosch release. Many other Bosch releases have been created for deploying systems like MySQL, Redis, Kafka, and more. Now let's return to our Kubernetes cluster with our deployed application, streaming application. Notice that it's accessing the Kafka server outside of the Kubernetes cluster. What if we wanted to deploy our Kafka server inside the Kubernetes cluster? At this point in time for production deployments, I would actually recommend deploying Kafka server onto its own physical servers or VMs if physicals aren't available. But in the not too distant future, it may be recommended production practice to deploy Kafka into the Kubernetes cluster. Let's look at a couple of ways to deploy Kafka server into Kubernetes. The first is very similar to how we deploy our own applications. We start with Docker images of the Kafka server services, ZooKeeper, the Kafka broker, and the Kafka Connect. We use corresponding Helm charts for ZooKeeper, Kafka, and Kafka Connect, and a Helm chart of charts to install all of the services with the single command. The result is three stateful sets, one for ZooKeeper, one for the Kafka brokers, and one for Kafka Connect. The second way to deploy Kafka server into a Kubernetes cluster is even more exciting than that. Kafka Kubernetes has a concept of a Kubernetes operator. An operator is a method of packaging, deploying, and managing an application, a Kubernetes application. A Kubernetes application is an application that's both deployed on Kubernetes and managed through the Kubernetes API, or Kube control tooling. Confluent is developing a Kafka operator for Kubernetes, which has a number of benefits. Confluent has used their operational knowledge of deploying Kafka at scale in cloud environments to automate the operational tasks in the Kafka operator. They can perform rolling updates on the Kafka server as new versions of Kafka are released. They can automate adding and removing brokers and rebalancing the partitions across the brokers. It can also perform a number of internal and external configuration tasks. So we can deploy Kafka server into our Kubernetes cluster. What else could we deploy in there? How about our entire application and all of the supporting subsystems? This is a future state for sure, but something we will be keeping a close eye on. Before we conclude, I'd just like to say a few words about Comcast Open Source. I'm very proud of the open source efforts of Comcast. Comcast encourages engineers like myself to open source the projects that we develop at Comcast. Comcast contributes to a large number of open source and open standards groups, including the Linux Foundation, the OpenStack Foundation, the Apache Foundation, Cloud Foundry Foundation and the Internet Engineering Task Force, just to name a few. Comcast is committed to being part of the global open source community now and into the future. If being a part of this vision appeals to you, feel free to stop by the Comcast booth and we can talk about opportunities. Dan and I would like to thank you for your attention and for being part of the Cloud Foundry open source community. Questions? I guess we can take questions if anybody has any. Yes? Okay, so yeah, so we have physical clusters. Repeat the question. Oh yeah, sorry. So the question is, how big are the Kafka clusters that we use in our implementation at Comcast? So we have six physical servers for the Kafka brokers. And this actually performs extremely well. We haven't hit any capacity limits on those servers. Do you know? No, I've had 200 or something thousand nodes. We have a process called WAPR that runs through a complete node analysis of all 40 million devices every four hours. That's running on top of that infrastructure. And like Dan said, we have 57 instances at this point of our application. Yes, the Kafka Streams Spring Java applications run in Cloud Foundry application runtime. And just to kind of as a technical note, we originally thought it would be good to have sort of odd number or a prime number for our number of partitions and instances. But in hindsight, I think power of two is probably a better choice, right? Because then you can start out like with 64 partitions and four instances. And then you can double it and double it and double it and you have the same number of partitions being allocated to each instance. Yeah, I think we actually are running 57 just because we thought maybe an odd number would be good. Yeah, right, so when you configure a state, I'm sorry, the question is, what are we using as our state store mechanism? Because Kafka state stores can be in-memory state stores or they can be on-disk state stores. And the on-disk state stores use RocksDB as the mechanism. And it's really a simple data structure. It's a key and a value, right? So it's just a key value lookup. So my understanding is that when the JVM's memory reaches a certain limit, it will actually default automatically over to on-disk storage using RocksDB. So that you don't need to specify I'm using one or the other. It'll just overflow onto the disk when it needs to. We don't specify anything in configuration, but I think it overflows to the disk and when it does it uses RocksDB. But we've been working on this for a while. Most of the concepts that you see in here, I would say we had functional prototypes working pretty rapidly on a laptop. Like that's actually one of the really good benefits of this architecture is you can take a laptop and in a day have it set up and running the Kafka Streams app against a local Kafka server create topics and start experimenting with it. And that actually is one of the key decision-making factors for me when I'm evaluating something is can I learn it? Can I experiment with it? Can I test it out and validate that it's actually as good as it looks in the PowerPoint slides? But I'd say we've been working on this for a year. And we're now running in production in parallel with our production system and getting really good results. The base app with the concepts was probably three months and that was learning Kafka, learning how to interact with our app. So just to run on our laptops house was pretty quick and then to scale it up took a little bit longer. But all the concepts were the same. Like I said, there's only a few of them to get all our use cases done. You know, as we look at the next use case, we're like, oh wait, we can solve it with a very similar pattern we already have. So it was very, I thought it was very quick development. Yeah, I'd say in three to six months if you're actually starting up and you're able to deploy your Kafka server into a Kubernetes cluster and you have a Kubernetes cluster, I think your time frame could be compressed significantly. So the question is, we're recommending that you use physical servers for Kafka, for the Kafka servers. And the question is, why are we recommending that over deploying the Kubernetes into the Kubernetes cluster? And really it's just based on what we see out there in the marketplace. Like Confluent is developing these things, but they're all like pre-production. Like, you know, there are Docker container images out there of these services, but they're specifically marked as being not for production use at the moment. Like, you know, the Confluent Kafka operator is under development and you can get early access to it, but it's not, you know, it hasn't been out there and had a chance to bake yet. I really think ultimately that will be the, you know, the good path is to, you know, to be able to deploy all these things into a single environment. It has some nice security benefits like Sergey was talking about. You kind of create this perimeter around all of your sensitive information and then you only have to defend the ingress and egress from the outside world, right? So that there's a much smaller attack surface and things like that. Thank you. All right, thanks a lot. Thank you.