 So, I hope you're here for the Kafka talk. Just FYI, it is a sponsored talk, so if you want to go to people who actually got in, you should go to other people's talks. Today is May 4th, so happy Star Wars Day. Did not bring these Jet-Rubbs, but I do own them. They're pretty awesome. I'm Terence Lee, if you haven't heard of me. I go by the Twitter handle, own02, and yeah, feel free to tweet at me if you have questions about the talk. Famous for my blue hat, because I keep wearing it. It's over a decade old at this point, and it's getting kind of afraid. But I do have blue hat stickers that look much more pristine, so if you do want those, feel free to come talk to me, and I'd be happy to give you one. I've been helping this Ruby Karaoke movement, and I think for the last three or so years, I've been able to have a karaoke event, whether I've organized it or not, which has been amazing. At every Ruby conference I've been at, which is pretty great. This is a picture from last RubyConf, and it was a lot of fun, and if you are intimidated about singing, you don't have to, but a lot of people come to Ruby Karaoke as their first karaoke event, and I don't know of anyone who's had a bad time. We'll be doing it tomorrow at 7 p.m. at off-key, so come show up, sing, bond with your fellow Rubyists. I think Charlie Nutter is going to be there, and it's definitely an experience like watching him sing Shakira is something that you really don't want to miss. I work for Heroku, so that's what this sponsor talks from. I work with Richard Schneeman over there on the Ruby experience, but I think I make him do most of the work, so if you have Ruby problems, you should probably talk to him and not me now. So now that we're done with the intro, let's talk about Kafka. So I assume most of you have come here because you've heard of Kafka, you may not know what it is, but there's probably some buzzer hype, and you're wondering what do I do with this technology, and why should I care about it? And so to kind of just go like how we're going to go through this talk is we're going to do a short intro about Kafka and the topics and vocab related to what Kafka is. We'll talk about how you can use Kafka with Ruby specifically. I built a specific demo app for this talk that will kind of walk through and do the deep dive there, and then my co-worker and friend Tom is going to go through kind of other patterns and new cases you can do with Kafka, besides the one we're talking about in the demo. So what is Kafka? And if you go to the Apache, it's an Apache project, if you go to the official documentation, you get these great two sentences as like the first paragraph that you read, it says, Kafka is a distributed partition replicated commit log service, provides the functionality of a messaging system, but with a unique design. I remember first reading this when I was trying to learn about this, and I pretty much had no idea what this actually said. So what Kafka, unlike a high level, for me, if you're to think about it kind of simply is it's like a Pub sub-meshing system. So if you use Redis, and you've used its Pub sub-features, or if you use MQP, it's kind of analogous to kind of that feature set, and it's also distributed, and you'll see why that's important as we go through the talk. And it's designed with these kind of guarantees that it's fast, it's scalable, talking about the distributed nature, and it's durable, and that being a really important point. So with regards to performance, we can process hundreds of thousands to millions of messages a second on a small cluster. And if you're thinking about MQP as a reference point, you're probably talking about tens of thousands of messages on a single node. So it's a huge difference in performance for even just using it as a simple Pub sub-message bus. And I don't know if you guys have used Skylight, the project, it's this great product that the Tilde folks do for monitoring performance of your Rails application, but from the ground up, when you're doing that kind of app, you need to have it be scalable and be able to do this kind of work, and they have Kafka as a core part of their architecture. And one of the tenets of that is that it is durable. So it's common to think that you need to restart your processes because of some crash or something, and Kafka allows you to basically have these commits where you have position and time. And then if your services go down, you can basically know when the last time you've done something and replay it to the most recent point as a transaction and not lose any data, which is a really important point when you're relying on a service to do metrics for you. So I know the Skylight folks have been, I've talked with them about this, and they've been very happy with Kafka in their application architecture for powering Skylight. So now we've talked a little about that. Let's kind of step back and go through some of the vocab that you're going to have to deal with when dealing with Kafka. So at the heart of it, we talk about producers and consumers. So if you've done any PubSub stuff, you're probably very familiar with this. Like you've probably used Sidekick and some of your Rails apps. And producers are the code inside of your Rails app that actually produces the work that is going to be done. And Sidekick would be the, or Redis would be the equivalent of the Kafka cluster here. And then consumers would be the actual workers, the Sidekick workers on the other end that are actually processing the jobs and doing work. And so that's very similar of how you, like a workflow of what you would do with Kafka itself. And inside the Kafka cluster, you have multiple nodes and those knowns are known as brokers. And when you're writing and reading stuff inside of Kafka, kind of the core like Adam piece of unit is called the message. And the message is really just a byte array. And so that means it can actually represent anything. So the simple example is a string. And so when you can have any type of string data and insert it in there. But you probably want something that's actually more structured. So I think it would be very common to do something like JSON. And you can really just have any kind of format there. But you can store anything as like the message value for these messages. And you're gonna have a bunch of these messages. And we use topics to basically organize these messages. So you can think of topics as a feed that you would subscribe to that has some set of these messages that you're going to want to insert into and read from. And you also need to have partitions. Cuz if you need to have high throughput, when you're dealing with millions of messages, you're not gonna want to store that all on a single machine. So partitions allow us to split that across multiple brokers and then also have replication and kind of divvy up the work there. And so each topic partition has this guarantee that it is ordered and it's immutable. So you can only write to the end of it. And you can't actually change an order of the existing messages inside. So it's an append-only log, and that's really an important point. Cuz it allows us to achieve some of these other guarantees as you kind of process that stream. And so since it's ordered, we can actually just uniquely identify a message inside of a partition by looking at the offset number. So each message has an offset, which is its position inside the partition. So Kafka allows you to start at any point within given an offset. So you can seek to any point inside that partition. There's another thing that you can have besides the message. You can actually have keys, so you can key a message. And the importance of this is that we have all these partitions and only within each partition is it ordered. But the keys allow you to basically put the messages with the same key will go to the same partition. So you can guarantee that if you had some key like base on user ID, all the messages pertaining to that user would go to that partition, which allows you to do potentially certain things with the consumer on that consumer cuz you know all those users will be there. And speaking of consumers, we have consumer groups. So this is a high level concept inside of Kafka, which is really neat. But allows us to basically have a bunch of consumers inside of a single group and have them subscribe to particular topics. And one of the things that's nice about the consumer groups is that Kafka will guarantee that each message is delivered at least once, as long as Kafka is up. So you may get a message more than once, but you can at least guarantee that it's at least delivered once, which is nice to know. So as an example, here we have this Kafka cluster, and so we have four partitions here, P0 to P3. And then we have two different consumer groups. So one might be like, say we're archiving data, like some advanced stream. So that's consumer group B because that's high throughput, so we might want more consumers there. And then consumer A might be doing some metric aggregation calculation. And so as we can see, as we add consumers to the group, the partitions get divvied up against them. And so if you only had a single consumer in a consumer group, it would get all the partitions for that topic that you're subscribing to. But as you add more of them, they get split up across them dynamically, which is really nice. And there's also a heartbeat system as part of Kafka. So if it detects that one of these consumer groups goes down, is not available, it will automatically redivvy the partitions across the rest of the consumers in that consumer group. So as you can also see, you can have multiple consumer groups that basically process the same topics or different topics. And it would have access to that data. And you'll see in these other, as we go through use cases, why that's important. So that's kind of a really quick run through of kind of vocab and things in Kafka. And things will make a little more sense as we start to go to the demo and see how we can use this in Ruby. So there's really two main Ruby libraries if you want to interact with Kafka. The first one that has been around the longest is JRuby Kafka. And this is actually just a JRuby wrapper around the official Kafka libraries from the Kafka team. So these are job APIs. So if you're using this, you actually have to get pretty comfortable reading the job API docs, which might be a little jarring at first, but it's actually not too bad. But the downside is that if you have an MRI app that you want to use this on, that you actually have to convert it to JRuby. The other one is called Ruby-Coffee, and it's done by the Zendesk guys. And it's actually a fairly new project, but it's pretty clean. It's the interface, the UI, UX is actually very good for it. I was pretty impressed. There are some other older Kafka libraries like Poseidon and other things, but they don't support consumer groups. There are some plugins, and they don't support the new Kafka 0.9 APIs, which are simply better than the 0.8 stuff. So if we look at a simple producer, instantiating Kafka is very similar to instantiating Redis, except the one caveat here is that we have the seed brokers and we pass in two instances of Kafka here. One thing to know is that you don't actually need to know all the brokers in your cluster, you actually just need to have a seed of them, and then once it connects to them, it will actually figure out all the other ones based off of the initial seed, which is kind of nice. And then we just call producer to actually produce a Kafka producer, and this is a synchronous producer. And so we just call produce on this, pass in the message that we want to send, and then we specify a topic of what the feed it goes to. We were talking about keys before, so in order to specify key to get those kind of guarantees of where the partitions that this message can go to, you can use this key symbol, and then you can actually even specify down to, I know I want this to go to this exact partition, so you can do that as well. And at the end, you just call deliver messages which will batch up all these messages and then deliver them and is a synchronous thing, so it will block here. And if it fails to deliver any one of these messages, you'll get an exception that you can rescue and kind of deal with. So this is that at least one thing that I was talking about before. But you probably don't want to do that in a Rails request, because you don't want to be blocking during the Rails quest. And so Kafka has an async producer, which is probably what most of you are going to be using in production. So the nice thing about the async producer is that all the work, basically, for sending messages happens in a thread safe queue and some background thread for you. And you can set these parameters to basically tell the async producer how often you want the messages to be delivered. So the threshold says, I'm going to wait until 100 messages are in my buffer. And then once it hits that, I'm going to deliver that message, deliver all those messages to Kafka. And then delivery interval will say, I'll wait a certain amount of time. And then when that hits, I'll then deliver whatever is in the buffer. We're talking about how messages can really be anything. And this is a quick example of how you would do JSON. So if you take some hash, convert to JSON, you can then just dump that into Kafka and then on the serialization end, you can just parse the JSON and read it out. If you're going to use this inside of Rails, you actually just need to set up an initializer in your config initializers, boot up Kafka like you would in the simple producer example. The one thing that's different here is that you can pass in the Rails logger, so all the logs from Kafka go properly to your Rails logs. And then we're booting up an async producer, because we don't want to be blocking those Rails requests. And then we just want to make sure when we shut down the Rails server that we also shut down Kafka, so we're not leaking any handles or resources there. And inside a controller, this example is basically creating all the orders. So we have an event stream of all the orders that are ever created. And it's very simple to actually do that inside of Rails. So the next part now that we know how to actually push messages into Kafka is like, how do we read any of the data that's in there? And this is the part of the API. Like I was saying, the Zendesk library is new, is that it's experimental and might change in the future. So kind of beware a little bit. So there's a few APIs that are out there, but the one that is probably the most pertinent if you're doing stuff in production is the consumer group API. And so we open a connection to Kafka like you would with the producer. You'd Sanchez it, pass in the seed brokers and stuff. And then when we call the consumer method, we want to basically specify group ID. And the group ID would be like the archiver or like metrics or something that describes what this group is actually doing. And then we call subscribe for any of the topics that we actually want to subscribe to. In this case, we have a greetings topic. And then we basically just have an iterator like you would normally in Ruby and just you can process each message as it gets it. And this is a blocking request or this is a blocking method. So this thing will not exit after it won't just fetch a single batch of messages, it will actually continue to pull for them. And one of the nice things is in Kafka 0.9, they actually added SSL support. So it actually allows you to encrypt the messages, the transport between your app and Kafka. And right out of the box, the Ruby Kafka client supports SSL natively, which is really nice. So you can just pass in your SSL certs and it just works. This is in contrast to the JRuby Kafka thing, which uses the Java APIs. Which if you try to do SSL on that is a much less pleasant experience. Because you actually have to write the cert to disk and then do a bunch of things and it is not as easy for sure. So now that we kind of have an overview of how we actually produce and consume messages, let's look at an actual example of doing it. So the use case I'm gonna do is metrics. And what I mean by that is we have basically some web traffic through some Heroku app. And based off of those logs, we want to produce data, like metric data on that traffic. So in Heroku, all your logs, you can actually consume them over HTTP. And we have this log drain that posts these messages out. And we're gonna post to this producer Kafka app that we have that will then insert that log stream into Kafka. And then on the other side, we have another Ruby app that can be the same as the Ruby app or it's just another Ruby process and it will consume it. So this isn't a web server, this is just the consumer group stuff. And it puts stuff into Redis so it's easily, you can easily pull it out in your Rails app or something. So this doesn't have to be Redis at the end. You can insert it into any data store that you want that's easily querable but in this case it was just easy to do this stuff in Redis. And I didn't build the actual Rails part of that cuz I feel like if you're at RailsConf you probably have a good idea how to read stuff from Redis. So I really wanna waste your time with that. So let's talk about kind of the first part, like let's look at the log drain and see how we can actually get data out of this thing. So if you run Heroku logs, so if you've ever been to coachtriage.com, this is that app and we can look at all the router logs, which is what we're interested in. And so here's a single router log from that app that I pulled this morning actually. And as you can see, we can figure out the path that we went to on that app. We can see the connect time of how long it took to connect to the dinos. We can see the service time of how long the dino took to service that request. We can actually see the status code. So those are the kind of the four things that we're interested in that we're gonna do stuff with this router log. And so we can simply add a drain on issue triage itself to some other app. So in this case it's called log drain. But this app probably doesn't exist or if it does, it's not my app. But it's some app out there that we can set it to. And that means that app will then get all these requests for every single log that comes in through the log drain. And what the requests look like is it's a post request and this is what's in the body. It's a collection of logs and actually one of the headers tells you is like log message, like count. And so you can see how many logs that are actually in it. But this is an example of what that looks like. And this is mostly follows RFC 5424 which is the syslog like RFC format. But we leave out this structured data thing. We don't replace it with the nil value which is like a minor caveat of like what is in the RFC. But luckily for us that there's actually a Ruby gem that we can use. There's a syslog Ruby gem that was created that allows us to easily process that we'll see. So now that we understand like what the requests are that we're receiving, like how do we actually build an app around that? So if we have a Sinatra app, we can actually just take the body, return like a two or two saying that we received it and then say accept it. And so the body would just be like the body of the request, right? And inside that process message, we need to have access to Kafka. So Kafka, the Ruby client is actually not thread safe. So you can't share the Kafka connection between multiple threads. So if you're using like Puma or something that is multi-threaded, you actually want to create a connection pool. So I'm just using Mike Parham's connection pool gem and creating a bunch of async producers, in this case five. And then when I process the message, this is the syslog stream thing I was talking about. You don't really have to actually understand what it does. But just know that it produces this array at the end of basically those messages as individual items in that array. And then with that, I can just iterate through them and then send them to Kafka. And so this will batch it up based off of the async like thresholds that I have. So when it hits those thresholds, it will then batch all those things up and send it so it's a single request. So that's that. And then looking at Kafka, so definitely one of the reasons we're here is we're here to talk about, we just launched last week a public beta Heroku Kafka, which has limited access. If you go to Heroku.com slash Kafka, you can kind of find out more about the service. And compared to running this stuff locally on your machine, it's really easy to once you have access to it to boot up a Kafka cluster. And so you just like any other add-on, you just run add-ons create Heroku dash Kafka and the beta dev is just like a development plan that has a minimal number of nodes and brokers and stuff that you'd want to use. And it takes a little while to boot up because it actually has to boot up all these instances, but like Postgres, there's a weight command and you can just have it spin until all that stuff comes up for you. And since you're consuming this thing over the network and we have the SSL support, Heroku actually provides a Kafka URL, which is an HTTPS like SSL URL to your Kafka cluster. In addition, it provides all the certs as mbars to your app. So using those environments, this is how you would connect to the Kafka thing from your Rails Ruby app. And then since this is a public beta, the Kafka namespace is not in the CLI itself, so you need to install this Kafka plugin. And then once you do, we can do probably the most important thing you need to do with this namespace, which is to create topics. And so when you do Kafka, it creates a topic in the Kafka cluster. So when you're doing stuff locally, you probably have it configured to actually just auto create topics as you use them. But in production, you don't really want to do that. So you have to mainly create this router topic, which is what we're using in the demo. And you can run info to kind of get information about the cluster itself, like how many topics you have, how many consumers and other things are running, what's the traffic, etc. You can even introspect individual topics and get some more information there. And so for instance, by default, you get 32 partitions when you create a topic. And one of the other cool things is like log's tail, you can have broken Kafka tail per topic. So you can actually tail the traffic as you're putting stuff in it. So finally, we're gonna talk about the consumer. So now that we have stuff being pushed into Kafka and we have an actual Kafka cluster, how do we actually do stuff with it? So this is very similar to basically the consumer group example. There's not much more code than that. So we connect to Kafka, we have this metrics consumer. So we're gonna name the group metrics. We subscribe to that router topic and default offset here, basically says if there's not an offset that we know about, we'll just start at the end. And the other one is like earliest, you could start at the earliest offset that there is. But it's fine to start at the end. We're using Redis, so we need to have Redis here to actually insert the data and then basically we're gonna go through each of the messages that we get and then we're gonna insert that data into Redis. And we'll see kind of how this metrics insert thing works. And so basically what this code does, it should be pretty digestible if you sit and read it for a while. But that's not that important for this talk. But it basically goes through and per route, we're calculating the averages for the service and connection. And in addition, we're also counting the status codes that you have. So based off this Redis thing, I can actually go through and see for every route how many 500s have happened or what the 200s are for that app, which is kind of neat. This isn't, something that would be definitely more useful would be having a window of time for all this information. So maybe a weekend since of being like, hey, what is the average response time for this route during this week? Or since the last deploy or something could be more useful, right? But since it's a demo, I basically did the simplest thing possible. And we're talking about doing multiple consumers. So I can actually do a different consumer that would consume the same topic that would get the same data and not compete with the existing queue. So I can have a replay consumer. And what's cool about this is I can basically take the production code triage app, see all the routes that have happened, and then basically replay them against my staging app. So now I can get production traffic load that is replayed to my staging application to get a sense of, does this thing scale? Like, how is this new code change working out, right? And it's really easy to kind of add these things and consume these topics. And the nice thing is that you have all this orchestration and something like Sidekick to actually scale this stuff. All you have to do on Heroku is really just scale the process, because we're dealing with consumer groups. So when a new consumer gets entered in the consumer group, partitions get divvied up across those consumers. So literally, this is all the code you need to do the consumer stuff, which is really nice. So all that demo code is available here at this GitHub repo. So you can go there and check it out. It's set up with Docker to make it easy to kind of run all the Kafka stuff locally, and I'm actually running the stuff in JRuby for the replay stuff specifically. And yeah, if you have any questions, definitely come stop by our booth, you're having problems running it. So at this point, I'm gonna introduce Tom Crafford, who actually works on the Kafka product. He's gonna talk about how we use it at Heroku and then other use cases. Two of the other ones around. Other use cases, and then how we use it. Sure, that's cool. All right, so I think- Oh, is that mic work? I think this mic works, right? Can you guys get it? Okay, sorry, other use cases for Kafka. One of the big ones people use Kafka for is messaging, right? Finance is like a big thing here. People actually use it a lot in finance, right? You have all these messages flying around about stock trades, about quotes, about other kinds of things, and you want them to be durable. You also need low latency, you know, you need high throughput. The stock market actually moves pretty quickly, it turns out. And you also need to be durable, right? You don't wanna lose messages. So messaging a big one, right? You've got something that's communicating to something else. And Kafka's great for that. Another one that's actually really, really good, Kafka's really good for, and that it was originally almost built for, is activity tracking. So LinkedIn built Kafka way back when, and been working on it really hard since then. And so what they do is they track everything any user does on LinkedIn, right? Like, user A looks at user-wise profile. But then all these other downstream teams need to know about that, right? Like you might have an analytics team that want to know about that. You might have a recruiting team that wants to know about that. You have all these people who wanna consume the same feed. And so there's just one feed that tracks all the activity every user does and all the teams consume off it. Shopify use Kafka. I'm not sure exactly what for. I haven't talked to them about that stuff yet. But you can imagine if you have a e-commerce site, it would be pretty good to know what people are looking at things, what people are buying and so on and so forth, what people have signed up, all that kind of stuff. Inside Roku, the Roku metrics dashboard is powered by Kafka and Roku Kafka. The metrics dashboard has to process hundreds to, well, hundreds of thousands to millions of messages a second, right? There are lots of apps on Roku. They're doing lots of things, lots of issue requests. And the metrics, the thing that powers the metrics batch board, sorry. The thing that powers the metrics dashboard has to consume off all of these and compute graphs. And these graphs are actually still in Postgres. Postgres can't do hundreds of thousands to millions of messages a second. What it can do is accept batches every now and again. So there's a thing that consumes off Kafka and every so often it writes, hey, over the last minute, this app had this memory usage. Yeah, that team's been an all successful, there's literally two people build it in a very short amount of time and it's kind of impressive that it works so well. The other big use we have internally is the Roku API event bus. So the Roku API is like what you talk to if you didn't use the CLI, you just talk to the Roku API. If you use the web dashboard, you're just talking to the Roku API. It knows about everything and that means every other team wants to know what the API did, right? For example, Billy wants to know when you scale up your app because they need to charge you at the end of the month. They want to know when you add it out on. But other teams also want to know about this stuff. For example, the Postgres team wants to know if you added a Postgres out on. So all these teams consume off this event bus from API and all we have to know is that these messages are in the topic. That's it. Yeah, and there are like 13 teams internally consuming off the Roku API event bus and we'll have more, I think. I think that's where I got to. Yes, so hopefully after learning a little bit about Kafka, it's interesting to potentially start looking at it to see how it fits in your application architecture. I think Rails is in a good spot these days where it was, it's been around for a while where it's starting to be in more critical path things and it has been for a while for certain businesses. It's dealing with more traffic than ever as part of that and you need to have things that are fast. You need to be able to deal with that traffic, make it scalable. And there's definitely certain concerns when you're talking about transactions and things where you care about durability and you don't want to be losing those kind of records if you're using something like Redis. And hopefully this gives you some insight or inspiration to explore and stuff with Kafka. And that's kind of all I had for my talk and I would like to, I guess shout out to Joe who helped me. He works with me on the Java, or he works with me, but he does the Java stuff at Roku and he helped me a bunch with the demo bits and I guess Godfrey as well who's sitting in this front row helped me with the amazing logo at the front of the talk. And yeah, we're gonna have community office hours tomorrow during the happy hour. That's at 410 at our booth. There'll be people from Rails Core and contributors. I'm committing Tom and Charlie who are in town to also be at the booth. They don't know it yet and of course there'll be Roku people as well. So if you have any questions about this particular talk, about Roku civic things, definitely come reach out to us. Be happy to talk about it more. Thank you.