 Okay, so I'm going to talk about as we I guess we're almost have everyone in and this should still just be introductory material About scalable real-time architectures in Python. What do I mean by that? I? Want you to walk away from this session with a couple key ideas, okay? Specifically around partitioning and fault tolerance and how we can achieve that in building such scalable real-time architectures My focus will be on storm But the ideas are applicable to other tools as well such as spark Streaming or tools that you might roll yourself And I think the reason that we want to be doing this is of course, we're dealing with more data But we also want to be more responsive to that data I will show python But more at the end So i'm core developer jython. You may have seen my state of jython lightning talk yesterday I let me plug this book if you're ever interested in jython um, this is a great book I work at rackspace on these types of issues I have had the chance to work in um distributed computing for a while and especially failures at scale I teach the principles of programming language course occasionally at the university colorado Which is fun, but actually is done in scala um, not python and I Work with this user meetup on storm. We're probably going to be changing this to real-time streaming in the near future What sort of real-time architectures would you like to build? Well, I can think of a few and i'll just mention a few since we are time constrained Maybe real-time aggregation. This is your classic approach that you'd be doing in hadoop, but again streaming So you're not looking at this on a batch by day basis instead as it comes in you're updating your counts or other aggregate aggregations Perhaps you're building a dashboard So some extension of this Of the real-time aggregation I'm particularly interested in the idea of decision making Where you will be responding to information in your environment and taking some action So what are some of the common real-time characteristics Of such systems? Well, you're consuming streams of events You are Being event-oriented as an event occurs. You may take some action. You may want to compute. You may be doing something downstream You want to go and minimize the latency from the arrival of that event to that Potential computation certainly not ours Ideally going down to seconds or below in terms of that threshold of latency Such systems are often called complex event processing or you could call it stream processing It doesn't matter. These are the concepts the one thing that we are not doing here is doing hard real-time systems Oh one last thing You might have written one of your own type systems in fact. In fact, I'm going to show you what you might have written in the past So you might have written something like this Does this look familiar? Well, it should because it is about as generic as you get with a unix pipeline But what I wanted to show here is is that You have compositions. So you're being able to go and build this pipeline out of reusable filters Another nice property You can rerun this pipeline if any intermediate step failed But some problems I obviously Didn't show some of the details around you know tailing what not But there's also these aspects of how do you go and implement these other things such as joins windows But most importantly, how would you scale that thing up? I mean, I think we can all think about how that could be done. You might go and In some way describe One set of files is being processed by this machine One set of files by some other machine, but you have to go and actually manage that partitioning yourself So at rackspace we're using this Framework for complex event processing called espar Which has very much the same properties or again these this homegrown code And here's the problem. You need to ensure all relevant events about that given customer. It's in one place All right, if I'm going to be able to know something about this given customer I need to bring in all the relevant information. I have to go and put that in one place for that to happen We have to have some locality um again in terms of a This rackspace example where we're implementing global alarms. We want to allow customers Some degree of customization So we don't want to make it too hard coded Then run some computation. So again a very classic complex event processing system And simple sharding the one that we might just know be able to do Um quite readily Um by say well, maybe we'll do it all by all by customers and whatnot It doesn't necessarily um get us as far as we'd like because You might have again sharded on some of the key you might need to shard on multiple keys So what observations can we make? Obviously small problems are easy um How do you make a small a large problem easier you divide and conquer it Um to divide and conquer you need to have some horizontal scaling. We're no longer Building systems such that they always will require just that larger machine And said we build so that they can scale to a cluster of such machines But what do we know the more machines we add to the mix the more likely we're going to have failure as well Especially since we like to use commodity systems um And once we have failure then we have to go and coordinate Um So I have seen and maybe you've seen in your own environments that some sometimes people will propose Add zookeeper or add some coordination system And that just doesn't go and make the problem go away Even if zookeeper is awesome You still need to go and consider how do you manage failure? So yes zookeeper can go and give you all this in its toolbox And it's fantastic as a consequence But I'll tell you that that doesn't You know just assuming that you have distributed locking in your environment doesn't tell you how to recover From a failure and then going and releasing those distributed locks Um, the solution is not reboot the cluster How many have done that? So That's not a solution. We want to be it's okay for a given node in the cluster to fail But rebooting the cluster defeats the purpose of running a cluster So storm has some Terminology which I will Introduce by um as we go through but there is this idea of an event source which we call spouts We have event processing nodes called bolts And some topology to link it together in a directed acyclic graph There is strong support for partitioning fault tolerance these key elements That we started with in terms of thinking about how would I go and be able to build up a problem that I could divide and conquer on Storm is written in closure, but it exposes a java api hence my Use of jython here, although of course you could use some IPC mechanism to talk With you know a c python for example And that was actually done in the talk on tuesday where they were looking at the parsley system for that support It uses zookeeper To manage things, but you don't necessarily see it although you can use it as a resource for your own coordination And as part of the apache incubator Actually, there are some other apache incubator projects that are competing in the space probably the most notable would be spark streaming it actually Looks great in terms of The Support you can do with that Especially with its python integration. I think it's the the top contender and has some nice properties that One can look at that In terms of being functional that you do not see in storm These others samsa and s4 I don't know. I think that these are not Nearly as competitive and there's some interesting stuff around there like zero vm as well that you could potentially be doing So storm lets you partition streams So you can break down the size of your problem that returning to that idea of being able to divide and conquer your problem If the node fails Storm over started But here's an even more important aspect of what that means Oftentimes when we are thinking about Building systems, especially distributed systems. We want to think about what are the invariants provided by the underlying framework What does it give us? And this actually helps explain also Why you have this distinction between an event source and storm and event bolts this these processing nodes Because sources are the things That you are producing events that have to eventually be Acknowledged in storm Okay, so that you can ensure that All events are in fact eventually processed And when I said eventually by the way it actually means in an eventually consistent scheme um and likewise This idea of bolts Distinguishing that because you can always replay for them Hence this topology and The topology in terms of how many nodes you might have so this is describing how your cluster is being split up in terms of its resources Here's the Perhaps one of the more important invariants that's being provided The number of nodes for a given spout Maybe a Kafka event source The number of nodes for a given bolt Maybe that something that is actually doing some interesting processing of that event is held constant So during the lifetime of this topology, you know that you always have seven nodes for um your Kafka ingest Your Kafka spout What does that mean in terms of your ability to know exactly what's going on in your environment in order to maintain appropriate counters in other state? It's very strong Okay, so storm is taking an account one Extremely important aspect of how you divided that problem up because during the lifetime of this topology If you always know that someone is handling this problem and you can always go think about if we had people here You can always go to that person um Now you don't have to think about oh what happens if Um, we add some additional people. It's nice to be able to scale up problems. So that is a separate Issue in terms of scaling up the size of a topology. You would do that outside of a given um A run of your real-time computation Um, and I'll if that isn't sufficient motivation, I'll give you an example in just a moment Okay, I think I'm running a little bit slow. So I'm gonna try to speed up now um But these are most important concepts your your takeaways So You have computational locality You know that the events For a given node are the ones that are supposed to be there because that's how you define your routing Um, you can route on some sort of what they call field groupings um, again an example is your customer a tenant in a cloud um a region of some kind some way of breaking up the problem space in terms of your events And again What possibilities do you have if you knew all of the? Information you needed to know about a given customer and it was in one place It changes how you think about things So normally when you go and Write your queries and say, um, you know using a relational database the data is over there and you bring it to you for a given computation instead In systems like storm You move the computation to to the data that is the fundamental idea in a map reduce system like storm like hadoop Um, but unlike something like hadoop This is in real time So you're able to keep all of that data in memory Compute on it in real time And do something interesting so I should mention of course since it's mapping, um, you you might have multiple customers on the given node So you have to consider that But that's an easy distinction to make So again, you will know that it will be on this node and only on this node um There are other ways of routing such as random shuffling Um global grouping, which means that there's just one node everything goes there Obviously not scalable, but useful for getting totals Um, storm will track that success All you have to do is consider what your retries are and there are other ways of doing this in terms of Doing exactly once event processing, but knowing that at least once all events were Successively successfully observed and computed on Is pretty fantastic again think back to that pipeline that unix pipeline. I was showing you That's that same idea If the pipeline goes down, I can retry it Of course You have to handle retry, but what does that look like? Um, here's the first python code. It's easiest, right? Um, if you've already seen that record Then you can ignore it Otherwise process it if you Re um try this computation Then you if you haven't seen it in the in the context of actually being successful with it, then you can do that retry All right, that's a merge function. It's a simple. It can be that simple But it will depend on the nature of your problem. For instance, if I am doing something that's transactional is nature Um You are um wiring me $1,000 Okay You can retry that many times and I'll be very happy about that So don't think that Retries are always going to be successful and that's the nature of eventually consistent systems Now sometimes you actually have to have strong consistency Um, and there are of course other ways you could go and have some sort of balancing compensation or something like that and say Well, give me back that $1,000 And and actually in real wiring of funds that in fact does happen Um, oh another thing. Um, and you see this in any type of streaming systems Your streaming Should not capture everything that comes through This is that, you know, old thing, you know in order to do your computations You have to download the internet if you're doing web crawling. Sure. You do that Um But again, that's done an appropriate system So you have to window um, another cap caveats would be around there are no query languages You have to build your own and that's actually kind of interesting again with parsley has some approaches I expect that we will see these things emerge over time. Think of these as building blocks That give you these capabilities, but again These capabilities are a lot like a unix pipeline imagine and we know what we could build with those Um, so you may have to make your own maybe use some of these other libraries Okay, I'm going to go and just skip through this a little bit Um Except that I will say that um Zookeeper is an important aspect Of most systems that you would build And so you probably will be interacting with in some way so your Um Spouts are ultimately Responsible their event sources. They're responsible for ensuring that all events were at Um, eventually played through successfully Eventually acknowledged And you need to go and ensure that appropriate handshaking. So as you are consuming from Kafka You are updating your offsets in it accordingly Uh I should tell you that again as some experience with zookeeper if you actually try to do Uh, one thing I didn't mention at the very beginning. This is that people run a million events a second through a storm node and um Uh, and of course you can have many of those nodes running Um, if you try to do a million events or more per second against zookeeper That's not going to work. You have to do some sort of clever batching just like a storm does. Okay so And so you get into things like this in terms of the kafka handshaking What that would look like and I will go back through this in just a moment, but I need to move on so you can have Spouts like this. Um, and some of these are already written um mostly in java but Or in scala, but the advantage of storm is it's multi lingual in the sense that it all runs on the jvm You could run feeds like twitter through you can pull in data from kassandra The important thing to know is it's not that difficult to implement and I'll show you in just a moment Um, another thing is is that you can push or pull events into the system There isn't something where storm is just saying you can only it will only accept events at a certain time You can always um push events in But it will ask you when it wants events as well. So you can use that to help balance things Um Again, you're going to be using that topologizing invariant to know how your work is being split up So you might go and have um Something like a real-time dashboard. You might be using rack space cloud files or some other Cloud provider to send things out You might have some sort of real-time decision-making. I'm interested in auto scaling In an environment, you might often have contradictory information about how things are working converge on some action That's the point of which why you're bringing all that data in one place You may have something along these lines where you are doing some sort of real-time aggregation Um A couple of minutes. I will describe Um python on storm using clamp, which I briefly described in the lightning talk yesterday as well um Let's face it python is a great fit for writing your storm code Um, and we have this system called clamp that allows you to readily Wrap your python class classes so they can be readily used from java in some cases in just one line of code Um, so I discussed this yesterday where you could go and um have some bar clamp class Um, this is sort of a hello world example um, you can readily in Um, a couple lines that are unique to um clamp just go and um add it You can construct an uber jar Um, I'm in germany. I can say this um An uber jar your one jar Um, or a single jar build it has everything and distribute it to um storm Um, you can use it in this fashion but Here's what It really looks like here's some code that you could readily use say you have a monitoring spout You're going to be opening up your connection to kafka Um, there again. I I actually gave this talk at rack space. It looks like a There's an internal system that um, of course we talk about publicly called adam hopper, but again some others have a feed like twitter um You can go and read parse events by as it's asking for it from next to next tuple But again if you have something where you're pushing events you can emit at any time And then you're responsible for managing these callbacks Fail and act what do you do under those circumstances? um, perhaps you want to go And do some sort of computation on this. What would it look like? This is the pseudocode associated with it um This is all that you need to this is the basic thing that you need to implement you need to implement these three methods You're pretty much done. How complicated is that? It could be like your unix pipeline. It could be something that's um Really simple or could be more complicated again If i'm trying to weigh Events that are telling me. Oh, it's going this way and it's going that way and I have to figure out what really is happening But that's the advantage of having it all in one place So conclusions about this storm lets you horizontally skate out your real-time architecture You have to consider partitioning and fault tolerance In fact to me this is these are the key questions you answer This is how you actually think about what it means to divide and conquer Your large problem so that it works On this cluster of machines answer these questions And you get to go from what you were previously doing in terms of a simple unix pipeline To which was analyzing log files something very similar To something that can be scalable and again real-time All right, you can choose your favorite language, but I know what your favorite language here is Um, it's python and you know, you could again use um some of the mechanisms that are out there in terms of communicating Uh via c python or whatnot, but you could also use again jython and type directly into this big data Um, uh system, so I I definitely would advocate for it. I should Um Your strategies will work with storm Such as test urban development Have fun So this talk is available On my github Repo Jim Baker talks and any questions Yes Mm-hmm Yes Yeah, so so that's a great question. Um, yeah, sure. So, um The question is the long lines of They currently have a multiple data centers. They're running a storm cluster in each of these And they needed to go and consolidate that information in one place Now the first thing to know is that You do not want to There are very few systems that will span multiple data centers And storm is not one of those systems Okay, you are going to be running a storm cluster in one data center Another storm cluster in some other data center the way that you would Um do the spanning problem is use some q To ensure that data is Pushed to some central data center. You may you you may need to go of course consider what failure you might have It gets more complicated. So you you basically move that problem to your cues use something like Kafka For example Yes That's the best. Yeah, I mean there's certainly not something that you can do in storm because again it really Is depending on the fact that you know It's running on zookeeper and zookeeper doesn't span multiple data centers. I mean, yes, you can in a theoretical sense That's not going to be great So don't do that Use systems that actually are proven to work And you're not just doing some interesting innovation along those lines Which you will find is not fantastic innovation Um, so we are we are So, yeah, I I don't know if there are any other questions I this is something that I can take out Afterwards because I obviously would love to spend lots of time on it Anything else? so I guess we're done. Um, so please ask me those questions that you might have