 Yn ymddych chi'n ddweud, mae'r clwrs yn ymddych chi'n ddysgolol, ddydd y ddysgol, ddydigolol a ddysgol ar gwrth gwrth gwrth. Aeth y gweithio'r gweithio ar y gweithio ar y gweithio o'r llwythau yn y ddysgol. Mae'n meddwl i'r gweithio'r lathau o'r lef, oherwydd mae'n meddwl i'r clwrs, wrth gwrs mynd i'r llwythu meddwl i'r cwrs. Ie ddim yn gweld o'r cyfnodd yw bir ydych chi i fynd i chi'n gwybod eu byddai i'r hefyd yn y gallu bwysig. Ie, mae'r Tybs. Mae fydd y dyfodd i gael coding yn myf wahanol ddifŏ. Felly, y gwaith ywer o'r ffordd, mae'r dyfodd i gael rhiwm yn gweithio i'r dyfodd i'r bobl. Felly, y gallai eu cydweithio ar hyn ychydig oherwydd, oedd anodd wedi'i wneud ychydig yn ei wneud mewn gwhodol, Dwi'n dweud yw rydyn ni'n meddwl am gweithiaeth'n casgu i'r ddysgu a phasio, dwi'n sgrifennu i allaf i ddechrau. Rwyf yn cyfnod i'r gymryd hyn gyfrannu, ac mae'r cyfrannu i'r ddechrau i ddechrau yn buidlant o'r llai o bobl yng Ngysud Rhifmen, pan mae'r llai phanodau llai llifaf mwy o'r maeth ac sy'n ceisio fe wneud y bydd y cyfrannu hon! Mae'r llai llai llai llai llai llifaf, croes, cafgar, cysandrau, bwysinnon, y dylai'r ysgolnig a gael i chi weithio, mae'n grwp y ffeirio'r ysgolwch. Saeth yn iawn ei obon fel gorfod yw mae'n'n debyg llyfr逮d ers eu cyfrigiau a'i lle'i collad i gael eu cyfrigiau, a gennym i'r cyfrigiau a'u lle neu'r cyfrigiau, ac yna'r hyn i'w mwynhaeth a fydd yn bach yn eu wneud. Ac mae'n ddweud yw'n ddiwedd yn cael ei gael eich cyfrigiau, ac mae'n golygu i'w cael eu mwyawr. I'm not going to use fish and chips as an excuse to do a very simple simulation of a shop. I started with a simple model, gradually making it a little bit more complicated. The demo code is all available online and there are some ideas for things you can do to extend the demos. I'm not going to run live demos today. I don't never trust Wi-Fi. So I'll be swapping between my slides and recordings. I apologise for that. The source code for the demos is all in Python. The source code is entirely independent of Ivan. It all works with open source packages. For running and setting up the source code, for running and setting up Kafka and so on, of course, I work for a company that does this, so I use Ivan stuff to do that. The explanations of how to run the demos also does the same. For that I apologise, but you don't have to use our platform to use Kafka. You can use anyone's Kafka for this purpose. I've been in the industry for a good long time. I started out working with transfer formats where you would send a file, possibly on magnetic tape, to someone as the way you transfer data, but relatively soon we got into sending messages. Some of the things I cared about in my career are sending messages between components on set.boxes, so perhaps between the audio and video and the actual program information areas. Configuration between microservices. Again, that could be on any computer. Moving stuff to and from Internet of Things devices. Typically, you might be getting monitoring data off your Internet of Things devices. You might be sending firmware upgrades back to them. Between components on a set.boxes, you probably want to be using something lightweight like 0MQ. I was once involved in writing an inter-process messaging system called K-Bus, which was a kernel module. Between microservices, you probably should be using some sort of state machine-related thing to manage what state you're in. But when you're talking about moving lots of messages at scale reliably over the Internet, Kafka's a really good fit. This is where it begins to be interesting to me. Traditionally, one might have been using something like RabbitMQ. RabbitMQ is a wonderful product, but it's a right pain to set up. You get a variety of problems like back pressure and things, but you really don't want to have to think about it. If you don't know what back pressure is, ask me for stories afterwards. What I want from messaging is I want to have multiple producers of messages, so this might be the IoT devices sending messages back home. I want multiple consumers because if you've got tons of messages, it doesn't scale to just have one node consuming them. I'd like a good attempt at single delivery. We work with real physics. Single delivery can't be guaranteed, but I'd like a reasonable promise of it. Similarly, with guaranteed delivery, I don't want to be losing messages at a reasonable level. I'd like to know that if some component in the whole network crashes, everything is resumable without me having to do any great amount of work. In particular on set-down boxes, this is a whole nightmare. As I said, I don't want back pressure handling. I don't want the queue I'm writing to to fill up. I've got to wait before I can send another message. This is where ApacheCaft comes in. It was originally designed by LinkedIn to handle the tons of messages that you get when you are doing a distributed system that's talking with people about people information. There's a paper about how messages and logs, moving events and static data in databases are the inverse of each other. They're two different views of the same world. When I read that paper, my mind was exploded. When I got the ability to play with Kafka, I thought that was great fun. Some terminology. Kafka has messages, they're events, so they're reporting that something has happened. Producers send messages and consumers read them. You can have multiple producers. You can have multiple consumers. They are independent. They're not tied together in any way. A producer sends messages to one or more topics, and those are named. They have an ASCII string name. And a consumer reads from one or more topics. It says which topics it wants to read from. Internally, to Kafka itself, we have partitions which are used to spread the load within a topic. We'll get to that briefly later. This is a very simple diagram. We have producers sending to a topic in Kafka, which is in the cloud, and we have a consumer. So here we've got a producer producing events, one, two, three, four, and it sends those to Kafka, and the topic holds events one, two, three, four in the same order, and the consumer will read them out, one, two, three, four in the same order. So we've got order preservation, which is also important to me. If we have multiple producers, one is sending one, two, three, the other is sending BCD, which is interleaved in the topic, but they will still retain their ordering. The consumers, both consumers here are independent, so they will read those in the order there in the category in the topic, but they will read them independently and the order will be maintained. Sometimes that's not what you want. Sometimes you want your consumers to share the reading. So here we've got two topics, one with one, two, three, and one with BCD in it. The producers can choose how the data goes there. You can choose which topic by hand, say I want topic one, two, three, or you can say here is a key choose by the key which topic could go to. The consumers at the top right are in the same consumer group. They said we want to share consuming the topics. Each consumer will get one or more topics, but they won't both see the same topics. So one of them is seeing one, two, three, and the other is seeing BCD. The bottom consumer is not in a consumer group. It's entirely independent. It's like the previous consumers, it's just in all the messages in order. So we're going to model a fish and chip shop. I have a colleague who talks about Catherine in terms of pizza restaurants, but he's from Italy, so that's natural. I come from the UK, our food is greasier, so we'll start with a fish and chip shop. It's not going to be a very realistic model, but it's going to be nice and simple. We're going to start with a shop that handles cod and chips, which are always ready to be served, so I probably should explain some words. So cod, which I just mentioned, is the traditional white fish that's served in English fish and chip shops. And it's dunked in batter and then shoved into hot fat until it's cooked and then taken out. Apparently this is actually not as unhealthy as it sounds. Chips, if you're not familiar with English chips, think about french fries but kind of fatter and soggyer. Place, which we'll come to later, is another fish, but it's a flat fish. And I tend to say till when perhaps I should say cash register, I think till is a more American term, but it's what I tend to say. So notionally, we're serving a customer. The customer comes in, you can see the customer at the left, they go to the till, they place their order, the order gets sent by some means to the person who prepares the food. I'm afraid I'm stuck with preparer or food preparer. I could use server, which would be more natural, but then we're in a cloud infrastructure that would get really confusing when the server talks to the server and find out what they should serve. The food preparer gathers the food out of the hot cabinet in the middle of the fish and ship shop, wraps it up traditionally in paper, would originally have been newspaper, but that's not very sanitary. Gives it to the customer. So two things there, the shop is organised as a till at the front, so there's a table you go up to, you pay your money. Behind that there are the servers, behind them there is a hot cabinet and on the top there's glass panes with heated compartments in which the fish that has already been prepared is sitting and below that there are baskets in which the chips that have already been cooked are sitting. So this is fast food. Behind that there's someone who cooks stuff and puts it in the hot cabinet so there's a nice separation of concerns there as well. Now I'm not going to mention the customer again. The customer for our simulation is essentially redundant, we're not going to mention those on the diagrams again. It's basically got a till, the order gets sent through Kafka and the food preparer handles it. We'll assume the customer stuff is instantaneous. So how do we represent an order? Well, we probably want to count Wilkin, he's Jason. So we probably should have an order number so we can count how many orders we've had in the day. That's going to be nice and that keeps the order separate. Traditionally you'd get to be given a little slip of paper with a number on it and that's your order so you don't have to give your name or anything. And then the components of the order is the person here has ordered one portion of cod and chips, that's very traditional. The other thing is that you can order just chips or you might order a large portion of chips. For simplicity I'm representing a large portion of chips as chips twice. I'm not saying that that actually works in terms of scale but it's for simplicity. So now we have the first demo. So I'm using a program that does a terminal user interface for the demos and I now need to do the very clumsily switching screens. So apologise for that. We want demo one. So on the left here we have a till and on the right we have a food preparers. The left is a producer who's printing out their producing and on the right you can see the food preparer receiving it. So what's actually happening is the food preparer is receiving the message and then it waits for a tiny moment and then it converts the message to a ready in it so that's to simulate handing it to the customer. And you can see it's nice and simple. Stuff goes in, stuff goes out. It's a queue. This is the most awkward bit of the thing but because I'm using PDF for my slides I can't embed video. So the libraries we're using for this I'm using AIO Kafka which is a lovely Python library to talk to Kafka. It's asynchronous, it doesn't have to be but it's got beautiful asynchronous support. To set up the topics I'm using Kafka Python which is an older, nonsynchronous Python library that isn't actually particularly maintained. It hasn't been touched for some years. It still works but as Kafka support moves on it will lag behind the times. I will hopefully at some point be able to use other means to manage my topics which is all I'm using that for. To create the Tumblr UI I'm using a lovely package called Textual which is based on a thing called Rich which is going to investigate those separately. They're wonderful. It's a lot easier than spinning up a web service and then having to manage those in a demo. So to actually create our stuff we need an SSL context we need to authorise ourselves to Kafka so we download various certificates and we create a context. There's a nice helper in AIO Kafka to create a context and then we can pass that to our producer and consumer. So here's our asynchronous producer we specify where our server is I don't know why it's called... bootstrap server because Kafka is multi-node so the bootstrap server I'm guessing is the node you talk to first or something like that. So we have to specify host and port. We say we're using SSL, there are other options we pass that context and remember we're using JSON Kafka moves bytes around it only understand bytes so we need to say we want to serialise JSON and shove it in out as ASCII and then we'll wait our producer to say it started and then we have a nice simple loop on while the shop is open or some other boolean where we await sending things to the orders topic. In the demos the topics have different names but I've kept them simple for here. To receive stuff it's very similar, the consumer has much the same arguments the differences are we say the topic we want to listen to at the beginning so that's orders, at the end instead of having a serialiser we have a deserialiser. If you don't understand Lambda just ignore that for the moment there's a posh way of getting your change of data in there. We wait for the consumer to start and then we can do an async for loop and that will just keep giving us messages as they come in which is really nice. Traditionally with Kafka stuff you don't ever think much about stopping because why would you stop? Use other techniques for deciding to stop. So we get busier, we have more customers how do we cut with that? Traditionally you would add more tills we've got producers so now we're going to add three producers and we'll add three topics because then we can say each till we'll send to a separate topic the food preparer is still reading from all of them so it will get things interleaved nicely so let's add a till to the Jason from before so we can give a record of which till it's coming from this isn't particularly useful to our food preparer but we'll see later we might be doing auditing and things later on so at all to the code in the demo that you look at you'll see there is code for creating topics we just need to say that we want three partitions and that will give us three partitions and then we create three producers three instances of the producer instead of one so this is a demo for for that so on the left we'll see we've got three producers now three tills and on the right is one food preparer and you will probably spot the problem quite quickly if you look at the bottom we're up to 30, 16, 19 the food preparer is still working on board of 15 they can't keep up we've been very cruel to our person working behind the counter we're over stressing them so let's not do that instead we need multiple consumers so let's add one more food preparer and see what happens so at all to the code first of all we just create two food preparers however we want them to be the same consumer group if you remember earlier if you want them to share the messages we need to have a consumer group and all we do is when we're creating the CAF consumer we make sure that both of them specify the group idea as the same consumer group and that's just a name here it's a constant now in practice if I run the demo more than once there's a chance I might have left over things from the previous demo when I hit quit in the queue and I'm not interested in those this is not the sort of thing you'd run into a shop because at the end of the day you shut things down and start again the next day there's various solutions for this CAFca lets you say where you want your consumer to start from in a variety of ways one of which is you can say seek to the end of the queue so start at the end of the queue similarly we were sending to different partitions as I said earlier I can actually specify which partition I want it to go to that's the bottom option I'm saying partition till number one I can specify a key out of the value field so I could specify that's the middle option that I want it to use till as the key and hash it and decide which partition and the top option is let CAFca decide how to split messages between the partitions that's changed over the years as to what that means that's not the solely about solution because I've got a very simple model my model is choosing the till maps to the partition directly I've only got three so we now do a demo with our new extra preparer so we've got stills, we've got three tills on the left now because of the way I've set things up here it takes a little while for everyone to sort out so I have them all wait until they're all ready but now we see that actually two food preparers is just about enough to set things up which is good because that saves me hiring another person and there's only so much room in that space between the counter and the food cabinet as I said you can play with these at home later if you want the demos are available now like all good things it's traditional to if you go to the code I've got for setting things up it's all done with command line stuff but as is traditional we have a web console for setting things up you can see you can see if you look at the logo it says nodes 3 that's saying how many nodes there are actually so being used for Kafka Kafka believes in redundancy so you would not normally run Kafka with fewer than three nodes that all have the same data share between them for reproducibility and you can see there this is giving me the URI I use to connect to Kafka and so on and so forth you'd expect to do introspection for anything you're doing if you're doing things like Kafka you will have web pages that show you information about them you'll set them up yourself if you can't find them out in the box so here we can see the actual names of our partitions of the topics I'm using for these demos and information about what's been happening with them, how many partitions each had so you can see that demo 3 for cotton chips has three partitions and they have one here is a bar chart showing that of those three partitions they've actually had the messages distributed between them reasonably reasonably evenly that means my thing that's making our orders and pretending to be customers is visiting the tills reasonably randomly which is good but this would allow you to see this partition is not actually doing anything what's gone wrong and here you can see this again is showing the consumer groups that demo 3 is using this is the sort of thing you'd normally set up when you're setting up your instrumentation anyway and we have lots of metrics for this is CPU usage which is not sure how useful that is to know but you can also find all the other standard things out any system or any complexity you'll want to be setting up monitoring because otherwise you have no idea what's going on and when something does go wrong that would lead you to where the cause might be so I like place now there are two reasons I order a place at fish and ship shops one is I really like place it's sweeter for earth fish the other is place is not normally sold often enough for it to be sitting in the hot cabinet so your place is freshly cooked makes me feel special so we need someone to cook we need to simulate the cook back to the very simple model where we've got one till, one food preparer just for simplicity but now we're going to introduce a cook so the idea is that when the food preparer gets an order with place in it they can't make it up immediately so they tell the cook using the cook topic there in the cloud to prepare the food, the cook cooks it and then they send it back into the order queue into the order topic again because it's now ready for the food preparer to give to the customer so there's a little loop there we aren't going to try and stimulate cooking or anything at this stage so here's an order that's got place in it the third entry there is place in chips our model works but actually I want a ready flag to say is this order ready to be given to the customer so in the code the food preparer so that's the person who got the order that's come from the till a method called is all order available so they take the order and if there's not a ready flag already in the order they're going to put one in there if if you get all the items in the thing using ittotools.chain if you don't know the ittotools module go and look it up, there's some fun things in there and then they say we'll set order ready it's their place not in the order if there's not place in the order the order is already we can get it out of the hot cabinet if there is place we're going to need to send it to the cook it's not ready although they return whether the order is ready or not so we've mutated the order bad thing to do in general but we've also returned the value of the ready flag the convenience so then in the second bit of code there it finds out whether the order is available and then it sends it to the cook topic and so it's sending it to the cook topic with the flag ready set to false we have a new cook who is a consumer and a producer so they consume from the cook topic they wait a bit to pretend to cook the thing that's the async.io.sleep and then they mark the order as ready which says I've cooked it it's ready to be served and then they send it back to the order's topic so now this came into the cook with the ready flag set false and they've sent it back to the order's topic with the flag set true and remember the consumer looks to see if the flag's there the food preparer so this is a demo of of that I'm afraid this is one of the places where staring at the code by yourself later is often easier to understand what I was just on about somewhere in here is a demo 3 yeah so we're back to I don't want demo 3 I want demo 4. My apologies this is why I couldn't find demo 3 because demo 4 was next up so we're back to a food preparer a till at the top left food preparer at the top right nothing at the bottom left it's just there to make the screen a nice size and the cook at the bottom so orders coming in that are ready have a tick mark and you'll see that occasionally something comes with the code and chips it then gets sent to the cook recieves it when it's ready it says order available and it goes back to the food preparer so what you see on the food preparer is things come in with that and then some of them come out of sequence so we got lots going on there but it's only the orders we're placing that have to go to the cook that's the last demo so I can stop playing with the cursor now so so far we've looked at modelling the order in serving of card and chips we've looked at having multiple producers and consumers and we've had a very simple order for how we do place so that's all the demos I'm doing now I've now got two homeworks for you if you want to go and play with the demo code and extend it so the first one is we should simulate that bit in the middle where the hot food is and the fact that the cook is doing stuff so obviously we would use a reddish cache for that because reddish is one of my favourite things postgres, squalite, reddish and kafker four of my favourite things so we'd create a reddish key store with entries of the hot cabinet so we'd have a key for cod and how many portions are ready a key for chip and how many portions are ready and a key for place that would initially start at nought all of them are nought, the cooks and stuff puts some cod and chips in so the preparer would compare the order it's just been given to the things in the so the counts in the hot cabinet cache if there's enough, decrement the cache pass the order on it's all done, if not send the order to the cook the cook, well wait a moment to pretend to be cooking stuff and then update the cache so for place they'd add as many places into the cache as we're asked for for cod and chips they'd round up to a reasonable number so that there's a reasonable number of orders there and then they send the order they've just been given back to the order topic and the food preparer would use that so the food preparer would say now I can make it up because I've got the order again I don't remember having it before but I've got the order now and now I can make it up so this would be a really quite simple model that just brings using Redis as well and I think would be quite fun to do I haven't written the code for this yet I've ran out of time homework 2 is perhaps more interesting with keeping how many orders we have because we've got an order number we're keeping which till things came from we obviously somewhere have a price list all this stuff is natural to put into a database so we can do analysis at the end of the day we can figure out how many ingredients we're consuming, how much do we need to buy how are we performing are we making a profit we need to put information into a database so one way to do that would be to create another consumer that writes to for instance Postgres we know how to do that but that means that my python client the one doing that little demo for us is now busy writing to Postgres as well and that doesn't scale if we actually had a truly mega efficient shit warehouse that wouldn't scale so the answer is we can do that back in the cloud there's a technology called Kafka Connect which allows you to create Kafka connectors that will talk to databases for you and you can either read from a database or other technology into Kafka or you can have a sync which goes from Kafka to the thing so I would do it by creating a database I'd use, I'd make sure the Kafka service had this technology enabled and then I'd set up a sync connector to send events to Postgres I have actually already written this code actually in two different ways and that's available in the GitHub repository and it's fun you can do it as well separately so finally we've done all the things done before but we've also talked briefly about modelling the hot cabinet which you can try later I will probably add a demo for that as well and we talked briefly about using Kafka Connect to share data all of these products have registered trademarks associated with them which I wish to acknowledge because they're nice people if you do want to use Ivan you can get an extra amount of credit for using Kafka there you get a month and $400 worth of credit that means you can actually pay with Kafka without having to set it up on your laptop we are hiring the QR code will take you to the GitHub repository thank you very much