 Hi, I am Jeremy Howard from fast.ai and this is a hackers guide to language models. When I say a hackers guide, what we're going to be looking at is a code-first approach to understanding how to use language models in practice. So before we get started, we should probably talk about what is a language model. I would say that this is going to make more sense if you know the kind of basics of deep learning. If you don't, I think you'll still get plenty out of it and there'll be plenty of things you can do. But if you do have a chance, I would recommend checking out cost.fast.ai, which is a free course and specifically if you could at least kind of watch, if not work through the first five lessons, that would get you to a point where you understand all the basic fundamentals of deep learning that will make this lesson tutorial make even more sense. Maybe I shouldn't call this a tutorial, it's more of a quick run through. So I'm going to try to run through all the basic ideas of language models, how to use them, both open source ones and open AI based ones. And it's all going to be based using code. As much as possible. So let's start by talking about what a language model is. And so as you might have heard before, a language model is something that knows how to predict the next word of a sentence or knows how to fill in the missing words of a sentence. And we can look at an example of one. Open AI has a language model text of enchi 003. And we can play with it by passing in some words and ask it to predict what the next words might be. So if we pass in when I arrived back at the Panda breeding facility after the extraordinary rain of live frogs, I couldn't believe what I saw. I just came up with it yesterday and I thought what might happen next. So kind of fun for creative brainstorming. There's a nice site called nat.dev. nat.dev lets us play with a variety of language models. And here I've selected text of enchi 003 and I'll hit submit. And it starts printing stuff up. The pandas were happily playing and eating the frogs that had fallen in the sky. There's an amazing site to see these animals taking advantage of such a unique opportunity. The staff took quick measures to ensure the safety of the pandas and the frogs. So there you go. That's what happened after the extraordinary rain of live frogs at the Panda breeding facility. You'll see here that I've enabled show probabilities, which is a thing in nat.dev where it shows. Well, let's take a look. It's pretty likely the next word here is going to be there. And after there, since we're talking about a panda breeding facility, it's going to be pandas were. And what were they doing? Well, they could have been doing a few things. They could have been doing something happily or the pandas were having, the pandas were out, the pandas were playing. So it picked the most likely. It thought it was 20% likely it's going to be happily. And what were they happily doing? Could have been playing, hopping, eating and so forth. So they're eating the frogs that and then had almost certainly. So you can see what it's doing at each point is it's predicting the probability of a variety of possible next words. And depending on how you set it up, it will either pick the most likely one every time, or you can change muck around with things like p values and temperatures to change what comes up. So and each time then it'll give us a different result. And this is kind of fun. Frogs perched on the heads of some of the pandas. It was an amazing sight, et cetera, et cetera. Okay. So that's what a language model does. Now you might notice here it hasn't predicted pandas. It's predicted panned and then separately us. Okay. After panned, it's going to be us. So it's not always a whole word. Here it's un and then harmed. Actually it's un, ha, mood. So you can see that it's not always predicting words, specifically what it's doing is predicting tokens. Tokens are either whole words or subword units, pieces of a word, or it could even be punctuation or numbers or so forth. So let's have a look at how that works. So for example, we can use the actual, it's called tokenization to create tokens from a string. We can use the same tokenizer that GPT users by using tick token. And we can specifically say we want to use the same tokenizer that that model text eventually 003 uses. And so for example, when I earlier tried this, it talked about the frog splashing. And so I thought, well, include they will encode, they are splashing. And the result is a bunch of numbers. And what those numbers are, they're basically just lookups into a vocabulary that open AI in this case created. And if you train your own models, you'll be automatically creating or your code will create. And if I then decode those, it says, oh, these numbers are they space are space spool hashing. And so put that all together, they are splashing. So you can see that the start of a word is the space before it is also being encoded here. So these language models are quite neat that they can work at all. But they're not of themselves really designed to do anything. Let me explain. The basic idea of what chat GPT GPT four, but etc. are doing comes from a paper, which describes an algorithm that I created back in 2017, called ULM fit. And Sebastian Ruder and I wrote a paper up describing the ULM fit approach, which was the one that basically laid out what everybody's doing, how this system works. And the system has three steps. Step one is language model training. But you'll see, this is actually from the paper, we actually described it as pre training. Now what language model pre training does is this is the thing which predicts the next word of a sentence. And so in the original ULM fit paper, so the algorithm I developed in 2017, then Sebastian Ruder and I wrote it up in 2018, early 2018. What I originally did was I trained this language model on Wikipedia. Now what that meant is I took a neural network and a neural network is just a function. If you don't know what it is, it's just a mathematical function that's extremely flexible. And it's got lots and lots of parameters that initially it can't do anything. But using stochastic gradient descent or SGD, you can teach it to do almost anything if you give it examples. And so I gave it lots of examples of sentences from Wikipedia. So for example, from the Wikipedia article for the birds, the birds is a 1963 American natural horror, horror, natural horror thriller film produced and directed by Alfred. And then it would stop. And so then the model would have to guess what the next word is. And if it guessed Hitchcock, it would be rewarded. And if it gets guessed something else, it would be penalized. And effectively, basically, it's trying to maximize those rewards. It's trying to find a set of weights for this function that makes it more likely that it would predict Hitchcock. And then later on in this article, it reads from Wikipedia, any previously dated Mitch, but ended it due to Mitch's cold, overbearing mother, Lydia, who dislikes any woman in Mitch's. They can see that filling this in actually requires being pretty thoughtful because there's a bunch of things that like kind of logically could go there. Like a woman could be in Mitch's closet, could be in Mitch's house. And so, you know, you could probably guess in the Wikipedia article describing the plot of the birds as actually any woman in Mitch's life. Now, to do a good job of solving this problem as well as possible of guessing the next word of sentences, the neural network is going to have to learn a lot of stuff about the world. It's going to learn that there are things called objects, that there's a thing called time, that objects react to each other over time, that there are things called movies, that movies have directors, that there are people, that people have names and so forth, and that a movie director is Alfred Hitchcock and he directed horror films and so on and so forth. It's going to have to learn an extraordinary amount if it's going to do a really good job of predicting the next word of sentences. Now, these neural networks specifically are deep neural networks. This is deep learning and in these deep neural networks, which have, when I created this, I think it had like 100 million parameters. Nowadays, they have billions of parameters. It's got the ability to create a rich hierarchy of abstractions and representations which it can build on. And so this is really the key idea behind neural networks and language models is that if it's going to do a good job of being able to predict the next word of any sentence in any situation, it's going to have to know an awful lot about the world. It's going to have to know about how to solve math questions or figure out the next move in a chess game or recognize poetry and so on and so forth. Now, nobody says it's going to do a good job of that. So it's a lot of work to find, to create and train a model that is good at that. But if you can create one that's good at that, it's going to have a lot of capabilities internally that it would have to be drawing on to be able to do this effectively. So the key idea here for me is that this is a form of compression and this idea of the relationship between compression and intelligence goes back many, many decades. And the basic idea is that yeah, if you can guess what words are coming up next, then effectively you're compressing all that information down into a neural network. Now I said this is not useful of itself. Well, why do we do it? Well, we do it because we want to pull out those capabilities. And the way we pull out those capabilities is we take two more steps. The second step is we do something called language model fine tuning. And in language model fine tuning, we are no longer just giving it all of Wikipedia or nowadays we don't just give it all of Wikipedia, but in fact, a large chunk of the internet is fed to pre-training these models. In the fine tuning stage, we feed it a set of documents a lot closer to the final task that we want the model to do. But it's still the same basic idea. It's still trying to predict the next word of a sentence. After that, we then do a final classifier fine tuning. And then the classifier fine tuning, this is the kind of end task we're trying to get it to do. Now nowadays, these two steps are very specific approaches are taken. For the step two, the step B, the language model fine tuning, people nowadays do a particular kind called instruction tuning. The idea is that the task we want most of the time to achieve is solve problems, answer questions. And so in the instruction tuning phase, we use data sets like this one. This is a great data set called Open Orca, created by a fantastic open source group. And it's built on top of something called the Flan Collection. And you can see that basically there's all kinds of different questions in here. So there's four gigabytes of questions and context and so forth. And each one generally has a question or an instruction or a request and then a response. Here are some examples of instructions. I think this is from the Flan data set, if I remember correctly. So for instance, it could be, does the sentence in the Iron Age answer the question, the period of time from 1200 to 1000 BCE is known as what? Choice is one, yes or no. And then the language model is meant to write one or two as appropriate for yes or no. Or it could be things about, I think this is from a music video, who is the girl in more than, you know, answer, and then it would have to write the correct name of the member model or dancer or whatever from that music video and so forth. So it's still doing language modeling. So fine tuning and pre-training are kind of the same thing. But this is more targeted now, not just to be able to fill in the missing parts of any document from the internet, but to fill in the words necessary to answer questions, to do useful things. Okay, so that's instruction tuning. And then step three, which is the classifier fine tuning. Nowadays, there's generally various approaches such as reinforcement learning from human feedback and others, which are basically giving humans or sometimes more advanced models, multiple answers to a question such as, here are some from a reinforcement learning from human feedback paper, I can't remember which one I got it from, list five ideas for how to regain enthusiasm for my career. And so the model will spit out two possible answers or it'll have a less good model and more good model, and then a human or a better model will pick, which is best. And so that's used for the final fine tuning stage. So all of that is to say, although you can download pure language models from the internet, they're not generally that useful of their on their own until you've fine tuned them. Now you don't necessarily need step C nowadays, actually people are discovering that maybe just step B might be enough, it's still a bit controversial. Okay, so when we talk about a language model where we could be talking about something that's just been pre-trained, something that's been fine tuned, or something that's gone through something like RLHF all of those things are generally described nowadays as language models. So my view is that if you are going to be good at language modeling in any way, then you need to start by being a really effective user of language models. And to be a really effective user of language models, you got to use the best one that there is. And currently, so what are we up to September 2023? The best one is by far GPT-4. This might change in some time in the not too distant future, but this is right now GPT-4 is the recommendation strong, strong recommendation. Now you can use GPT-4 by paying 20 bucks a month to open AI, and then you can use it a whole lot. It's very hard to run out of credits, I find. Now, what can GPT-2? It's interesting and instructive, in my opinion, to start with the very common views you see on the internet or even in academia about what it can't do. So for example, there was this paper you might have seen GPT-4 can't reason, which describes a number of empirical analysis done of 25 diverse reasoning problems and found that it was not able to solve them. It's utterly incapable of reasoning. So I always find you got to be a bit careful about reading stuff like this, because I just took the first three that I came across in that paper and I gave them to GPT-4. And by the way, something very useful in GPT-4 is you can click on the share button and you'll get something that looks like this, and this is really handy. So here's an example of something from the paper that said GPT-4 can't do this. Mabel's heart rate at 9 a.m. was 75 beats per minute. Her blood pressure at 7 p.m. was 120 over 80. She died at 11 p.m. Was she alive at noon? It's of course very human, we know obviously she must be. And GPT-4 says, hmm, this appears to be a riddle, not a real inquiry into medical conditions. Here's a summary of the information and yeah, it sounds like Mabel was alive at noon. So that's correct. This was the second one I tried from the paper that says GPT-4 can't do this, and I found actually GPT-4 can do this. And it said that GPT-4 can't do this, and I found GPT-4 can do this. Now, I mentioned this to say GPT-4 is probably a lot better than you would expect if you've read all this stuff on the internet about all the dumb things that it does. Almost every time I see on the internet saying something that GPT-4 can't do, check it, and it turns out it does. This one was just last week. Sally, a girl, has three brothers. Each brother has two sisters. How many sisters does Sally have? So have to think about it. And so GPT-4 says, okay, Sally's count as one sister by each of her brothers. If each brother has two sisters, that means there's another sister in the picture apart from Sally. So Sally has one sister. Okay, correct. And then this one I got sort of like three or four days ago. This is a common view that language models can't track things like this. Here's the riddle. I'm in my house. On top of my chair in the living room is a coffee cup. Inside the coffee cup is a thimble. Inside the thimble is a diamond. I move the chair to the bedroom. I put the coffee cup in the bed. I turn the cup upside down. Then I return it upside up. Place the coffee cup on the counter in the kitchen. Where's my diamond? And so GPT-4 says, yeah, okay, you turned it upside down. So probably the diamond fell out. So therefore the diamond in the bedroom where it fell out. Okay, correct. Why is it that people are claiming that GPT-4 can't do these things when it can? Well, the reason is because I think on the whole they are not aware of how GPT-4 was trained. GPT-4 was not trained at any point to give correct answers. GPT-4 was trained initially to give most likely next words. And there's an awful lot of stuff on the internet where documents are not describing things that are true. There could be fiction. There could be jokes. There could be just GPT people saying dumb stuff. So this first stage does not necessarily give you correct answers. The second stage with the instruction tuning also like it's trying to give correct answers. But part of the problem is that then in the stage where you start asking people, which answer do they like better? People tended to say in these things that they prefer more confident answers and they often were not people who were trained well enough to recognize wrong answers. So there's lots of reasons that the SGD weight updates from this process for stuff like GPT-4 don't particularly or don't entirely reward correct answers. But you can help it want to give you correct answers. If you think about the LM pre-training, what are the kinds of things in a document that would suggest, oh, this is going to be high quality information. And so you can actually prime GPT-4 to give you high quality information by giving it custom instructions. And what this does is this is basically text that is prepended to all of your queries. And so you say like, oh, you're brilliant at reasoning. So like, okay, that's obviously your primary to give good answers. And then try to work against the fact that the RLHF folks preferred confidence, just tell it. No, tell me if there might not be a correct answer. Also, the way that the text is generated is it literally generates the next word. And then it puts all that whole lot back into the model and generates the next next word puts that all back in the model generates the next, next, next word and so forth. That means the more words it generates, the more computation it can do. And so I literally, I tell it that. Right. And so I say first, spend a few sentences explaining background context, etc. So this custom instruction allows it to solve more challenging problems. And you can see the difference. Here's what it looks like. For example, if I say, how do I get a count of rows grouped by value in pandas? And it just gives me a whole lot of information, which is actually it thinking. So I just skip over it. And then it gives me the answer. And actually in my custom instructions, I actually say if the request begins with vv, actually make it as concise as possible. And so it kind of goes into brief mode. And here it's brief mode. How do I get the group? This is the same thing, but with the vv at the start. And it just spits it out. Now, in this case, it's a really simple question. So I didn't need time to think. So hopefully that gives you a sense of how to get language models to give good answers. You have to help them. And if it's not working, it might be user error, basically. But having said that, there's plenty of stuff that language models like GPT4 can't do. One thing to think carefully about is, does it know about itself? Can you ask it? What is your context length? How were you trained? What transformer architecture are you based on? At any one of these stages, did it have the opportunity to learn any of those things? Well, obviously not at the pre-training stage. Nothing on the internet existed during GPT4's training, saying how GPT4 was trained. Probably Ditto in the instruction tuning, probably Ditto in the RLHF. So in general, you can't ask, for example, a language model about itself. Now, again, because of the RLHF, it'll want to make you happy by giving you opinionated answers. So it'll just spit out the most likely thing it thinks with great confidence. This is just a general kind of hallucination. So hallucinations is just this idea that the language model wants to complete the sentence and it wants to do it in an opinionated way that's likely to make people happy. It doesn't know anything about URLs. It really hasn't seen many at all. I think a lot of them, if not all of them, you pretty much were stripped out. So if you ask it anything about like what's at this webpage, again, it'll generally just make it up. And it doesn't know, at least, GPT4 doesn't know anything after September 2021 because the information it was pre-trained on was from that time period, September 2021 and before, called the knowledge cutoff. So here's some things it can't do. Steve Newman sent me this good example of something that it can't do. Here is a logic puzzle. I need to carry a cabbage, a goat, and a wolf across a river. I can only carry one item at a time. I can't leave the goat with a cabbage. I can't leave the cabbage with a wolf. How do I get everything across to the other side? Now the problem is this looks a lot like something called the classic river crossing puzzle. So classic, in fact, that it has a whole Wikipedia page about it. And in the classic puzzle, the wolf would eat the goat or the goat would eat the cabbage. Now, in Steve's version, he changed it. The goat would eat the cabbage and the wolf would eat the cabbage, but the wolf won't eat the goat. So what happens? Well, very interestingly, GPT4 here is entirely overwhelmed by the language model training. It's seen this puzzle so many times, it knows what word comes next. So it says, oh yeah, I take the goat across the road, across the river, and leave it on the other side, leaving the wolf with a cabbage, but we're just told you can't leave the wolf with a cabbage. So it gets it wrong. Now the thing is though, you can encourage GPT4 or any of these language models to try again. So during the instruction tuning and RLHF, they're actually fine-tuned with multi-stage conversations. So you can give it a multi-stage conversation. Repeat back to me the constraints I listed. What happened after step one is a constraint violated? Oh yeah, yeah, yeah. I made a mistake. Okay. My new attempt, instead of taking the goat across the river and leaving it on the other side, is I'll take the goat across the river and leave it on the other side. It's done the same thing. Oh yeah, I did do the same thing. Okay, I'll take the wolf across. Well, now the goat's with a cabbage. That still doesn't work. Oh yeah, that didn't work either. Sorry about that. Instead of taking the goat across the other side, I'll take the goat across the other side. Okay, what's going on here? Right? This is terrible. Well, one of the problems here is that not only is on the internet, it's so common to see this particular goat puzzle that it's so confident it knows what the next word is. Also, on the internet, when you see stuff which is stupid on a web page, it's really likely to be followed up with more stuff that is stupid. Once GPT-4 starts being wrong, it tends to be more and more wrong. It's very hard to turn it around to start it making it be right. So you actually have to go back and there's actually an edit button on these chats. And so what you generally want to do is if it's made a mistake, is don't say, oh, here's more information to help you fix it, but instead go back and click the edit and change it here. And so this time, it's not going to get confused. So in this case, actually fixing Steve's example takes quite a lot of effort, but I think I managed to get it to work eventually. And I actually said, oh, sometimes people read things too quickly. They don't notice things, it can trick them up. Then they apply some pattern, get the wrong answer. You do the same thing, by the way. So I'm going to trick you. So before you're about to get tricked, make sure you don't get tricked. Here's the tricky puzzle. And then also with my custom instructions, it takes time discussing it. And this time it gets it correct, it takes the cabbage across first. So it took a lot of effort to get to a point where it could actually solve this because yeah, when it's, you know, for things where it's been primed to answer a certain way again and again and again, it's very hard for it to not do that. Okay, now something else super helpful that you can use is what they call advanced data analysis. In advanced data analysis, you can ask it to basically write code for you. And we're going to look at how to implement this from scratch ourselves quite soon. But first of all, let's learn how to use it. So I was trying to build something that split into markdown headings, a document on third level markdown headings. So that's three hashes at the start of a line. And I was doing it on the whole of Wikipedia. So using regular expressions was really slow. So I said, Oh, I want to speed this up. And it said, Okay, here's some code, which is great, because then I can say, Okay, test it and include edge cases. And so it then puts in the code, creates extra cases, tests it, and says, Yep, it's working. However, I discovered it's not. I noticed it's actually removing the carriage return at the end of each sentence. So I said, Oh, fix that and update your tests. So it said, Okay. So now it's changed the test update the test cases, let's run them. And oh, it's not working. So it says, Oh, yeah, fix the issue in the test cases. Nope, it didn't work. And you can see it's quite clever, the way it's trying to fix it by looking at the results. And but as you can see, it's not. Every one of these is another attempt, another attempt, another attempt until eventually I gave up waiting. It's so funny each time it's like debunking again. Okay, this time, I got to handle it properly. And I gave up at the point where it's like, Oh, one more attempt. So it didn't solve it. Interestingly enough. And you know, I again, it's, it's, there's some limits to the amount of kind of logic that it can do. This is really a very simple question. I asked it to do for me. And so hopefully you can see, you can't expect even GPT for code interpreter or advanced data analysis is now called to make it so you don't have to write code anymore. You know, it's not a substitute for having programmers. So, but it can, you know, it can often do a lot as I'll show you in a moment. So for example, actually, OCR, like this is something I thought was really cool, you can just paste sorry, paste to upload. So GPT for you can upload an image, advanced data analysis, yeah, you can upload an image here. And then I wanted to basically grab some text out of an image, somebody had got a screenshot with their screen. And I wanted to, which was something saying, Oh, this language model can't do this. And I wanted to try it as well. So rather than retyping it, I just uploaded that image, my screenshot. And so can you extract the text from this image? And it said, Oh, yeah, I could do that. I could use OCR. And like, so it literally wrote an OCR script. And there it is. Just took a few seconds. So the difference here is it didn't really require to think of much logic. It could just use a very, very familiar pattern that it would have seen many times. So this is generally where I find language models excel is where it doesn't have to think too far outside the box. I mean, it's great on kind of creativity tasks, but for like reasoning and logic tasks that are outside the box, I find it not great. But yeah, it's great at doing code for a whole wide variety of different libraries and languages. Having said that, by the way, Google also has a language model quad bar. It's way less good than GPT for most of the time. But there is a nice thing that you can literally paste an image straight into the prompt. And I just typed OCR this. And it didn't even have to go through code interpreter or whatever. It just said, Oh, sure, I've done it. And there's the result of the OCR. And then even commented on what it just does yard, which I thought was cute. And oh, even more interestingly, it even figured out where the OCR text came from and gave me a link to it. So I thought that was pretty cool. Okay, so there's an example of it doing well. I'll show you one for this talk I found really helpful. I wanted to show you guys how much it costs to use the open AI API. But unfortunately, when I went to the open AI webpage, it was like all over the place, the pricing information was on all separate tables and it was kind of a bit of a mess. So I wanted to create a table with all of the information combined like this. And here's how I did it. I went to the open AI page, I hit Apple A to select all. And then I said in chat GPT create a table with the pricing information rows, no summarization, no information not in this page, every row should appear as a separate row in your output. And I hit paste. Now that was not very helpful to it because hitting paste, it's got the nav bar, it's got lots of extra information at the bottom, it's got all of its footer, etc. But it's really good at this stuff. It did it first time. So there was the markdown table. So I copied and pasted that into Jupyter and I got my markdown table. And so now you can see at a glance the cost of GPT4, 3.5, etc. But then what I really wanted to do was show you that as a picture. So I just said, oh, chart the input row from this table and just pasted the table back. And it did. So that's pretty amazing. Now, so let's talk about this pricing. So so far we've used chat GPT, which costs 20 bucks a month. And there's no like per token cost or anything. But if you want to use the API from Python or whatever, you have to pay per token, which is approximately per word. Maybe it's about one and a third tokens per word on average. Unfortunately, in the chart, it did not include these headers GPT4 GPT3.5. So these first two ones are GPT4. And these two are GPT3.5. So you can see the GPT3.5 is way, way cheaper. And you can see it here, it's 0.03 versus 0.0015. So it's so cheap, you can really play around with it and not worry. And I want to give you a sense of what that looks like. Okay, so why would you use the OpenAI API rather than chat GPT? Because you can do it programmatically. So you can, you know, you can analyze data sets, you can do repetitive stuff. It's kind of like a different way of programming, you know, it's things that you can think of describing. But let's just look at the most simple example of what that looks like. So if you pip install OpenAI, then you can import chat, chat completion. And then you can say, okay, chat completion.create using GPT3.5 turbo. And then you can pass in a system message. This is basically the same as custom instructions. So okay, you're an Aussie LLM that uses Aussie slang and analogies wherever possible. Okay, and so you can see I'm passing in an array here of messages. So the first is the system message. And then the user message, which is what is money? Okay, so GPT3.5 returns a big embedded dictionary. And the message content is, well, my money is like the oil that keeps the machinery of our economy running smoothly. There you go. Just like Aquila loves its eucalyptus leaves, we humans can't survive without this stuff. So there's the Aussie LLM's view of what is money. So the really the main ones I pretty much always use are GPT4 and GPT3.5. GPT4 is just so, so much better at a thing remotely challenging. But obviously it's much more expensive. So rule of thumb, you know, maybe try 3.5 turbo first. See how it goes. If you're happy with the results, then great. If you're not pony out for the more expensive one. Okay, so I just created a little function here called response that will print out this nested thing. And so now, oh, and so then the other thing to point out here is that the result of this also has a usage field, which contains how many tokens was it? So it's about 150 tokens. So at $0.002 per thousand tokens for 150 tokens means we just paid 0.03 cents, 0.0003 dollars to get that done. So as you can see, the cost is insignificant. If we were using GPT4, it would be 0.03 per thousand. So it would be half a cent. So unless you're doing many thousands of GPT4, you're not going to be even up into the dollars and GPT3.5 even more than that. But you know, keep an eye on it. OpenAI has a usage page and you can track your usage. Now, what happens when we are, this is really important to understand, when we have a follow up in the same conversation, how does that work? So we just asked what goat means. So for example, Michael Jordan is often referred to as the goat for his exceptional skills and accomplishments. And Elvis and the Beatles referred to as goat due to their profound influence and achievement. So I could say, what profound influence and achievements are you referring to? Okay, well, I meant Elvis, Presley and the Beatles did all these things. Now, how does that work? How does this follow up work? Well, what happens is the entire conversation is passed back. And so we can actually do that here. So here is the same system prompt. Here is the same question. Right. And then the answer comes back with role assistant. And I'm going to do something pretty cheeky. I'm going to pretend that it didn't say money is like oil. I'm going to say, oh, you actually said money is like kangaroos. I thought, what is going to do? Okay, so you can like literally invent a conversation in which the language model said something different. Because this is actually how it's done in a multi-stage conversation. There's no state, right? There's nothing stored on the server. You're passing back the entire conversation again and telling it what it told you. Right. So I'm going to tell it. It told me that money is like kangaroos. And then I'll ask the user, oh, really? In what way? And it's kind of cool because you can like see how it convinces you of something I just invented. Oh, let me break it down for you, Colbert. Just like kangaroos hop around and carry their joeys in their pouch, money is a means of carrying value around. So there you go. It's a make your own analogy. Cool. So I'll create a little function here that just puts these things together for us. System message, if there is one, the user message and returns the completion. And so now we can ask it, what's the meaning of life? Passing in the Aussie system prompt. The meaning of life is like trying to catch a wave on a sunny day at Bondi Beach. Okay. There you go. So what do you need to be aware of? Well, as I said, one thing is keep an eye on your usage. If you're doing it, you know, hundreds or thousands of times in a loop, keep an eye on not spending too much money. But also if you're doing it too fast, particularly the first day or two, you've got an account, you're likely to hit the limits for the API. And so the limits initially are pretty low. As you can see, three requests per minute. So that's for free users, page users, first 48 hours. And after that, it starts going up. And you can always ask for more. I just mentioned this because you're going to want to have a function that keeps an eye on that. And so what I did is I actually just went to Bing, which has a somewhat crappy version of GPT-4 nowadays, but it can still do basic stuff for free. And I said, please show me Python code to call the OpenAI API and handle rate limits. And it wrote this code. It's got to try, checks the rate limit errors, grabs the retry after, sleeps for that long and calls itself. And so now we can use that to ask, for example, what's the world's funniest joke? There we go. Here's the world's funniest joke. So that's like the basic stuff you need to get started using the OpenAI LLMs. And yeah, it's definitely suggest spending plenty of time with that so that you feel like you're really a LLM using expert. So what else can we do? Well, let's create our own code interpreter that runs inside GPT-4. And so to do this, we're going to take advantage of a really nifty thing called function calling, which is provided by the OpenAI API. And in function calling, when we call our askGPT function, which is this little one here, we had to pass in some keyword arguments that would be just passed along to check completion.create. And one of those keyword arguments you can pass is functions. What on earth is that? Functions tells OpenAI about tools that you have, about functions that you have. So for example, I created a really simple function called sums and it adds two things. In fact, it adds two ints. And I am going to pass that function to check completion.create. Now, you can't pass a Python function directly. You actually have to pass what's called the JSON schema. So you have to pass the schema for the function. So I created this nifty little function that you're welcome to borrow, which uses pydantic and also Python's inspect module to automatically take a Python function and return the schema for it. And so this is actually what's going to get passed to OpenAI. So it's going to know that there's a function called sums, it's going to know what it does. And it's going to know what parameters it takes, what the defaults are, and what's required. So this is like when I first heard about this, I found this a bit mind-bending because this is so different to how we normally program computers. The key thing for programming the computer here actually is the doc string. This is the thing that GPT-4 will look at and say, oh, what does this function do? So it's critical that this describes exactly what the function does. And so if I then say what is 6 plus 3, right, and I just, I probably wanted to make sure it actually did it here. So I gave it lots of prompts to say, because obviously it knows how to do it itself without calling sums. So it'll only use your functions if it feels it needs to, which is a weird concept. I mean, I guess fields is not a great word to use, but you kind of have to anthropomorphise these things a little bit because they don't behave like normal computer programs. So if I ask GPT what is 6 plus 3 and tell it that there's a function called sums, then it does not actually return the number 9. Instead, it returns something saying, please call a function. Call this function and pass it these arguments. So if I print it out, there's the arguments. So I created a little function called core function, and it goes into the result of open AI, grabs the function call, checks that the name is something that it's allowed to do, grabs it from the global system table, and calls it passing in the parameters. And so if I now say, okay, call the function that we got back, we finally get 9. So this is a very simple example. It's not really doing anything that useful, but what we could do now is we can create a much more powerful function called Python. And the Python function executes code using Python and returns the result. Now, of course, I didn't want my computer to run arbitrary Python code that GPT4 told it to without checking. So I just got it to check first. So say, oh, you're sure you want to do this. So now I can say, ask GPT, what is 12 factorial system prompt, you can use Python for any required computations and say, okay, here's a function you've got available, it's the Python function. So if I now call this, it will pass me back again, a completion object. And here it's going to say, okay, I want you to call Python, passing in this argument. And when I do, it's going to go import math result equals blah, and then return result. Do I want to do that? Yes, I do. And there it is. Now, there's one more step, which we can optionally do. I mean, we've got the answer we wanted, but often we want the answer in more of a chat format. And so the way to do that is to, again, repeat everything that you've passed into so far. But then instead of adding an assistant role response, we have to provide a function role response and simply put in here the result we got back from the function. And if we do that, we now get the pros response 12 factorial is equal to 471 million 1600. Now, functions like Python, you can still ask it about non Python things. And it just ignores it if you don't need it, right? So you can have a whole bunch of functions available that you've built to do whatever you need for the stuff which the language model isn't familiar with. And it'll still solve whatever it can on its own and use your tools use your functions where possible. Okay, so we have built our own code interpreter from scratch. I think that's pretty amazing. So that is what you can do with or some of the stuff you can do with open AI. What about stuff that you can do on your own computer? Well, to use a language model on your own computer, you're going to need to use a GPU. So I guess the first thing to think about is like, do you want this? Does it make sense to do stuff on your own computer? What are the benefits? There are not any open source models that are as good yet as GPT for. And I would have to say also like actually open AI pricing is really pretty good. So it's it's not immediately obvious that you definitely want to kind of go in-house, but there's lots of reasons you might want to and we'll look at some examples of them today. One example you might want to go in-house is that you want to be able to ask questions about your proprietary documents or about information after September 2021, the knowledge cutoff, or you might want to create your own model that's particularly good at solving the kinds of problems that you need to solve using fine tuning. And these are all things that you absolutely can get better than GPT for performance at work or at home without too much money or travel. So these are the situations in which you might want to go down this path. And so you don't necessarily have to buy a GPU on Kaggle. They will give you a notebook with two quite old GPUs attached and very little RAM, but it's something. Or you can use Colab and on Colab you can get much better GPUs than Kaggle has and more RAM, particularly if you pay a monthly subscription fee. So those are some options for free or low cost. You can also of course go to one of the many GPU server providers and they change all the time as to what's good or what's not. RunPod is one example and you can see if you want the biggest and best machine, you're talking $34 an hour, so it gets pretty expensive, but you can certainly get things a lot cheaper, $0.80 an hour. Lambda Labs is often pretty good. It's really hard at the moment to actually find people that have them available, so they've got lots listed here, but they often have none or very few available. There's also something pretty interesting called Vast AI which basically lets you use other people's computers when they're not using them. And as you can see, they tend to be much cheaper than other folks. And then they tend to have better availability as well, but of course for sensitive stuff you don't want to be running it on some Rando's computer. So anyway, so there's a few options for renting stuff. You know, I think if you can, it's worth buying something and definitely the one to buy at the moment is the GTX 3090 used. You can generally get them from eBay for like $700 or so. A 4090 isn't really better for language models even though it's a newer GPU. The reason for that is that language models are all about memory speed. How quickly can you get in and stuff in and out of memory rather than how fast is the processor? And that hasn't really improved a whole lot. So the $2,000 bucks. The other thing as well as memory speed is memory size, 24 gigs, it doesn't quite cut it for a lot of things, so you'd probably want to get two of these GPUs. So you're talking like $1,500 or so. Or you can get a 48 gigram GPU, it's called an A6000, but this is going to cost you more like $5,000. So again, getting two of these is going to be a better deal and this is not going to be faster than these either. Or funnily enough, you could just get a Mac with a lot of RAM, particularly if you get an M2 Ultra. Macs have, particularly the M2 Ultra has pretty fast memory. It's still going to be way slower than using an Nvidia card, but it's going to be like you're going to be able to get, you know, like I think 192 gig or something. So it's not a terrible option, particularly if you're not training models, you just want to use other existing trained models. So anyway, most people who do this stuff seriously, almost everybody has Nvidia cards. So then what we're going to be using is a library called Transformers from Hugging Face. And the reason for that is that basically people upload lots of pre-trained models or fine-trained models up to the Hugging Face Hub. And in fact, there's even a leaderboard where you can see which are the best models. Now, this is a really fraught area. So at the moment, this one is meant to be the best model. It has the highest average score. And maybe it is good. I haven't actually used a particular model. Or maybe it's not. I actually have no idea because the problem is these metrics are not particularly well aligned with real life usage for all kinds of reasons. And also sometimes you get something called leakage, which means that sometimes some of the questions from these things actually leaks through to some of the training sets. So you can get as a rule of thumb what to use from here, but you should always try things. And you can also say, you know, these ones are all the 70B here that tells you how big it is. So this is a 70B parameter model. So generally speaking for the kinds of GPUs we're talking about, you'll be wanting no bigger than 13B and quite often 7B. So let's see if we can find here's a 13B model, for example. All right. So you can find models to try out from things like this leaderboard. And there's also a really great leaderboard called Fasteval, which I like a lot, because it focuses on some more sophisticated evaluation methods such as this chain of thought evaluation method. So I kind of trust these a little bit more. And these are also GSM 8K is a difficult math benchmark, big bench hard, so forth. So yeah, so, you know, Stable Bluga 2, WizardMeth 13B, Dolphin Lama 13B, et cetera, these would all be good options. Yeah. So you need to pick a model. And at the moment, nearly all the good models are based on Meta's Lama 2. So when I say based on what does that mean? Well, what that means is this model here Lama 2 7B. So it's a Lama model. That's just the name I met a call it. This is their version 2 of Lama. This is their 7 billion size one. It's the smallest one that they make. And specifically these weights have been created for hugging face. So you can load it with the hugging face transformers. And this model has only got as far as here. It's done the language model for pre-training. It's done none of the instruction tuning and none of the RLHF. So we would need to fine tune it to really get it to do much useful. So we can just say, okay, create a, automatically create the appropriate model for language model. So causal LM is basically refers to that ULM fifth stage one process or stage two, in fact. So get the pre-trained model from this name, Meta Lama Lama 2 blah, blah, blah. Okay. Now, generally speaking, we use 16 bit floating point numbers nowadays. But if you think about it, 16 bit is two bytes. So 7B times two, it's going to be 14 gigabytes, just to load in the weights. So you're going to have a decent model to be able to do that. Perhaps surprisingly, you can actually just cast it to 8 bit and it still works pretty well, thanks to something called discretization. So let's try that. So remember, this is just a language model, it can only complete sentences. We can't ask it a question and expect a great answer. So let's just give it the start of a sentence, Jeremy, how it is. And so we need the right tokenizer. So this will automatically create the right kind of tokenizer for this model. We can grab the tokens as PyTorch. Here they are. And just to confirm, if we decode them back again, we get the original plus a special token to say this is the start of a document. And so we can now call generate. So generate will auto-regressively, so call the model again and again, passing its previous result back as the next input. And I'm just going to do that 15 times. So this is, you can write this for loop yourself. This isn't doing anything fancy. In fact, I would recommend writing this yourself to make sure that you know how, that it all works okay. We have to put those tokens on the GPU. And at the end, I recommend putting them back onto the CPU, the result. And here are the tokens. Not very interesting. So we have to decode them using the tokenizer. And so the first 25, sorry, first 15 tokens are Jeremy, how it is, 28-year-old Australian AI researcher and entrepreneur. Okay, well, 28 years old is not exactly correct, but we'll call it close enough. I like that. Thank you very much. Llama 7B. So, okay, so we've got a language model completing sentences. It took one and a third seconds. And that's a bit slower than it could be because we used 8-bit. If we use 16-bit, there's a special thing called B Float 16, which is a really great 16-bit floating point format that's usable on any somewhat recent Nvidia GPU. Now, if we use it, it's going to take twice as much RAM as we discussed. But look at the time. It's come down to 390 milliseconds. Now, there is a better option still than even that. There's a different kind of discretization called GPTQ, where a model is carefully optimized to work with four or eight or other, you know, lower precision data automatically. And this particular person known as the bloke is fantastic at taking popular models, running that optimization process, and then uploading the results back to HackingFace. So, we can use this GPTQ version. And internally, this is actually going to use, I'm not sure exactly how many bits this particular one is, I think it's probably going to be four bits, but it's going to be much more optimized. And so look at this, 270 milliseconds, it's actually faster than 16-bit. Even though internally, it's actually casting it up to 16-bit each layer to do it. And that's because there's a lot less memory moving around. And to confirm, in fact, what we can even do now is we go up to 13B, easy. And in fact, it's still faster than the 7B now that we're using the GPTQ version. So this is a really helpful tip. So let's put all those things together, the tokenizer, the generate, the batch decode, we'll call this gen for generate. And so we can now use the 13B GPTQ model. And let's try this. Jeremy Howard is a, so it's got to 50 tokens so fast, 16-year veteran of Silicon Valley, co-founder of Kaggle, a market-closer predictive model. His company Kaggle.com has become to do the science competitions, what I don't know what I was going to say. But anyway, it's on the right track. I was actually there for 10 years, not 16, but that's all right. Okay, so this is looking good. But probably a lot of the time we're going to be interested in, you know, asking questions or using instructions. So Stability AI has this nice series called Stable Beluga, including a small 7B one and other bigger ones. And these are all based on Lama 2, but these have been instruction tuned. They might even have been RLHDF, I can't remember now. So we can create a stable Beluga model. And now something really important that I keep forgetting, everybody keeps forgetting, is during the instruction tuning process, during the instruction tuning process, the instructions that are passed in actually don't just appear like this. They actually always are in a particular format. And the format, believe it or not, changes quite a bit from fine-tuned to fine-tuned. And so you have to go to the webpage for the model and scroll down to find out what the prompt format is. So here's the prompt format. So I generally just copy it and then I paste it into Python, which I did here, and created a function called make prompt that used the exact same format that it said to you use. And so now if I want to say who is Jeremy Howard, I can call Jen again. That was that function I created up here, and make the correct prompt from that question. And then it returns back. Okay, so you can see here, all this prefix, this is a system instruction. This is my question. And then the assistant says, Jeremy Howard is an Australian entrepreneur, computer scientist, co-founder of machine learning and deep learning company, faster AI. Okay, so this one's actually all correct. So it's getting better by using an actual instruction tune model. And so we could then start to scale up. So we could use a 13b and in fact, we looked briefly at these open orca dataset earlier. So llama2 has been fine-tuned on open orca, and then also fine-tuned on another really great dataset called platypus. And so the whole thing together is the open orca platypus. And then this is going to be the bigger 13b. GPTQ means it's going to be quantized. So that's got a different format, okay, a different prompt format. So again, we can scroll down and see what the prompt format is. There it is. Okay. And so we can create a function called make open orca prompt that has that prompt format. And so now we can say, okay, who is Jeremy Howard? And now I've become British, which is kind of true. I was born in England, but I moved to Australia. Professional poker player. Definitely not that. Co-founding several companies, including Fast.ai, also Kaggle. Okay, so not bad. It was acquired by Google with 2017. Probably something around there. Okay. So you can see we've got our own models giving us some pretty good information. How do we make it even better? Because it's still hallucinating. Lama 2, I think, has been trained with more up-to-date information than GPT4. It doesn't have the September 2021 cutoff. But it still got a knowledge cutoff. We would like to be able to use the most up-to-date information. We want to use the right information to answer these questions as well as possible. So to do this, we can use something called retrieval augmented generation. So what happens with retrieval augmented generation is when we take the question we've been asked, like, who is Jeremy Howard? And then we say, okay, let's try and search for documents that may help us answer that question. So obviously, we would expect, for example, Wikipedia to be useful. And then what we do is we say, okay, with that information, let's now see if we can tell the language model about what we found and then have it answer the question. So let me show you. So let's actually grab a Wikipedia Python package. We will scrape Wikipedia grabbing the Jeremy Howard web page. And so here's the start of the Jeremy Howard Wikipedia page. It has 613 words. Now generally speaking, these open source models will have a context length of about 2000 or 4000. So the context length is how many tokens can it handle? So that's fine. It'll be able to handle this web page. And what we're going to do is we're going to ask it the question. So we're going to have here question and with a question. But before that, we're going to say answer the question with the help of the context. We're going to provide this to the language model. And we're going to say context and they're going to have the whole web page. So suddenly now our question is going to be a lot bigger. They're prompt. Right. So our prompt now contains the entire web page, the whole Wikipedia page followed by a question. And so now it says Jeremy Howard is an Australian data scientist, entrepreneur and educator, known for his work in deep learning, co-founder of fast AI, teaches courses, develops software, conducts research, used to be, yeah, okay, it's perfect. Right. So it's actually done a really good job. Like if somebody asked me to send them a, you know, 100 word bio, that would actually probably be better than I would have written myself. And just so even though I asked for 300 tokens, it actually got sent back the end of stream token. And so it knows to stop at this point. Well, that's all very well. But how do we know to pass in the Jeremy Howard Wikipedia page? Well, the way we know which Wikipedia page to pass in is that we can use another model to tell us which web page or which document is the most useful for answering a question. And the way we do that is we can use something called sentence transformer. And we can use a special kind of model that's specifically designed to take a document and turn it into a bunch of activations where two documents that are similar will have similar activations. So let me just let me show you what I mean. What I'm going to do is I'm going to grab just the first paragraph of my Wikipedia page. And I'm going to grab the first paragraph of Tony Blair's Wikipedia page. Okay, so we're pretty different people. Right. This is just like a really simple, small example. And I'm going to then call this model, I'm going to say encode, and I'm going to encode my Wikipedia first paragraph Tony Blair's first paragraph and the question which was who is Jeremy Howard. And it's going to pass back a 384 long vector of embeddings for the question for me and for Tony Blair. And what I can now do is I can calculate the similarity between the question and the Jeremy Howard Wikipedia page. And I can also do it for the question versus the Tony Blair Wikipedia page. And as you can see, it's higher for me. And so that tells you that if you're trying to figure out what document to use to help you answer this question, better off using the Jeremy Howard Wikipedia page than the Tony Blair Wikipedia page. So if you had a few hundred documents you were thinking of using to give back to the model as context to help it answer a question, you could literally just pass them all through to encode, go through each one at a time and see which is closest. When you've got thousands or millions of documents, you can use something called a vector database where basically as a one off thing, you go through and you encode all of your documents. And so in fact, there's there's lots of prebuilt systems for this. Here's an example of one called H2O GPT. And this is just something that I've got that I've got running here on my computer. It's just an open source thing written in Python sitting here running on port 7860. And so I just gone to localhost 7860. And what I did was I just uploaded, I just clicked upload and uploaded a bunch of papers. In fact, I might be able to see it better. Yeah, here we go. A bunch of papers. And so, you know, we could look at research. Yeah, I can. So for example, we can look at the ULM fit paper that said Bruder and I did. And you can see it's taken the PDF and turned it into slightly crappily, a text format. And then it's created an embedding for each, you know, each section. So I could then ask it, you know, what is ULM fit? And I'll hit enter. And you can see here it's now actually same based on the information provided in the context. So it's showing us it's been given some context. What context did it get? So here are the things that it found, right? So it's being sent this context. So this is kind of citations. Goal of ULM fit proves a performance by leveraging the knowledge and adapting it to the specific task at hand. Now, what techniques be more specific? Does ULM fit? Let's see how it goes. Okay, there we go. So here's the three steps, pre-trained, fine-tuned, fine-tuned. Cool. So you can see it's not bad, right? It's not amazing. Like, you know, the context in this particular case is pretty small. And it's, and in particular, if you think about how that embedding thing worked, you can't really use like the normal kind of follow-up. So for example, if I say it says fine-tuning a classifier, so I could say what classifier is used. Now, the problem is that there's no context here being sent to the embedding model. So it's actually going to have no idea I'm talking about ULM fit. So generally speaking, it's going to do a terrible job. Yeah, see, it says used as a Roberta model, but it's not. But if I look at the sources, it's no longer actually referring to Howard and Ruder. So anyway, you can see the basic idea. This is called retrieval augmented generation, RAG. And it's a, it's a nifty approach, but you have to do it with, with some care. And so there are lots of these private GPT things out there. Actually, the H2O GPT web page is a fantastic job of listing lots of them and comparing. So as you can see, if you want to run a private GPT, there's no shortage of options. And you can have your retrieval augmented generation. I haven't tried, I've only tried this one H2O GPT. I don't love it. It's all right. So finally, I want to talk about what's perhaps the most interesting option we have, which is to do our own fine tuning. And fine tuning is cool because rather than just retrieving documents, which might have useful context, we can actually change our model to behave based on the documents that we have available. I'm going to show you a really interesting example of fine tuning here. What we're going to do is we're going to fine tune using this no SQL data set. And it's got examples of like a schema for a table in a database, a question, and then the answer is the correct SQL to solve that question using that database schema. And so I'm hoping we could use this to create a, you know, a kind of, it could be a handy use, handy tool for business users where they type some English question and SQL generated for them automatically. Don't know if it actually work in practice or not, but this is just a little fun idea. I thought we'd try out. I know there's lots of startups and stuff out there trying to do this more seriously. But this is, this is quite cool because it actually got it working today in just a couple of hours. So what we do is we use the hugging face data sets library. And what that does just like the hugging face hub has lots of models stored on it, hugging face data sets has lots of data sets stored on it. And so instead of using transformers, which is what we use to grab models, we use data sets. And we just pass in the name of the person and the name of their repo, and it grabs the data set. And so we can take a look at it. And it just has a training set with features. And so then I can have a look at the training set. So here's an example, which looks a bit like what was just seen. So what we do now is we want to fine tune a model. Now we can do that in, in a notebook from scratch takes, I don't know, 100 or so lines of code is not too much. But given the time constraints here, and also like I thought why not, why don't we just use something that's ready to go. So for example, there's something called axolotl, which is quite nice in my opinion. Here it is here, another very nice open source piece of software. And again, you can just pip install it. And it's got things like GPTQ and 16 bit and so forth ready to go. And so what I did was I, it basically has a whole bunch of examples of things that it already knows how to do. It's got llama to example. So I copied the llama to example. And I created a sequel example. So basically just told it, this is the path to the data set that I want. This is the type. And everything else pretty much I left the same. And then I just ran this command, which is from there, read me accelerate launch axolotl passed in my YAML. And that took about an hour on my GPU. And at the end of the hour, it had created a QLaura out directory. Q stands for quantize, it's because I was creating a smaller quantized model. Laura, I'm not going to talk about today, but Laura is a very cool thing that basically another thing that makes your models smaller and also handles can use bigger models on smaller GPUs for training. So I trained it. And then I thought, okay, let's create our own one. So we're going to have this context. And this question, get the count of competition hosts by theme. And I'm not going to pass it an answer. So I'll just ignore that. So again, I found out what prompt they were using. And created a sequel prompt function. And so here's what I'm going to do. Use the following contextual information to answer the question. Context create table. So there's the context question list or competition hosts ordered in ascending order. And then I tokenized that called generate. And the answer was select count hosts comma theme from farm competition group by theme. That is correct. So I think that's pretty remarkable. We have just built, you know, so it took me like an hour to figure out how to do it. And then an hour to actually do the training. And at the end of that, we've actually got something which which is converting pros into sequel based on a schema. So I think that's that's a really exciting idea. The only other thing I do want to briefly mention is doing stuff on max. If you've got a Mac, you there's a couple of really good options. The options are mlc and llama dot CPP currently mlc in particular, I think it's kind of under appreciated. It's a, you know, really nice project where you can run language models on literally iPhone, Android, web browsers, everything. It's really cool. And and so I'm now actually on my Mac here. And I've got a tiny little Python program called chat. And it's going to import chat module. And it's going to import a discretized seven B. And that's going to ask the question, what is the meaning of life? So let's try it. Python chat.py. Again, I just installed this earlier today. I haven't done that much stuff on max before, but I was pretty impressed to see that it is doing a good job here. What is the meaning of life is complex and philosophical. Some people might find meaning in their relationships with others. They're impacting the world, et cetera, et cetera. Okay. And it's doing 9.6 tokens per second. So there you go. So there is running a model on a Mac. And then another option that you've probably heard about is llama.cpp. Lama.cpp runs on lots of different things as well, including max and also on CUDA. It uses a different format called gguf. And you can again, you can use it from Python, even though that was a CPP thing, it's got a Python wrapper. So you can just download again from hugging face, a gguf file. So you can just go through and there's lots of different ones. They're all documented as to what's what. You can pick how big a file you want. You can download it. And then you just say, okay, llama model path equals pass in that gguf file. It spits out lots and lots and lots of gunk. And then you can say, okay, so if I called that LLM, you can then say LLM question, name the planets of the solar system, 32 tokens. And there we are, run Pluto, no longer considered a planet. Two, Mercury, three, Venus, four, Earth, Mars, six. Oh, never had other tokens. So again, you know, it's just to show you here, there are all these different options. You know, I would say, you know, if you've got a Nvidia graphics card, and you're a reasonably capable Python programmer, you'll probably be one of you use PyTorch and the hugging face ecosystem. But you know, I think, you know, these things might change over time as well. And certainly a lot of stuff is coming into Llama pretty quickly now, and it's developing very fast. As you can see, there's a lot of stuff that you can do right now with language models, particularly if you feel pretty comfortable as a Python programmer. I think it's a really exciting time to get involved. In some ways, it's a frustrating time to get involved because you know, it's very early. And a lot of stuff has weird little edge cases and tricky to install and stuff like that. There's a lot of great Discord channels, however, FastAI have our own Discord channel, so feel free to just Google for FastAI Discord and drop in. We've got a channel called Generative. You feel free to ask any questions or tell us about what you're finding. Yeah, it's definitely something where you want to be getting help from other people on this journey, because it is very early days. And, you know, people are still figuring things out as we go. But I think it's an exciting time to be doing this stuff. And yeah, I'm really enjoying it. And I hope that this has given some of you useful starting point on your own journey. So I hope you found this useful. Thanks for listening. Bye.