 So again if you just joined us and you plan to follow along on your laptop There's a link there. It's in the slides. It's also online Which will take you to the to the lab the workshop and the instructions there. There's There's a pre-work portion of the workshop. So again, if you do want to follow along and you want to test some of these things out You'll be able to sign up for a Watson account and a free trial and play around with some language models, too. And if anyone has any issues signing up or looking for the link or Any issues that I'll just let me know Hopefully those of you who have made it there you probably already started clicking around and seeing It may be even throwing some problems at some random language model already But as an overview, I'll talk a bit about the lab itself Hopefully that's that's seeable, but everyone should also have a tab open with the lab itself just because Some of the instructions are there and some of the exercises that will go through there. So The purpose of this lab is basically just to have fun I mean, we're just trying to throw prompts at some language models and Learn a bit about how to to tune our prompts to get some of the output that we want We'll learn some basic examples of how to how to do just that We'll talk a bit about some of the techniques and maybe some of the parameters that you can play around with To kind of tune your output and and we'll go from like a very simple output and try to make You know generate some text that that we could actually do something with or learn something from so Here I'm just showing the lab itself You've probably all seen this page But basically if you click on workshop Again, you're you've taken to the pre-work and if we click on lab one lab one is a bit Kind of a read-through type of lab, but in our case, I'm gonna actually show most of Most of what's on here. I'm gonna actually demo so you don't necessarily have to read through this yourself But all the information that I go through is there so on your own time You will always have this free trial and access to it So you'll always be able to kind of go through and and look at some of these these The ideas that are presented here And then in lab two it's more of kind of an exercise-based lab where there's ten prompts or ideas exercises and Along with some example answers if you click it And each of them are a different task. So for example ones Generating maybe some creative sentences another one summarizing a transcript and so on This part we'll we'll try to do individually, but for the first I don't know maybe 20 minutes or so We'll go through we'll go through lab one and the prompt lab itself So let me just sign back in here You're all just gonna watch me go through the same Process that you have No, you would be able to get there from from IBM cloud There's an easier way to get there, but I Try to stick to the the pre-work instructions to not You know IBM cloud comes with it all of the other buttons and all the other products that are there the whole suite So and rather than bogged down in all those details that rather just showed this you know so You've all or at least most of you have seen this page and you're on this page at some point and There's a lot that you could do here. This is what's an x.ai you can train models tune models You have notebooks and all of these things But the thing that we're going to actually play around in is prompt lab and that's this first This first button here experiment with foundation models and build prompts that sounds like exactly what we want to do So again, if you're on prompt lab, hopefully follow along even some of the the text that I put in hopefully you will also Type in and see what what output you get. I think that's kind of a fun part of language models is is kind of the comparison of What we get and what our output is so this is prompt lab and prompt lab again it's part of Watson x.ai and it's basically a Fun way of being able to experiment with a bunch of different models play with a bunch of different parameters And and there's sample prompts. There's you know the ability to save prompts and all of these things and This will be the first view when you do open up prompt lab And you can see this is called the structure view and just let me know if you want me to zoom in more But I'll kind of be zooming in and out. It's accidentally went back. I'll be zooming in and out like this Throughout so hopefully that'll be visible when I am kind of focusing on a specific part but this is the structured view and At a glance, this is what we see we see the setup we see somewhere to put our instruction and in the gray text we see This is where we would actually put, you know, again the instructions summarize the transcript Tell me a funny story. Whatever the prompt might be. This is where we would put it in the structured view here We have some examples. So for example, if you wanted to summarize a transcript this input Example would be an a sample transcript and you would actually put in the output And so you would put your summary to of the transcript and that would be an example. That's passed to the language model For it to actually learn from or learn some structure from and then here you would be putting in your your test input and Generating your test output But for this lab or at least this demo, you know All of you can feel free to use the structured view if you'd like to try it But we'll go to the freeform view and the freeform view We simply see a text box and we see a pretty important reminder here Which is the fact that this is not a chat interface, right? How many of us have played with chat GPT at some point, right? So we all kind of expect especially with something as big as chat GPT. We're expecting some very coherent Responses, you know, give me an itinerary for this certain country or something and it gives us a nice list of Responses but in this case, this is not a chat interface This is you have to kind of think of these language models when you're when you're prompting them in this fashion as Text generation or sentence completion or you know, you have to kind of think of it like that And so right away like let's let's try a prompt and we'll try with something simple again This is all kind of covered in lab one, but I'm just going to be talking through as we do it list ideas to start a dog walking business Right and again in our experience of chat bots and chat interfaces We might expect a list maybe a numbered list of Good ideas to start that type of business and when we click generate right away It actually spits out the exact same thing that we said You know and again try it yourself see if you get something different you probably won't But there's a couple of things to note here first of all It's where it generated it if I had my cursor up here right at the end of the sentence and I click generate in This case it would have generated right there So, you know, that's just a small thing But where we we kind of place our our new line in our cursor. That's where it's going to generate it So Now we have to think about a few things, you know First of all, why are we just getting back? You know this this this this output it obviously doesn't help us. It's not what we wanted So now we have to start thinking about the prompt itself But there's a few things to think about there's the prompt itself We'll get into kind of sharing examples with the model and trying to help it or guide it to give us our A good response, but the first thing I'll point out is the top here where it tells us which model We're actually playing around with or or, you know, we're sending requests to The default I think for for everyone should be this flannu l2 model And and if you open that up and you click view all foundation models This is I think one of the very fun things of prompt lab is that you have all of these models at a glance And and you're able to do exactly as I did with each of these And so we have all of these models to choose from and so the first thing when it comes to you know I want some output from a large language model is which model are you going to use because some models are Specified to a certain task, you know, some might be Experts at kind of code generation another one might be more instructions or creative Language models, right? So it's very possible that you're simply using the wrong model for this task For example, if I click on one of these Suggest star coder as the name kind of suggests This is a model for code generation and if you actually scroll down you'll see at a high-level view a Model card or a bunch of information about the model Maybe how it was trained how big it is It's intended uses and sometimes there's even hints in these model cards For example, this tells us that it's not an instruction model and commands like write a function that da da da Do not work well Right, so sometimes they're hint. They're hints there with the models itself You could even see Lama 2 if we actually just open this one up and we click select model Let's see how different our output would be if I just click generate here with the same prompt We actually so in this case I actually clicked on llama to this top this bottom left and again You could choose any model, but if you're really following along I'm choosing this llama to model and then at the bottom right there's a select model button and so that'll actually select the model and at the top right now we can see which model we're working with and When we click generate at that point we can see we we get much different output Already we're getting I guess, you know, we can probably consider that better output Research the market understand the demand for dog walking services in your area and So It's much better. We're using a much bigger model But I'll point out here that we can see that the the output cut off for some reason and Theoretically maybe the model decided that that's the end of the sentence, but how do we know for sure? well at the bottom left of the text box I'll just zoom in there and We see The more yeah, okay cool So we'll actually see the stop reason here and here the stop reason is max tokens parameter reached Other situations for example might be maybe end of generation reached or end of output reached something like that That would tell you that the model actually decided to finish its sentence there But here we have max tokens parameter reached we have some information about the tokens right 14 input tokens 20 generated tokens 34 out of these 4096 tokens a lot of information down there and basically if you don't know tokens are kind of like the basic unit of Characters or or data that a model uses as input and output So when you're passing a model an entire sentence it goes through a process of tokenization at some point And this is it could be up to the model it could be up to the scientist whatever it is But it'll actually create subsets of those of that data So it could be three characters. It could be a whole word. It could be the whole sentence and so on in our case We can kind of think of it as a short word So maybe you see something like 14 input you might think okay 14 short words Maybe 14 four character splices something like that but we see 20 generated and Again, we saw that this this output cut off and so now at this point We'll open up the parameters the model parameters button here, which is on the top right and if we open that up there's a ribbon of the model parameters that that pops open and Here now we have even more levers and buttons to press At the bottom of that we see max tokens and I'll only point this one out for now But of course this max tokens is 20 as we saw our output actually stopped at 20 tokens And so now we can up this to whatever we want. Maybe 50 you could try 100 150 But there's different reasons of course for different numbers. We'll keep it at 50 because of Because usually in our cases at least that that that takes but if you wanted a longer story For example, you might do 150 if you find your model is is spewing out too much output and and going on tangents or something like that You might want to bring those tokens down the minimum tokens, of course does the opposite it Either forces the model to at least output a certain number of tokens Or none at all and so we'll leave that at zero So now let's regenerate this and see what happens So now we we see that it continued so research the market understand the man for dog walking services and identify your target market And then there's a number two develop a business plan and so on So, you know, it's it's giving us a full coherent idea And a couple of ideas and that's great I'm gonna go back to the default model where we were getting some poor output right this flan ul2 model and Again, when I click generate here, we're gonna get that output that we got in the first in the first place And that's list ideas to start a dog walking business So how do we make this prompt better because I know that this this this language model is an instruction-based model It's a pretty general model again if you click on on On the model card itself It'll actually tell you some information and even at the top it gives you these these categories It tells you at a glance question answering summarization Generation extraction and so on so in theory this model is a pretty good one to to use for this this case and That's where we're gonna get into our Maybe first thing and that's that's actually changing the prompt itself, of course it's a bit obvious but list ideas to start a dog walking business is It could be more descriptive in all of these things, but maybe we want a numbered list and so What we want to do is we want to kind of nudge a model to output kind of the things that we want the same way maybe in conversation you might You know shape the way you say something because you want the other person to Speak about a topic in a certain way So if you think of it in terms of kind of nudging or prompting a model to do something We actually include this number one We could include this number one in this list ideas to start a dog walking business Because hopefully that kind of implies that we want a numbered list right and we generate here and We shouldn't as expected it actually doesn't get much better. It just says dog walking Right. It is different output than before and now we have some more input tokens. We have 13 now, but Still no good So now When we're giving When we're kind of satisfied with this prompt again, this prompt is a bit vague It's very specific and and short, but we actually get into what's called one-shot prompting It's also known as few shot prompting multi shot prompting It basically just refers to the idea of actually providing a model with an example of what you want and so in lab one Maybe towards the the you know towards the top of the page. There there's a place There's an example here that that's there for you to copy and paste and This is what I mean by an example here. We have a different topic We have list ideas to start a lemonade business number one set up a lemonade stand partner with a restaurant arrange a Celebrity to endorse the lemonade those are three good ideas to start a lemonade business somewhat abstract a bit creative and then there's our original prompt here list ideas to start a dog walking business with our number one And so again, this is the idea of one-shot prompting. We're now not just giving it an example of You know dog walking business ideas, but we're giving the language model a Structured example of what we expect as output, right? So if I list ideas to start a lemonade business I want three ideas or numbered ideas about how to do that. And so now what happens when we click generate All of a sudden we're getting a much better response than before. I think we can probably all agree. There's a few things here though Well, first of all, I mean the ideas are pretty good get a dog walking certification Advertising local papers business license and so on those are those are all good ideas And if you remember our very first prompt we were actually just getting it to to respond or it was just repeating exactly what we gave it I wonder if we can fix this thing though, you know it seems like it decided to give a the one anyways and if we get rid of that one and Maybe don't pass it the one. Let's see what happens And again if you are following along like feel free to try your own things too as I'm talking and just you know Play around with it because it's it's meant to be Kind of a creative exercise and and all the different outputs that you can get it's it's You can just see here even how much it varies that number one actually turned out to be a pretty maybe important piece of our input Because here it just says start by walking your dogs in in your neighborhood. That's a good idea Yes Right exactly and I mean the idea here is that this is the base model, right chat EPT is a bunch of models and And a very curated type of model and it's also much much larger So this is only 20 billion. I forget how much chat EPT is but it's you know in over the it's bigger than all of these models here and There's reasons for using smaller models and being able to kind of prompt them in this way to get some output for example safety without the generalization Like without using these like overly generalized models You kind of have a smaller surface of attack for example for with things like prompt injection Things like bias, you know, so there there are these Use cases for using these more specific smaller models that can be fine tuned towards a very specific task for example if you can think of I don't know in a business maybe like an HR agent Something like that without that might have access to even things like employee information and That's the smaller it is and the more concise it is towards a specified task the less chance There is that at these kind of malicious attempts to to to get it to spit out more information than it's supposed to or Do something like that That being said This is It's true that you know, I Basically just want to show that you take these these models which are not like these chat GPT models You see the output that it gets which can be absolute can be gibberish and and and not good But just by providing a simple example now we've we've we've I think Gotten to a much better place where all of a sudden we're getting all of these ideas So with that said The next thing that we can do is We can actually start talking about some of these model parameters on the right as well So we're listing ideas to start a dog walking business and again, I'll just generate it here. We have some good ideas to start that and Again another way of kind of playing with these models is you know, we've we've chosen a model So you want to choose a model for your task we've provided an example to kind of prompt the model to use some of that structure and information for its response and its its text generation and Now we can get into the model parameters if we want to really get into the kind of the fine-tuning or or cherry on top type of playing around with this output and We see decoding so decoding is basically the the the method that a model uses to output its Generate its output and in imprompt lab You have all of these information buttons as well You can click on this information button and and it'll give you a hint as to what to do with these for example Set the coding to greedy to always select words with the highest probability So it'll always kind of be maybe on topic very succinct Typically greedy is kind of used for more succinct kind of fact-based Responses right well if you set the decoding to sampling there's a lot more randomness involved in the in the generation So you might customize the variability of the word selection So you might use sampling in cases like more creativity like you want you want more Abstract answers or more randomness and variance in the response You also have this repetition penalty We didn't see it here, but sometimes the output can can repeat itself Maybe it'll give you the the same instruction over and over Maybe it repeats what you've told it and that's what this repetition penalty is for So if you ever saw that behavior you could actually play with this variable You could set it to two which is the strongest penalty for repetition or to one of course Any what anything in between as well, but in one you would have No penalty at all and that's that's described here 1.0 means no penalty You also have this stopping criteria in this ribbon and the stopping criteria is to actually control when to stop generating output Sometimes as I mentioned before Especially with the bigger models like a llama to I could ask it for for for And maybe we'll see it in the exercises I could ask it for ideas for a dog walking business and once it gives it to me It might go on and start giving me ideas for a different unrelated business as well And it might do so by maybe giving me the list of ideas And then in a new paragraph giving me the next list of ideas about a completely unrelated topic So that's something for stop sequences where I could make sure a new line in the output That's being generated would actually stop the the the model from continuing to generate output And so you would do that by for example, these are two carriage returns and adding the sequence there And and I'm just showing you these Things that you can do too because if you do work through lab 2 and you kind of play around with some of these these Creative exercises, these are the things that you can do to try to you know Manipulate your output and generate something that you're satisfied with so if we change this to greedy We'll see new variables come up and a couple of these aren't our kind of more probability-based, you know and Bless you. There's there's some Information about these variables in the lab one writings itself But the one I want to focus on is temperature because I think this is the most um This is the one we can have fun with which is actually controlling the creativity of the generated text So the more you play with this temperature the more you add the more maybe randomness the output The output contains so let's try this now. So I haven't even Touched any of these variables yet. I did change this to sampling and let's just see how different our output will be And right away. We see that it's much different We can actually see at the bottom left that the max tokens parameter was reached So this output was longer than 50 tokens again We can we can up that if we want but let's see get a small leash collar and some dog treats Make a list of your best friends local businesses. So these are good ideas on topic But we can probably agree that they're way more creative, right? And that's what sampling does This is this is kind of the idea that we're now getting Some more variance in our responses. It's less Kind of succinct and concise and so on if we Put temperature all the way to two and we generate it here What do we see? We see that the output changes your charges can help you make ends balance without going to debt collectors also take advantage All of a sudden it's gone completely off topic And in some cases you might even get gibberish So like a sentence that doesn't necessarily make sense maybe typos and so on so that's that's the problem with temperature and Including randomness into your output because too much can can actually you know lead to unexpected output so Another thing I guess I'll show which sampling here is this random seed variable One of the thing with sampling is that every time that you generate you'll actually get different output And so when it comes to reproducibility, of course, that's not a good thing And so that's what this random seed is for so you could input any number and that would be your your your random seed for for the sampling and That would ensure that there's some reproducibility in your output So if you continue to use this random seed you would actually be able to see that this output comes out again so I Think that's just about everything that I wanted to cover at least in lab one this again kind of oh, yep So okay, that's where you get into these These kind of probability or math math You know like based type of variables, so by selecting k words with the highest probabilities at each step, so It just has to do with the probability of which words that this this this model will choose There's some information there There's even links in the lab itself that'll take you to some websites that go into much more detail if you are interested in kind of the the real like inner workings and research papers that come up with these these variables and these these decoding methods So if you're all following along or For any of you who are following along you can actually go to lab two in this one. What's an X prompt lab? if you need the link again, please just let me know but in This lab two as I mentioned before there are a bunch of exercises and these are what the exercises look like We can go through number one together and then from there we can Try it ourselves if we're interested and I'll walk around if anybody needs any help or or even wants to share with their output with me You can absolutely do that, but this first case this is what it looks like it'll be we have The task is to generate so we're just generating some creative sentences about donkeys You can use the examples below to help So of course, that's a prompt to to use you know one-shot prompting or or more to kind of get the the information that you want and The thing about these exercises too is use whatever model you want if you want to use llama to which will give you those Big creative examples you can use that if you want to try to use one of these smaller Models like I exemplified and and build up your your answer You know do that as well if you click this ribbon, it'll actually show you an example X answer with some output But this is maybe how we would take tackle this one. So, you know, let's try it if we go So what's the prompt here? Three creative sentences about donkeys right so I could say three sentences about talkies We could see that the the AI guardrails or the the filter actually decides that this is not a good response or not a good input and The idea here of course is to use the examples Because the examples will will especially with this flannu l2 model help quite a bit with the with the output Let's see. Yes, you're right. Yeah, I think that happens in the training of it But when you actually want specific structure in your output, that's kind of what these examples are for These are more just guiding it to to to output in the way that you want it to But all of the examples that's gotten in the past and you can read about its training, you know in these model cards and stuff That's where it's getting all of the general examples for all of the different tasks that it can do right You don't necessarily train just to get this, you know dotted list of of or numbered list of ideas about Donkeys or puppies or whatever it is That's what the fine tuning is for and that's where these examples come in. Okay. Sorry. What was that? Okay. Oh, okay good. Let's see. Yeah Now did you use the 70 billion or 13 billion 70? Yeah, and even here sometimes this even tells you the hint of This model works better if you provide at least one example. Oh, yeah. Yeah, okay I haven't seen that before and at least when I'm playing around with these But okay, so yeah, here we have the two sentences and I hit the maximum token parameter So maybe in this case I would Up the token the max tokens and get more information More output and here. Yeah, these are these are good sentences. So that's how I llama to is definitely a very easy You know, it's it's much bigger than the other models. You can see that it's a 70 billion parameter model Some of these are smaller 13 billion seven billion so on so of course the smaller models are a bit more or require a bit more prompting and and tuning And and yeah, so that's good Yeah, there you go and which model are using there? Okay, okay Yeah, and you know you have to get creative with your prompts too. It's not just I mean in these cases. I've I've just given Three sentences about puppies and then I just say three three sentences about whatever else But you can also say something like Three numbered sentences or three numbered creative sentences You know you add some descriptive words to actually continue to tune these the output here For example, I'll actually show you the I think that the The example answer is a bit interesting because the example answer is You know, so it starts with Hopefully that's that's legible But it starts with right three sentences about a topic that I give you Sentences should be preempted with a number, right? That's that's a certain language and that's very instruction based and and this one's using the flan you L2 model as well and Here it actually gives both examples so three three sentences about puppies and it also gives three sentences about kintons But what I find interesting about this one is that it even includes example one And example two so it's kind of like maybe if you you know any developers here or something like you're thinking of of Almost like it's like bracketing your your example because at the end here It says the output should be three sentences long Output structure should mirror example one and example two so those are almost like callbacks to The examples that I've provided above and I've kind of labeled them using example one and example two Before the actual examples itself so this might not be exactly what you thought of or how you'd think about You know creating a prompt but it can get as as kind of structured as this and with these these lines We're at the top. I'm telling it right three sentences and then at the bottom. I'm giving more information or more sentences That that hopefully instruct the model so So then we have the rewrite example I won't go through it. Hopefully, you know, if you're interested you're trying it out yourself And I'll kind of walk around at this point, but In this rewrite model where we're actually converting code and so The first thing here, of course is that there is a code generation model called star coder And it's possible that other models work as well But star coder is actually meant for generating code And and that would be interesting if if you're interested to actually try that model out and see what its output is and even see You know if you ask it to list ideas about dog walking business see what comes out It probably will spit back out, you know, some some some code or HTML text of some kind, but But yeah and And anyways, I'll walk around at this point And whoever if somebody still hasn't signed up or wants to sign up or got stuck I can help them out as well Just raise your hand if you have any questions