 Hello everybody. Thank you for being here with us. We are going to be covering today how what's the role of open source in the LLMs community. It's going to be mainly two points, the role of open source in the development and then how you can expand the capabilities of your LLM tools with the most common open source libraries. We are Antonio and Pablo. You can see that we weren't that smiling that day in the mountains. We are the engineer and data scientist for Elasta Cloud. Elasta Cloud is a data consultancy based in London. We work on a lot of different industries from finance to retailer and well lightly. Yeah so basically Elasta Cloud has been a data engineering company but for this last year we have been focusing a lot in LLMs and well OpenAI with Azure OpenAI since we are Microsoft partners but we really have worked with all the ecosystem and have seen how it's evolved. So this is a bit of our experience with it. So these are the main points that we want to cover in this talk. First we are going to talk a bit about the LLM development from the developer point of view but also from the historical point of view. Talk a bit about open source LLMs, private LLMs and a bit of the conflict that there is. Public private development of those LLMs and a bit of the community effort. What problems have the community found and what problems they have solved. What libraries have appeared and we will show a few examples and why we go. All right so the first thing we want to talk about is how despite the later popularity of Czech GPT and LLMs this year this is not something that followed of a tree on January. This is an ongoing effort that has been hardly supported by the open source community. As you can see in this diagram every single LLM that is open source is shaded. Well the breakthrough was with the attentional unit paper which marked the transition from encoder only to encoder decoder so as you can see that branch is almost not that good right now but all that powerful branch of the encoder only and encoder decoder has its roots on 2019-2018 and throughout all this process open source community was involved on it and all the models were published in an open source way. Here in this graph you can see that despite that the breakthrough was achieved by OpenEI at the beginning of the year last month of last year and they performed this marketing stunt where Czech GPT would be all over the place and in the common public and now it looks like they kind of are the owners of this technology and they're so responsible of it but our point here is and it has been discussed in this summit already but these are the groundbreaking papers just to illustrate our point is not something from last year but it comes from six years ago those are public researcher papers you can read about them so it's not the intellectual property of only one company but they make the breakthrough they make the marketing stunt and now let's get back to the one now what they are talking about is about the dangers of these LLMs they want to sway the public to a point where it looks like these AI tools are dangerous and need to be regulated and they want to keep the open source community and other potential competitors out of the scene which is something that is the real risk since we will lose the actual public scrutiny and our opinion and the democratization of the software. Yeah so even though that's what's happening with OpenEI there are a lot of companies that are putting their money in open source and there are 14 open source so you have companies like Hadinface, Lamaindex that have got millions of funding to develop this ecosystem and these tools and platforms you have models like and it's very important that the open source models go with hand with the commercial so you have for example the MPT that is from Mosaic that is it's a model now owned by Databricks we have worked a lot with them it gives you the option to use it in your commercial products that is something that wasn't there with LLAMAS an example LLAMA is for non-commercial use only so that way you have also the option to go from open source to to commercial available so let me go back a slide that we skip so this is just some examples of different models so you have Lama2 by META you have this the FLAM T5 that is from the encoder decoder branch but you have here a few others so MPT is the one that that is from Mosaic that I just mentioned but you have this and you have a lot of other models that you can choose depending on your use case or Falcon is also one that is that is quite widespread right now so let's talk a bit about the ecosystem that has grown around LLAM so LLAM is just a model so you give it an input and it gives you an input so to do things with it there has been a whole ecosystem that has grown around it so you have developer tools for prompt engineering like different patterns, chain of thoughts, tree of thoughts you have orchestration frameworks things like LMA index, land chain that have a PR there to make it easier and to also implement these research ideas that we will see it later but things like tree of thoughts so there is a research paper on it and then LMA index or land chains go and implement it you have tools like vector databases, API like platforms that host those models so there is a whole ecosystem we are going to touch a bit of this ecosystem and why they they appear here but it's much bigger than this if you have seen the gen AI slides you will see that they have a huge open source ecosystem and that's only the open source site. This is based on our personal experience while applying LLAMs in the industry and these are the solutions that we have found useful a lot more of there if we miss anyone, forgive us. So let's talk about the first issue we always find when working with LLAMs is the context window you have a chatbot that's very easy to to have a properly useful context window since you only need a few sentences to be the context of your LLAM however soon the outcome over the place asking for summaries from long documents and data mining from unstructured data. We quickly found that for K tokens limit that's between the answer and the actual prompt of the of the model wasn't going to be enough and even though the some companies open AI for example reacted very quickly to that releasing GPT models that are that have a 16 K tokens context window we also found that those models usually are way worse on their output when asked about some specific tasks since they are generalistic and they lose their concentration. Again the example is data mining we were having a solution with where I would just track key data from a text and when getting into a model with more tokens input limitation it would not be performing to the same quality. So one solution we found for that is actually one is to make embeddings. Embeddings will go through them I'm pretty sure most of you are familiar with them embeddings is the vectorial representation of text as an example here you can see how a very whole from a small corpus two different centers would be codified but you can go ahead and with LLAM these embeddings are a vectorial representation of a very big subject a whole broad subject then you can use that to find closest or similar subjects in this cloud we can see all the like five five thousand Wikipedia articles and you can see if you select machine learning you will get from the cosine similarity criteria you will get which articles are closer in meaning that's so much learning would be closer to artificial neural network recurrent neural networks etc that means that you can now give a tool which is in this case the one we use was Lama index you can give you can have a tool that will make chunks of your text take make an embedding of its of the paragraphs compare them all and then if you are asking a prompt to the LLAM in this case it would make an embedding of that prompt compared with all the chunks of the text which are in a sliding window that overlaps and then only fit the model with the relevant chunks of the document so the LLAM doesn't have to include the whole text but rather where it is going or it is going to find the actual answer that we are looking for yeah so as Antonio said Lama index is one of the indexing tools so this is one of the ones that we are more familiar because I mean we started at the start of this year working a lot with LLAMs and this one this was one of the first one that did indexing very well so they have integration with a lot of open source and closed source vector databases so you can use those vector databases to do this semantic search basically to search for documents that are similar to the ones that you have and they have integrations with lots of models through LLAM chain and you can you can deploy this in any platform that has Python so it's a really useful tool to build to build your own indexing so I'm going to show an example now so this is this code is in the repo that is linked to the presentation but basically this is a tool that we build ourselves it's something that I actually use a lot so this is a document summarizer it uses LLAM chain but you could implement the same purely with LLAM index but basically you can drop here at PDF or give it a URL so this is for example the Linux foundation newsletter for September and you can click summarize so this runs the index so normally it takes a bit more just so you know but I had it pre-computed so it will get the index so for example in this case this whole page the text of the page has 14 000 so it wouldn't fit in the context window for GPT 3.5 that is the one that this uses so this chunks it you can see it has 18 requests requests so it has it has done 18 requests for the different chunks of the documents so this is your summary and now it gives you here the summary of the web page so you can see the different points that has happened sometimes it misses a point or you need more information but in general it's quite useful when you have an announcement you want to summarize or even a document that you want to see what's happening here so this is using OpenAI but you can use any LLAM through the integration with luncheon if you want to host your own LLAM or do any other implementation so this is an example and the code for this is quite simple you can find it in the github people if you want to look at it so the other issue you should find with LLAMs is knowledge and capability limitations to these two levels the first level is the fine-tuning and the scope of your LLAM they are limited in what they know and in what they can be applied to let's say you have a code you have your own repository and when you want an LLAM to comment on your code and tell you what are the errors well you have a few options you either have a incredibly big LLAM that has been trained on all the possible codes of the world and knows what you're worried about you fine-tune it to know what your code is or you have used your code as the training dataset some of the options are obviously possibly costly for the industry or for the everyday user assume we are individuals just wanting a small solution um there and then the other option the other limitation sorry it's uh the capability of these LLAMs at the end of the day are just an an stochastic tool that will select the best chunk of text to complete a task they are not real programmers they don't really know what they're doing but sometimes in your solution you would like them to actually execute a piece of code that they are suggesting or for example I want them to write a summary of my repository and write that into the readme file what they can actually interact with the repository but there comes the agents and they do that yeah so agents are one of the patterns that appear to solve these kind of issues basically an agent is is a black box that receive an input normally a task so you ask it to give you an answer or to do something and then the LLAM acts as a reasoning engine so if you look at the problems that are normally used here you actually ask the LLAM to give you the steps that it would take or the tools that it could use to solve the problem you then parse that output um you uh you parse those actions into tools that are going to implement those actions and then those tools can actually interact with environment so you might be able to search for documents search for I don't know search Google for this answer search the Wikipedia or do any any other kind of actions that you that you need then this information is goes back to the LLAM and the LLAM needs now to make the decision has I reached out the the solution or do I need to take more steps or more tools and then you have the agent basically taking the the LLAM taking the decisions based on the context that you give it um so these agents it's it's very interesting the development of these agents because you can you can trace back to the original papers how it has evolved so one of the original papers is the MRKL system so the knowledge language systems so they basically define a system where you have this knowledge engine and then you have the instead of tools they call them experts that can give you more information about the subject so that way your knowledge your system is actually not trained in all the information in the world it's just trained to be very well at reasoning and then you have these experts giving you more information you have then the react agents that is the reasoning and acting that is an evolution of these ideas you actually see in November launching implementing both systems the MLK and the react in this PR and then actually when it got in the media is when you got auto GPT that implemented it and you have a lot of people using it to different things I mean you have some crazy stories using auto GPT to order food or when you didn't have enough in the fridge or things like that so you have this in the media so this is kind of the evolution from the research community to actually implementing it in open source tools that have factory adoption um so yeah so these are the the different some different tools uh or um that you can use for app development so they have uh they make your life easier to interact with them so you have agents chatbots pipelines question and answer it but they give you like templates to interact with the tools and they make it very easy to go let's say that you start with open AI because it's the easiest one right now but then you want to host your own model it makes it really easy to just change the mechanism go to your own model so based on we are very familiar with launching that's a tool that we have used in most of the examples but the others are also very good so now we're going to go to an example yeah oh well yeah so just to illustrate the the limitations is everybody is familiar with chat GPT and everybody knows that it's not really good at math uh just jump into a new chat and ask it to roll a day and see what the answer is this i'm bet it's going to be four actually four if you see the history we have a whole history of fours from a random die that's because four is for this model the actual more statistically uh probable answer so it's always going to be four and if we roll another one it's probably going to be two but i'm not going to risk that one um so this is just to illustrate how it is just that it's a completion machine but what it can do actually is and this is our spoilers for things i wanted to show because we have scrolled down before um so let's just using for example line change um and using LLM not as the engine that should roll the die but rather the thinking and writing of instructions of the whole agent we can build a python agent where the LLM would be the sort of the brain that will think well instructions for rolling a die using python would be to use running from one to six and print the print the answer right so this is really easy really you just have to configure your LLM uh and peeping style land chain and create an agent and there you go you just use natural language to interact with your agent this means really uh any um any person who doesn't really know how to code could talk in natural language with this agent um make python uh do stuff for him in this case let's talk about this example we were talking about so roll a dice we have run this before so hopefully yes your mouth is really weird so if we run this agent where we just wrote roll a die uh and print the result dies for some reason um it should uh as you can see here start the agent and in the agent it will invoke the main tool which is the python tool from land chain and it would make a query the query is the LLM right into a python console uh a piece of code which literally uh ask a random running from one to six and prints the result the result is three and now all your LLM is actually able to roll uh to roll dies uh well this is a very basic example but you can go wild uh what is the 10th Fibonacci number then it would design a more complex code to calculate it and print the result and it can operate with everything that python can really do but what about more complex tasks because this is where agents actually get interested and uh uh something that can be really powerful in our solutions or even in your day-to-day so this new agent I gave it three tools we gave it the the ability to use python we gave it the ability to search for files in our directory and the ability to use the shell tool for example in this case it actually went forward and installed the library it needed also there are risks with this it's going to be able to tamper with your of your system so maybe use a container um but yeah so this agent now is going to be asked is there are slides pdf file notice how in the prompt I had to actually write the correct path because otherwise you wouldn't understand that it can search on the whole uh on the root system but given the proper prompt is able to say yes this slide exists let's go a little bit forward with the idea and ask it to read that file to sum up its content and again I had to specify to use this library because otherwise it would always try to use pay 2 pdf why because that's a most a more common pay um library when you go into google or stack overflow and ask how to read pdfs and maybe it's not the best one so always a little bit of insight from the actual developer is needed so our jobs are safe for now um but yeah you can see it would is we make this a scroll event these are all the steps it made so the steps are the lm will yeah your mouse is really terrible so the on green you can see the step that the lm thought about doing so it's invoking and it's writing the the code it would like to to execute which is then executed loading and reading the pdf which is the slides from this presentation and then with all that text the lm will read it produce a summary and print it back to us effectively we just ask in natural language an agent to do that for us and it did using our system and our our directories let's go a step further there are specific agents for different tasks there is a pandas agents let's say you are a poor data scientist who don't really know how to code talking for a friend um you can let's just load an example that assessed here and we can set up an agent this is straightforward from line chain box you can just pip install line chain you import you import it and then you create an agent you can see here how easy it is to give it the data friend you have and you give your lm engine this again we are using for simplicity opinion but could be any other lm you have and then we will try some easy queries you can ask in natural language how many rows are in the in the data frame it will tell you 150 you can ask in natural language what's the deviation of a column you can ask what's correlations any of the usual um questions that any data analyst would do to add a frame but we can go further and maybe ask him to do a whole exploratory data analysis well that's a lighter um that macadam is wrong but let's ask it about uh run uh plotting the actual data it would try to do that you just have to write a plot me these column against this poll and it would try to do that of course again as i said before sometimes the plots are not that good because it's still uh an lm trying to to guess what the best action is and the most common plot is the line plot but you can specify just with another extra word make that an scatter plot and then you will have a proper plot which gives you a valuable insight or that literally is a nice visualization that you just built by asking a robot even more a step-to-step task is a whole that exploration of the data frame where you can see what his thick thought process is well where it will um start a change and do a series of steps trying to explore the data and see what the actual shape of it is it will do a describe it will do an info it will uh do correlations it would even um uh make a box plot plot on its own of all the columns so uh we have now just out of line chain box a very capable agent for interacting with uh or all that data frame or even do some simple python tasks now is to include this in your whole solution with a couple of different agents and giving it more tools because for example another very useful tool is to give it access to a search engine where you can have your bot to access the the tuk tuk search for example um so it could browse what are the best plots for this data frame or what are the best practices for an EDA and then it would include that in it in its actions these all these uh are in the repository we have another extra examples there is also a very good agent that can interact directly with github with your repositories um with sql another type of database it's really cool yeah so there are a lot of examples i mean um so we will um show later a few links where the way you can start looking at it so we have added some information here we don't want to go too long on this but this is the evolution of line chain uh that is the main tool we have used you can see how it has exploded um and it's still like linear growth so it's still in the growing phase so there is a lot of community there i think they have now built their own platform with lang smith um so we will need to see how they go with the open source side and now they don't platform uh because we have seen how that's going a few other um really good open source uh libraries and and platforms that we recommend uh listed here there's not too often to them too much but the sky pilot will allow you to build a framework for rolling your lm's or well any other batch that you have it will literally standardize your interaction with clothes providers and allow you and i searching of where is the cheapest virtual machine right now um other useful tools are psychic with this uh i mean another layer of interaction which is unified for all your common unstructured data sources we're talking not showing slack uh or even go google drive uh so you don't have to uh configure your lm for each one of those tasks fast chat straightforward you want to create a bot chat just out of box um virtual lm allows you to serve your custom lm as a as an api an api is the way for example uh check tpt open ai uh check tpt works uh most of the applications are now being built around using and consuming uh check tpt as an api which all the drawbacks that that entail for example in prompt drifting or the model changing because it's being constantly retrained and that's out of your control if you want to to change to move from that to your own control lm in your own virtual machine and still consuming as an api that's one of the best solutions and finally uh we wanted to to highlight these two libraries i like them especially not because of what they do which is still impressive but but because of the idea behind them and it's that all the prompt engineering and the development of solutions around lm's should be test driven um we should be controlling what our prompts uh make our lm's to do what the output is and how good that output is and we control it over time uh if you want to change from one lm to another want to change the parameters the size fine tuning you should do you should be doing that with uh the actual um uh regrowth uh measurement of the of the of the development and well uh include that in your development um that's all the tools we have mostly used there are much more um we have examples yeah so we have been showing the examples um so we're not going to go too deep now so you have the link here if you want to go to the repository see the code or do your own in the end the the what we wanted to transmit here in this call is that they're out of tools there is a big ecosystem um i think it's really easy to start building your own tools like you have seen that toolkit that i showed that i use a lot it was built in an afternoon so it's really easy to start with it these are a couple of links that i find that they are really good this also in lm and lm's practical guide so they are to repost that these developers have created um they have uh basically a list of libraries papers other links that you can use to to find out more it's the best place to be up to date yeah yeah and and it's also the best place to start so yeah i mean start building your own code and contribute to these platforms that's the conclusion of our talk and i think that's it so we have a few more links here if you want to find more information but i think these these two are the most slides in the repository as well thank you that's it questions complaints feedback i've seen that uh mostly every time is python the language being used in all ai but it's the only one now so there are frameworks in most languages so javascript is also another one that is used a lot but in the end the main language for data science is python um i mean there were some so we are very familiar with data bricks so data bricks had some statistics that 80 percent of the code running their platform is now python even if they started in scala so that's an example about how python has cannibalized the rest of the tools but you can find tools for other languages so you have uh for javascript you have implementation for some of these tools you have also for dotnet so depending on your platform you can find but probably the most uh at bounce and the ones that get the the first updates are the python libraries yeah and besides the model which is uh developed in a given language most of the time you have an api that you consume you can consume it and build your own solution around it in any language one which is a good point maybe let's encourage the open source community to not only develop those solutions in python but actually think about others or you can use that thank you is there a way to see the problems used by the agents yep yep actually you can see them here uh if this is uh there is always an option in well in length chain specifically where you give it when you create the the agent you give the option verb balls true uh where am i here today set up there here see verb balls is a little true and it will give you a chain of thought and action so in this case if you go to an actual query you can see that the thought is i need to count the number of rows so it writes that so the next code that is going to write follows that instruction so it's giving instructions to itself then it makes the action which is uh using the tool and the actual prompt is len uh df which so you can check that what he is doing is actually correct also with line change you can so you can enable something that is the the the book mode and it will if you want with lunching um so it would not only show you the actual line of thought from from a point of view but also what he's doing under the hood so you understand who is using the lm and what's the actual prompt being used all the time it's a bit slow i think it was yeah so if it runs and now i run this it should give you yeah you can see the input how many rows are there then it goes into the you can see the steps like agent executor then the it goes into the chain then it goes into the open ai and here is the whole prompt you this is an open source tool right so you can just control click on the code and go and see the prompts this is really useful when you are building your own lm because uh your own agent so your own solutions because you see what is a standard and what works for others and you can fine tune this to make your own solution um but yeah so the these tools are very transparent because in the end it's all prompt engineering for with these tools and the prompt is the most important tool to make them do what you want thank you thank you okay if there are no more questions um thank you all for coming um you can reach out to us anytime you want if you want to have a chat or we will be around here if you want to talk thank you