 Okay, so welcome back. I hope everybody had a nice lunch break and Now I'm very happy to introduce Jungle call developer, but more importantly for this talk and employee of elastic search the company and on the ground Hello So I'd like to tell you a story It's a story about how we developed five clients for us to search in five different languages Without losing our minds in the process much And so as any good story it starts a long time ago in a galaxy No, no, no it starts actually when we looked at the current landscape of the clients for elastic search and There were some things that we liked and what we've seen are good and some things not so much For example in the in the python landscape. There are many clients, but none of them actually implemented the entire set of APIs none of them did Everything that we would like to see in a client and none of them did it On a scale that we would we would be comfortable with as a result users had inconsistent experience with elastic search itself and naturally they they blamed elastic search because of the way they Interfered interfaced with it was not ideal So we decided to create our own clients sort of to control The the last mile how people talk to elastic search so we can make sure that their experience is good and consistent So we started with the with the design obviously and We sit down and and set all the things that we want our clients to be and for that We need to start with elastic elastic Sorry elastic search itself Elastic search is distributed that brings a lot of problems with it and a lot of opportunities It talks via the rest API over HTTP Which is both good and bad because it's good that it can be deceptively easy to create your own clients Then the number of clients just in Python or just in Ruby was staggering just because everyone thought that oh, it's just HTTP Right, I can just do an HTTP request and everything will be good, but there are a lot of corner cases there that they didn't count with There's also a lot of diverse Deployments of elastic search some people just deploy The cluster in their own networks and talk to it directly others would use load balancers Some people would use alternate transport like thrift to gain some to gain some speed or would use a set of client nodes and a Distributed multi rack setup or something like that And also just a set of endpoints that elastic search has is quite staggering It's almost 100 API endpoints with almost 700 parameters that the clients only to support and document but More than that we wanted the clients to be true to their language That's why we only developed the four or five in the beginning because those were the people that we had We we had a we had a Python engineer yours yours truly here We had a we had an excellent Ruby developer a pearl developer. We even managed to find a really great php guy Believe it or not So those were the clients that we started with because we felt confident that we can actually make it feel Like a pythonic library not like a library written by it by a Java guy in his pair of time in Python and We wanted the client to be for everyone We wanted people not to have any excuse to use it So that's my first lesson that we learned No opinions no decisions in order to make sure that everyone would use the client We had to abstain from any making any observations any decisions because whenever you have an opinion There is someone out there who would disagree with you So the only way how to make sure that that won't happen is not to have any opinions So we decided the client should be low-level just essentially a one-to-one mapping to the rest layer And they should be extensible We should design them in such a way that where you don't like some aspect of it You should be able to replace it or just hook into it and change it So we came up with this This is a foreign for an HTTP client This is a kind of complicated diagram that specifies how the client works You have you have the client itself that has a transport class Which has a serializer to serialize and deserialize data as they go over the wire Then you have a connection pool that actually stores a list of connections Connection pool in this case is a misnomer because it actually doesn't pool collections It just holds a Collection of connections to individual nodes in the cluster You can see why we would have the naming problem there and then we have then we have connection by default we use your lip 3 because that Ended up to be the best one for for Python and we also have a connection selector So when you connect to multiple nodes, they control the strategy on how do you do load balancing? Do you use random or do you use round robin by default? We do round robin over randomized lists of nodes and The goal why we did it this way is so that we are able to give you the option of overwrite any simple component in here just by subclassing the default implementation and filling in all the blanks So some examples If you want to create your own selector you just create the class and you pass it in everything is essentially Using dependency injection so you can just pass it in as a constructor parameter and we will use that instead So you see three examples here. The first one is not really Not really injecting your own code it just setting up the options. So the first one instructs the client to Talk to the talk to the cluster and get the current list of nodes on startup Then whenever a node fails and then also every every 60 seconds. This is excellent for a long-running process Let's say a web server So that even when you keep changing your your elastic search cluster you keep adding nodes and nodes keep dropping out You still talk to all the nodes that are available The second one is where you want to control the load balancing For example imagine a scenario when you have two racks and you want to buy default only talk to the elastic search nodes in The same rack as the application server and only fall back to the nodes in the other rack if none of those are available You can do that. You can just write a simple class that will that will do this So that's that's what we Mean when we say that we are modular and extensible The last example is just using a thrift connection, which we actually provide as a as an optional plug-in and Using a different serializer in this case Yama because why not some people like Yama better than Jason for some reason So that was that was sort of the first lesson no opinion. So people have no excuses not to use it The second lesson was to prototype everything Because you just don't come up with something like this without some preparation without some prototyping and More importantly, you don't come up with it for a single language and have it be applicable for all the others It's very difficult to find a pattern that would work for both Python people and Ruby people for example That's why we created a spike of a Prototype implementation in both Python and Ruby sort of to make sure that The design will work that the design will hold for both of these languages and also So that we have a reference That we can talk about that we can we can have the same terminology and then when we talk about connection pool We know what we mean even though it means different things in different languages and even how we use it It's not exactly correct, but we had a code that actually showed what it does So we could have at that point a clear conversation even with even with a php and pearl people Even with the JavaScript people that came on and even later with with net people who were developing the new client now so Prototype everything not just to see if your if your design works But also that you have something to to talk about that you are absolutely certain that you're on the same page Because you should never trust humans and just their understanding if you can do more so That's the that's the next lesson The next lesson is don't send a man to do a machine's job Humans are amazing. They are amazing and a lot of things consistency not one of them Consistency and repetitive tasks you really want to have something that doesn't get tired That doesn't get frustrated that don't doesn't mind doing the same thing over and over again To me that sounds like a computer So this lesson states that you should automate as much as possible and This why I'm talking about this is I Already mentioned we have almost a hundred API endpoints With the 700 parameters That's very difficult to track That's a lot of work a lot of boring tedious work that you don't really want to be doing you don't hire a decent Python being and Forced them to maintain a list of 100 APIs and 700 parameters if there is any other way So what other way is there? So first we thought that we could do a reference implementation We can just arbitrarily choose one of the clients and decide this is how it should this is The reference implementation for our API's This is the authoritative collection of all the API's all the parameters their possible values and descriptions But that doesn't really scale that well first of all, we only have one Person per language. We only have one Python developer. We only have one pearl guy. We only have one PHP guy So What if what if he leaves what if he's on vacation and there is a change that needs to be made like and also How do you how does that person make sure that's everything is everything synchronous? We found out even with the spike implementation of the of the transport layer that Maintaining it when we add more features and we need to add it to both Python and Ruby even though it was just two of us and Coincidentally the two of us that lived in the same city, which is not true for any other two people on the project It was very difficult. It was difficult to keep in sync So we discarded reference implementation as an option Next we looked at documentation Because obviously elastic search has documentation and all the API's are documented All the parameters are documented as well, but Again, they're documented for humans. It's a documentation that's intended for for the developers to read it and To understand it to make sense of it So again, it would require a tremendous manual labor just to make sure That everything that we need is there Someone would have to read actually all the documentation collect all the All the stuff and not just one person, but each and every author of the client would have to do that That's a very tedious job a job that I I haven't signed up for and I I Doubt we would ever find a person who would sign up for it for a job like that So what are their options? We're there We found the progress from from reference implementation to documentation had some Had some promise, but it wasn't there yet So we decided to take it one step further To actually extract all the information that's already in the documentation. That's already in the code and presented in a in a structured format So we chose a format that's human readable and machine parsable and we had elastic search. We really love JSON So we just decided that we should document everything in JSON and create a spec of former specification For our API's This is the one case where I was super happy that our six searches written in Java as a statically gen type language Because it provides you with a bunch of tools So we were actually able to write a tool that would just parse the source code for all the API's and extract 90 80 percent of all the API's and its parameters in in an automated fashion and we then just had to go once over it and We could actually share in this effort all of the client people and just fill in the caps Fill in the documentation for each of the parameter Fill in the option and the type whether this Option is required or not whether it's a list on or a single value a boolean or an integer so that made the effort so much easier and Going forward also so much easier to define So what did we choose to to capture in this in this document? first of all the The URL path all the different variants of URL path if you if it was dynamic Which most of the URLs in elastic search are it can optionally Include an index name or a list of indices on on which to perform the the action it can Include other dynamic dynamic parts. So we have to document those including all the different options how the URL can look as part of it also We had to do the HTTP HTTP methods. So is it a get or post? We decided to do very little than that Just to list all the parameters list all the ways how to combine them into URL We didn't actually capture all the dependencies That this parameter is only valid if this parameter is set to blah We just found out that we don't really want to have this information this validation in the client And we'll instead decide on our users to use it directly So that way we would have less overhead with maintaining it and it would the code would be the code would be much simpler So how does it look? This is this is an example For for the suggest API. This is just a fraction of it and you can see that we have a link to the documentation We have all the possible HTTP methods. So in this case post or get We have the all the different forms of the URL paths with the description for each for each part So here there is only one dynamic part, which is the in optional index and We have description of all the parameters We also have a description of the body What it contains an information whether it's required or not and This is all the information I need to write or in my case to pre-generate and a Python method I have the name. I have the list of parameters I know which ones are required and which ones are optional so I can actually choose that these ones will be positional these one will be keyword and I Have the way how to actually put all those information together to create a URL and send it over to the server the last nice thing about this is It minimizes the effort to maintain it because we stuck it into the same repository as elastic search code bases The same repository where the documentation is and that meant that Updating this just meant that whenever I make a change in elastic search itself whether I add a new API or just add a parameter Inside the same comment or all pull request. I also provide changes towards the Specification and then all the client people all they need to do is just monitor this one directory on github to see all The changes that they need to implement in their client but Again, we we approach that the difficult thing like people need to watch something and do something and whenever you rely on people You will get into trouble sooner or later. Hopefully later, but you will probably get into trouble So that brings us to our last lesson test everything don't trust Just verify everything in our case we needed to verify that all the clients are consistent and That they work well with the server So again, we created our own solution and we created a unified test suite. We again took a machine parsable language in this case YAML and Created a simple simple test suite with a setup and a bunch of actions and a bunch of assertions That enabled the code to run not only against Alas you search itself, but also against all the clients So this is how it looks so this is again a test for the suggest API and you can see there is a setup that will actually Call an action called index with the parameter of index type ID and body and It will then do our fresh so it will make the document available for search and then it will There is one test the basic tests for such as API And it will actually perform the suggest operation and then run to assertions So this test validates that the suggest API is actually capable of correcting our typos and this is a test that is run as part of the Java test suite as part of the integration tests and So that means that it's version specific it's in the same code base So whenever you have a branch of elastic search it can have its own its own test suite just like just like any other tests and Also, all the clients have an interpreter for these tests That makes us sure that we have the same naming. We have the same API coverage We have the same exception handling because we can know just one assertions But we can also run assertions that this should fail with this error code and Just by specifying it once We can make sure that all the clients are consistent These set of tools together Led to that when we decided to develop the fifth client after the original four It only took a few weeks for the for the JavaScript guy to write an interpreter for the for the test suite and the parser for the API specification and Make sure that everything is working as it's supposed to So these were my lessons that that we that we learned during during this process It was it was good times. It was bad times But we we made it through and I believe that the clients are working well for people and now we're sort of approaching the The next stage and that is to actually create a more high-level opinionated clients that will be more helpful to and our end users But also for those clients. We are okay with people not using them So thank you very much, and if you have any questions, I'll be happy to hear them. Thank you So questions Hi, thanks for the talk. Did you try to think about if it was possible to Generate the clients completely Since you already have everything in theory you have the parameters you have the return value so Some clients actually do that for example the JavaScript client. It doesn't actually contain much source code It just actually internalizes that the JSON specification and it generates the method on the fly for Python I actually wrote a script to generate the entire client and I used it as as the first draft and then I edited manually because It's great to automate everything But usually it's okay to automate just 90% and don't try to catch catch it all and just do the 10% manually It's the classic 80 80 20 problem So I started with the generated code and then I filled in all the exceptions All the exceptions to the rule so now when there is a change I run the generation process again, and I manually look at the diff and and see what parts Represent an Actual change and what was just a manual edit that I had to do in order for the API to actually fear more like Python Okay, have you been trying or considering to use the protocol buffers from Google? So Well Yes, we have we have actually considered that as an alternative transport Currently we are fine with just HTTP and JSON though. We provide some alternatives There is an there is an experimental transport with a memcache protocol and Redis protocol and thrift We haven't looked that much longer on Protobuf because the the trade-off didn't seem to be that huge to to Warn that that investment. However, we are still we are still looking for more effective Transports and encoding schemas. So it might still happen It's definitely something that you can implement yourself in a plugin for both the clients and the server Questions no more questions Okay, then that's it. Thank you again. Thank you very much