 I'm from Delhi. I've been programming in Erlang and Elixir for one year, for some month, not straight, but more for learning rather than actual work. So, this presentation mostly inspired by my journey mostly because I wanted to participate in this project by Google Summer Upcode where I had to write a database driver. Although I wrote the proposal, did not get it through but along the way I learned a couple of things and I'm here to share with you guys. And the first for most, it's really honor for me to have a person who inspired Elixir community, Bruce, attending my talk. So, this is really proud moment for me, like right up there. Maybe right above the talk I gave at ElixirCon by Google Hangout. So, this will be the second one, like on the top. So, I'm a little bit nervous but once I get in the zone everything will be fine. So, how many people know database drivers like in detail, like how many know what's a handshake packet is? Anyone know anything about drivers? Because I was worried like most people will know about this thing and then talk will go through. So, I will try to build intubations and then we look at my prototype and then we look at actual production. It is written by Wastak. He's a core member of ElixirCore team and he's maintaining this driver. So, yeah, so my blog and my community at Delhi, we are small community, we're trying to learn and grow. So, before my talk I want to thank Erlang Ecosystem Foundation for sponsoring my talk. They have provided me sponsorship for my travel and also really thankful for this community. It's a new community and because of that it's possible for me to be over here and give this talk. So, thank you Erlang Ecosystem Foundation. Do check out if you want to build a community and you need funds and resources or people just for a workshop and they can give you guidance. So, yeah, what's my goal is one of the most powerful and one of the most beautiful things which I like about Erlang or Elixir is this pattern matching. It's everywhere, either in function, you pattern match and stuff. It's just idiomatic to do this way. It's encouraged and it makes super convenient to build parsers and all and as well as we see how it makes more convenient to build database drivers, which is quite complicated in object-oriented language. Look at the source code of a couple of drivers written in JavaScript and no slash node or some other languages. It's way complicated because to do the sparsing of binary data, it takes a really good amount of efforts and engineering to accomplish that. But in Erlang, from this talk, I'm 100% sure you can go home and can write a pretty decent prototype of your own driver for any database driver. If you know the protocol well enough, you can prototype really quickly. So, how many of you know what is the elf binaries and all? Everyone know what's an executable file format for that. So, I'll explain it that one and we built a very minor parser for that. We'll parse those and I'll show you a command which already exists on most of the Linux platforms. So, you can use that to inspect what is my executable, has the information system, 64-bit and all. So, you can get that information right using this pattern matching concept. And for this, I want to thank Woztec again because he's given me this idea, this is possibility and all the tips he has given. Thank you. Thank you. And then we will learn about GenTCP APIs because we are talking to some sort of server. So, you need those communication over the wire. We need this. It's right over the BSD sockets. So, it's just an API in Erlang to do that. And then we understand a bit about protocol and we try to convert those Erlang, the clients of a protocol packets, like handshake packets, OK packet and all to Elixir native types. So, binding pattern matching, as I say, is a very powerful feature of extracting bits and pieces of any big, large binary blobs. It makes it super convenient. So, for example, this is a very simple example. Let's say we have a string which is itself in Elixir is a binary data. And we want to extract maybe the functional part out of it. So, you can describe literally this is the intuitive part. This is the basic fundamental idea about building power. So, you describe your data that this is a binary. It has a certain this size. And I'm looking for this and voila. It's right out there. You are able to extract that out of it. The first able to bind to that string. So, we have different types. And two, if you want to describe your binary data, it could be an integer or it could be a floating point number or a binary. So, it has all certain size. So, you need to know the unit so that you can figure out how long that integer is or how long that string is or how long that floating. So, you need to know this thing, although it becomes really intuitive. And we come later to this formula. And also, you can, you want to be very specific. You want to do a specific, very specific. So, you can have on those integer data or floating or strings. You can further drill down to fundamental building blocks. Like it's a signed or unsigned or it's what is the machines. It's an Indian or little or big and so on. So, you can have these modifiers to further drill down to when you're doing your pattern matching. So, the first example we're going to look is a PNG file. It's a binary image. It's a binary object, right? It's a image. So, if you look, I try to read the image and this is the binary description of that PNG file. So, if you look here, this is 137. It's this number described that in each PNG file, this number will be key pre-occurring, which signifies that this is a type of a PNG file. Similarly, for a JPEG image, the first will be 255. So, now we look how easy it is to write a parser for that. It's right out from the elixir docs. So, for the PNG signature, you describe, okay, the PNG files has this description of binary. If the binary data correspond to this particular string structure, it's going to be, this is a module attribute. So, it's like a constant in elixir. So, here we are taking advantage of, again, the in function definition instead of the pattern matching. So, when you have, when you try to read a file, for example, you can have like image type or dot type and you can pass in, for example, a binary image of a JPEG. So, it will pattern match the first binary strings and then save it to pick it up that it's a part of, it's a PNG or a JPEG image or otherwise, so on. So, this is what I was talking about. Each executable files in Linux operating system or in Mac, even in cases of Windows, we have like in Windows, I think it's a PXE, PX something, which is some sort of a cover to have your binary, your executable right into it. So, for example, here there is a built-in command in Linux operating system called readElf and I'm passing this flag dash h. I just want to know the header information of this executable. So, these are the header information, for example. The magic is just what it is. It's just a magic number. There's no meaning to it, but it just exists even if you look just a minute. Just take a look into in more detail how ELF files are. This is from the Wikipedia. Any Linux executable or any executable, but we are talking about Linux executable, these executable files has certain sections. The first one, the header, which is only this ELF header part we are talking about. And then we have sections and all. So, this describe what's an actual native executable looks like. So, which composed of header. The header is parsed. The program header is parsed. So, what's a ELF is cleared. Everybody right? So, we can have information such as this application binary, what its version is. So, for this case is zero, there are a bunch of more for different operating systems. Maybe we can look. So, this is the magic number we saw in the previous slides. Zero, seven, ELF, this is the ASCII sense. So, what was the ABI version? Over here. This is zero. It's often set to zero regardless of the, because most, even over here, it says system V type executable. It's a Unix, not a Linux. I don't know why it doesn't show for the Linux, but it's often just what this is. This is for different operating system. For example, for the hard kernel, it's zero X04 to describe its API. So, I try to convert that executable right into a basic or a hexadecimal so that we can look directly into it. So, we can see from here what these information is right present into this particular binary string. Even when, we can't even look into it, but this is more clear because all of these information right built into executable, these are information is required by the loaders and all those programs which actually load those executables and execute that program. So, these are the things already exist in this binary. But if I tell you to write some sort of parser in C or Python, it will be really hard to parse this thing. It will take a lot of effort. But in Elixir, you can do it in like, in an hour maybe. So, I'm creating a module called ReadElf, just to read up just a basic Elixir version of this program. And I'm defining a struct all the things I want from these executable. I didn't try to go all of those because figuring out all those, how big sizes of those strings are is really difficult. But once you figure it out, you can do it in very succinctly or easily. But you have to figure out this is the size this is. So, it's very tedious but very intuitive to write it. So, I describe for example, a magic number. I already know it's a binary and had this particular size. So, it's going to pattern match to that executable. And I just took out only the magic number or Elf version and its architecture. And once I get those information, I could have this kind of function where I took in path of that object file and I can read that file and then convert that in string and pass through the parse function, which is going to do a pattern match on those and take it out all those information and I can look into all the structures and how all those information could be taken out out of the binary blob of strings of this particular executable. So, it's fairly decent but we could have more information over here where I can have all those information which I've shown in the previous slide which read Elf command which could do. So, you can have something like that. It is very basic version of that particular program in Eluxor. So, now this is the first part like binary pattern matching. It's right there intuitive. You describe you what you want to get. But you want to have a way to communicate with something over the wire. So, you need some sort of an API or some sort of a linking so that you can talk. So, there is like VSD socket API as it comes in. We all learn about this in college. So, using socket features which provided by most of the operating system whenever you want to communicate with one computer or another. These are the APIs which you use. For example, for socket, using socket you create a socket or which in Linux is treated as a file. So, you can read something to that socket or write something or write to the socket and it will be treated. And you can have this inter process communication between the two different applications on two different computers. So, using connect you can connect or you can listen to some clients and then you can accept those clients and you can send and receive. All these APIs are available in Linux in C bindings for this thing. But you need in Erlang as well. So, this is a very overview of how it happens. You have a server and a client and client want to establish some sort of connection. So, these are the flow of how things work. You create a socket. You bind that to a particular port or socket. You listen on that whenever someone tried to connect, you accept that. And each one of the client's server has to create its own socket. So, this server when it sends information can write that to that socket because it's treated as a file. And you can use basic primitives which is provided by Linux, read and write functions to really send information back and forth. And you can later on close the connection. Similar ideas is available in the Erlang APIs as well. So, using connect and accept and close all of the things you could achieve. Same what you could do. This is just a binding for those socket connections. So, this is very fairly simple example where you have your database. For example, MySQL is running up on your local server and you want to connect to that server. So, you could have something of, for example, the default port of the MySQL database server is three, three, six. And you pass these options and what these options describe. What most important is this active false. By default, if you don't pass this, what's happening is these, when you try to connect, the information the server sent is directly converted into some sort of a, not some sort of, it's converted into a Erlang message. You don't want to overflow your buffer which each Erlang processes has. You want information when you need to have it. So, you make a blocking code, blocking code call so that, hey, I need this much information. Can you send this back and send some information? And then using this socket, you can send and receive these, using these APIs. So, here's more in Erlang. Same example, but with the different, you can have something this way and you can send using this particular socket some data or close that connection. And similarly, so you can look up more information on Gen TCP APIs on Erlang docs. This will come handy later on in the talk. Why behaviors? You have no behaviors because there is another one behavior which is a custom behavior written by James Fish, which is important for having a long running types of processes, especially database type drivers to build. You need a special kind of behavior which is a DB connection behavior. So, what's the behavior? OTP design pattern has two main components, a generic part which is called server or specific part where you describe what your specific things you want to have. So, generic and specific. So, why it's important? You can have a separation of concern. There are things which can be handled by the beam itself by providing these features by, for example, Gen server, Gen state and to building state machine or Gen event, to building event handlers and so on. So, you just have to plug in your component, that's it, and you're good to go. Of course, easy to understand the code. If everyone start with the basic primitive and spawn and all and have a read those recursion loop where you have to read, it will be quite hard to understand. So, having a design pattern is a good idea. So, this is one example from a Francesco book which is a process of a skeleton. You have to go through this loop. One thing you will figure it out that the server and the client part which we have discussed is intertwined. So, all of these could be separated. But here in this example, so, which is not the case. So, you are doing your specific part as well as your generic part, right, into one big process. So, these are the differences of generic and specific. For example, you could be spawning a server, a generic task, but what should be the state of that server? It's something very specific to your use case. What's the loop data would be? What's the state it's going to have? So, that's a generic. The data itself is specific to your use case and so on. And these are like the basic differences to understand. So, gen stand, Erlang, OTP provide ways, types of OTP behaviors such as gen server, gen state, gen server is used for storing states and gen state for building state machines. So, you can easily build state machines and gen event and supervisor. So, the gen events and state items are like worker processes and supervisor is like a main role who can... Who let... So, the supervisor will supervise. It doesn't have to... It does not control... Other processes doesn't control supervisors. Supervisors control all other processes like, for example, gen server and all. So, this is one example of a gen server. So, this is the generic. Like starting a gen server with... Which is a very generic thing. Sending a message is a generic task. So, call is a synchronous. Well, the cast is asynchronous. What does that mean? We'll just know. This is from Elixford Docs. So, this is a gen server of a stack. So, you want pushing something to a stack. So, a stack could have an initial something. You can have whatever the stack. And, for example, when you send a call, it's going to pattern match to this. Handle cast. Handle cast. I think I haven't looked the code samples. Something is wrong here. But in cases of cast, it will be synchronous. So, it won't be a no-deploy. It would be a reply because it's a blocking call. While in case of a call, it's asynchronous. So, it won't be a blocking. We come to this behavior later on. So, let's take a look at the clients over how the communications happen. So, what's the structure of a basic MySQL packet? So, in most cases, a MySQL packet has, like, you could have information, something like that. You can have a big packet on first three sections. One could have, like, total is four bytes, like, for the information itself. So, four bytes plus packet body. And the first one can have the packet length. What is the size of this particular packet? And which takes about three bytes to store that information. And then a sequence number you could have. So, this is the first packet I'm sending. So, sequence number zero. So, something come back. The sequence number one. So, just to have an order of events. So, we have something called sequence number, sequence in a common basic MySQL packet. And it will take one byte. And the packet body itself, which could be N, whatever the N size would be, which can be described from here, like, what is the actual packet sizes. So, you would see, like, it's a three byte. So, if you do calculations, which means only 16 megabytes of packet could be sent only one at a time in cases of MySQL. So, you have to send, if you have a large chunk. So, you have to chunk it out in a 16 megabyte size. And you have to send it again and again. So, this is very basic structure of a standard packet. So, this is how you, for example, that arbitrarily packet, if you want to pattern match it. So, you'll describe exactly like, for example, the MySQL server, send information back in a big Indian format, which is just the order of the integers. But in little opposite, it's kind of confusing. But because on Linux and most of the 64-bit machines, Chintel has, it's a little, it's supposed the order of the integers are little. But the MySQL server send it as a big. So, you have to convert that. And you describe the payload length. So, 24, which means three bytes. You can calculate. So, 8, 8, 8, 3, 24. So, size into unit. That's the point. So, you can have different types of data. You could have a protocol have, like you could have informations in forms of integer or strings. And then we look at the connections and the command phase. So, this is a flow diagram. How everything happens. So, when you create a socket connection with your MySQL, so you have your database server, and you want to make a socket connection. Making a socket connection. And you are the driver itself. The driver is a term for a client, which is trying to communicate with this database server. So, when you try to do it. So, this MySQL database server sends a packet, which is called initial handshake packet. So, we look at the... So, this entire is a packet, which is sent for the first time when you try to connect over the wire using this Gen TCP. So, this is the packet you're going to receive. It's going to have information what the protocol version is. The most cases in major version of either Maria DV as well as MySQL, the version is 10. And for backward compatibility and all. But most probably you have a protocol version to be 10. You can have a connection ID. You describe what the connection number and all. And other informations. So, this is the complicated packet and you have to parse it. So, how do you parse this information? And this is described like protocol version is. And what the server version is. This information is a human readable version. And the connection ID and bunch of fillers. Just null bytes, zero bytes. And such as what are the capabilities, flags and all. And status flag. So, this how you write a very simple parser. It's going to parse that initial handshack packet. So, to parse the version number, you describe, okay, this is an integer. It has a size of 8. And it's of a little Indian format. And the connection ID you describe that, okay, this is also a little Indianess. And it's an integer and the size is 32. And you describe all of those components. And you take that binary. You have to see the packet you receive from the MySQL database. And you just pattern match. And each of those components such as version number, server number, connection ID, what the plugin part one is the fillers and character site is using. It's a UTF-8 or some other cities and so on. What are the status flag? What are the capabilities, flags are and all? And you just pattern match because you describe exactly what the binary data is. And this way you're able to parse initial handshack packet. Once you got the packet, you're able to parse, you're able to get all the information you need. Now you have to complete this connection phase. You need to send another packet, which is called a initial response packet. So we'll take a look at how this response packet is. So this is the response packet. So capability flags for if you're using client protocol version like 4.1. It means that if you don't use this, so if no longer data is going to send you another packet. In cases of success, it's going to send you an OK packet. We will look at what structure of an OK packet or end of the file packet. But because you don't want to use, you want to just use OK packet for all the communication. So you just pass this capability flags. And capability flags are nothing but a bunch of numbers. And you take capability flags like you can think of. Like in binary, in bitwise operators, you have a bunch of numbers. And you can take all the bunch of numbers. And in single number, you can have a lot of information stored. So for example, let's say one is representing some capabilities my SQL, I want to have in my database to have because my client is supporting. And two is another feature. So I take an over of those bitwise operation. And the number comes in. So database server, for example, my SQL server, able to figure it out. OK, these are the features, this particular server, able to understand. So I need to, when I'm sending back any data or something, I need to keep that thing in mind. So there is updated or not updated version of client. So there are a bunch of types which are like string or null strings and all. Let's understand all those types of information. And then we go on. For example, this is a fixed-length integer. You could have some component which has only fixed-length of numbers. Or you could have an integer where you describe exactly the length of that particular object or the packet. In that packet, the components. And you could have something null-terminated like having an end with 0, 0 to describe, OK, this is this kind of packet. Or you could have strings which are, you can describe like this string is 34 character and so on. Or end with a length string and so on. These are the protocol data types. And you can see, for example, in this case, these are all those. So for example, resolved is a 23 character. You already know, because you already encoded. This is always going to be 23. So when you're going to parse it, for example, let's look at the packet itself. So for example, string it, the odd-plugging data part 1. So because I know already, so I can have those information because you don't going to figure it out. It's already always going to be 8 bits. And for then the handshake response. So these are all the information. And the fields like capability flags, which I already explained like a bunch of numbers, you club them together. It gives you this information. Whatever the max packet size is, all those information is going to be in this handshake response packet, which is the client, which is going to encode into a binary and send it over the wire using the GenTCP APIs. And for that, for example, this, the capability first. Suppose these are the capability flags. I just don't remember what those things are. But you can look into documentation. And you know like, for example, this, if you or these all capability flags, and you can have these much. These are the features which going to be by default enabled whenever you try to communicate with your database server. And then what kind of authentication plug-in you are using. For example, in here is a shatoo or there are others. Or for example, in cases here, what are the user name? What will be the auth response? All those payload information. And you describe it and convert that into a payload, which is another, where you describe the entire thing and you convert it into a binary string. And you attach a sequence. So the first information comes from the database to the client. Has a sequence ID is zero. Just to look into order of events which come first. And then the things which comes afterwards has an order of one. So, and then, and you have this data and the sequence ID and club into a binary packet. And then you can send using gen tcp dot send and send it to the server and server able to decipher it. The underlying database able to figure it. Okay, these are the information has been sent. And so all of these information is, can be done and you can literally read every pieces of, and understand what's going on. Because there isn't lots of engineering is happening. It's very simple binary pattern matching. So this one is an okay packet. So if the capability, there is a client protocol 41. So it's going to have corresponding hex number for this particular, this client protocol 41. It's have a hex decimal. So if you don't order it out. So for ending of the file, it's going to send you end of the file packet. But if you enable this, it always going to send you an okay packet. No matter if the only okay packet will be further. If you have a database which is, doesn't have these features enabled. So you don't have to worry about because it just, it won't able to like figure it out. It just move on and take advantage of other ones. And this is a description of an okay packet. It's going to have information. For example, it's going to have an, of a header which describe, okay, it's zero, zero for success. And all the affected rows, what are the last inserted IDs, and these all the information. So this is the structure of an okay packet. Similarly with okay packet, we can have different types of packets. For example, if you want to send a command or something like select star or something for that, you could use different types of packets and you use same basic ideas which I have presented. These binded pattern matching and you encoded that and you send it over the wire. And it will just give you back and you parse it information. But this is all good for a basic use case for understanding and playing with. But for production use, you want to have your, you don't want to use this kind. You want, in production cases, your driver. For example, you're trying to connect and your database was down. So you need to deal with source so that you can have like, for example, a backup mechanism so that after two seconds it's going to retry each time multiplied. So these are the mechanisms are provided by the, so these are the mechanisms and APIs are provided by this library, the connections written by James Fish is provide features such as, it's just a behavior just like a GenSover or GenStatum and all has a generic and specific generics are all of the hard part like pooling. You want to have a bunch of open connections and to a database. You don't want to open if you want to, for example, query to a database server. You don't want to each time open a new connection and send that request. It will be very inefficient. Rather, what you're going to have is you have opened a bunch of connection at time, you have in a pool, you take out those connections, do your query and put it back. These are all the basic abstraction is provided by this particular library. It's similar with, for example, it's provided all of these APIs. So for example, you could have, you want to start your client. So starting the client using DV connection, then you can have like supervision and all good stuff and easy compatibility active, which is a upper layer where much better abstraction because you don't want to talk to directly to a database driver because it's a breakable further down the line, something changes. So, yeah. So these are the APIs you have to implement, which is specific to your database driver. For example, a Postgres for MongoDB or MySQL, these are the things you have to implement and it's going to have a return in a certain format. Okay, certain if something happens or error or something. So in a tuple format and then it can figure it out, you can execute or you can have, for example, you want to have a prepared statements and all, all of these. So these are the DB connection libraries allow you to do that thing. So I'm going to show the driver. I have a prototype at that time. So using this basic ideas, general lecture program. So we have a higher level module which describes all of those, for example, the start link function, which for example, if I want to start my driver as a process which underlying the DB connection which going to handle. So for that you can have these, the external APIs through which you start a driver and all. So for example, DB connection start and you provide what are the protocols and for protocol case, if you go and look, here you describe the connect. For example, I have just implemented the connect API and you provide those information and you send information for example, okay state or if there is an error and it's not compliant with the API itself, but you get the gist. And similarly, for example, check in and check out. So if you want to take any database connections out of the pool, you're using this API, you can check in those connections and you're done with your work and you send it back to the pool so that other processes should want to use that connection. And for example, similar with the case with, parsing the handshake packet and all and sending and receiving information. And all the sending and receiving over the wire happens, it's going to be happening over here. So we directly, when we are trying to connect, you're connecting and you're providing all the information and then we're connecting and using these functions, which is just a wrapper over the GenTCP APIs, connect and receive and all and sending information and so on. So directly communicates to the, our database server. So this is a very basic ideas of how you could have a very simple database driver really quickly and using very simple ideas and very quickly. But if you want to have something like my SQL kind of, which is much more detailed and much more robust and also very easy to understand as well. You can literally read the code and understand most part of it, which is one of the beauty of binary pattern matching. The same, for example, the production, the open source project has the same API. It's implementing the startling function. So you can have my SQL module.startling and start the MySQL driver and for querying and all. So these are all the high level APIs it's implementing. So this was the com ping, for example, you want to ping using your client toward your database. So this is what's happening here. You're sending the com and if you look at the implementation, so you can read most of it fairly easily because it's very easy to read and binary pattern matching allows us to have this complicated piece of hardware and it's less than 5,000 lines of code but if you look at other implementation for different languages it's like tens and thousands of lines but even in that case you cannot read it directly but here you can understand fairly most part of it what's going on because the simplicity of the whole Elixir binary pattern matching and then API and ecosystem the Elixir brings to the table. So to learn more I have collected most of the resources so this is the book I have read in 2018 like it's cover to cover it's a really good book if you are into understanding deeper level how database servers actually works inside the code you cannot read the entire code it's very complicated but yeah, this book is really good and it has a chapter on client server and communication the series article by Vostek the author of the MySQL library and he has described all those components which I have presented in my talk and you can watch the latest talk and there is Andrea's post on handling different types of TCP connections so he also goes on to generic and specific how those connections could happen fairly easily and you can look into the DV connection the documentation itself thank you very much and I want to thank you again the Elixir ecosystem foundation because without them I wouldn't be here so thank you very much