 All right. Hi everybody. Can you hear me in the background? Please do this and only me all alone here. I'm already stressed enough on my way up a purple Up a pill or two children a purple one. Okay. Who he got who here consider themselves to do devops every day. All right Who here has a team that does application the other is to send them to Okay, who here has a team responsible for their application their business value and another one Another team of sys admins that maintain servers and all well and Here is it because you don't work in informatic or Okay Okay, who here believe they won't learn anything out of this talk and by the way this talks is a beginner talk. Oh Oh, wow, nobody. Oh, okay. Cool. So the only one who thinks he knows everything good Okay Who here's only talks English who here talks French and only French All right, so I'll do the talk in English I'm open to French question and English question both doesn't matter to me All right So for tonight the second presentation or as my old manager used to say the low-level shit presentation Who am I I'm sorry, who am I I'm Eric Lafontaine I've worked for seven years at Bell Mobility as a network operator and Since the last two months. I'm now at Ubisoft doing exactly the same thing, but as an online programmer So the agenda the agenda for today will be just to present some concepts so that we are all aligned on what's going on What I'm going to be presenting the concept you need to know Before going into the demonstration that I'm gonna do and then by the end of the talk I'm gonna present some second tips and tricks. I've learned In my journey and then as a network operator and I'm hoping those tips and tricks will be something that will be useful In your everyday life at some point All right, so let's start basic. What's a file? Who here can at least tell me what the file the concept of a file Should be or what? Int a file would be a int File descriptors. He's not bad. He knows his stuff. So the file concept is pretty basic. It's surely it's an abstract concept It's just a bucket of bytes. You can read from it or you can write to it That's basically what if the concept of the file wants to be Sorry Yes, so it's a bucket it doesn't have to be in sequence it doesn't the implementation behind it is up to interpretation Do we deal with files every day or do we do we deal with another kind of object? Someone already mentioned it? Exactly we we try not to deal with the implementation of a file because otherwise We have to know the order of the bytes we put them in is it the big end in this or the low-end in this? Is it in this block of bite or is it in this other block of bite that I'm supposed to be writing to? It's super complicated. So we try to abstract the file to something else and that's why we have file endlers File endlers are going to be given to us by sorry The file and there is a implementation in the program. It's given to the program through it No as system calls when you do the open it goes and do the system call open and returns to you a file handler And that's what with with what you're dealing for writing or reading from a file this file handler is a natural pointer to a file descriptor in the OS and The file and there is used to abstract the flow of how to store data. Do you store it on an extension for system or a? extension tree or I forget the other one ZFS or XFS So in summary how computers work now a day when you do a F get see sorry I've stole this really no shame. I stole this from a presentation because I loved it and Basically when we do Oh, come on. Where is it? Okay there when we are normally writing our programs. We're staying at this layer here We manipulate strings of data, which is the equivalent of streams because a string is actually a sequence of bytes And then when we want to write to a file we use to we call the endals Which will go and call the right system calls the right registers in the CPU as well Which will go then call the right file descriptors who knows how the file is stored Which will call the right commands and data to send them to the right media So this is how today your computer works now. Who did I lust with this? Wow, I'm very surprised sorry Yes I've get see I believe is a command and see like a little you could well actually it's just I maybe you can help me with this Nope, okay It's to get carry on the stream level Yeah, you you you pass it a file handler and you say I want to get a get next character, right? Okay, so I've get sees a function where you pass in the file and I just forgot about it for this audience Because I got a blank sheet in front of me now. Oh Sorry so This is all the file handler would normally be at the low-level I you The file concept would be hidden somewhere between The system calls and the actual media But what about sockets sockets is kind of a special Type of file. It's not something that we store the bytes on a driver. It's a different kind of media. Oh Shit, okay I'm gonna try to go a little bit faster. Sorry, so if we had to describe what a sock It would be it would be the representation of a remote file So if we go and read on Wikipedia, what they actually say is A socket is a local endpoint representing a network communication path Which is really too complex to say just that it's a remote file that is local Yeah, it's really different So again, thanks for Sada Kazdi for this session So again, thanks for Sada Kazdi for this small summary And so if we now look at this Where would the socket be? Well, a socket would be something we call as a syscall to tell the OS where what path we want to take So that it can open the right file for us Then it would be asking that same file handler To handle for us the read and the write So it would be a file descriptor as well and It would write to the network cards, which has the commands and data transfer to send down the wire Does that seem any different from what a file would do? Yes, no All right So what would Yeah, but in the So yes, a socket is an abstraction every abstraction a leaky For now, we're gonna consider that you can It's gonna be a real abstraction, but it's never gonna be So now I'm leaving the concept domain to go on to the CLI. Please wish me luck. Oh And who here knows if it's my first presentation or not Because it is All right So I'm gonna present what we are going to be running tonight So we're gonna have this small monitoring strip that we'll be running in the blue stream This blue stream will do a clear sleep just so that we get a French prompt all the time Run it gonna run and it It's gonna run a net stat on the port 15,000 So that we can see all the connection being made to that port and then for all the Python Script that I'm going to start we're gonna go and look at all the file descriptors By the way, is that any Is that something new for some people here the fact that you can see the file descriptor inside of process? All right, so some people learned. All right, my goal is already done. Oh, yeah, sorry my bad What we are going to be running tonight is this server file and this client file We're gonna go through them don't worry Yeah, we're gonna go here How I'm going to do this presentation is I'm gonna run both strips into a Python debugger That way we can analyze on each steps that I'm actually executing what happens on the left side And now I just realized this might be small for you, and I don't know if I can zoom in Let's see. Oh, yeah, that's okay too big Yeah, a little bit better. Can everybody see Come in come in. All right now. Can everybody see? Good finally some integers and All right, I'm gonna zoom in here, and I'm gonna zoom in here and He did that Sorry, not used to Windows 10 yet Okay, now I want this should be enough All right So first of all what we're going to do is on the Bottom left side in red what you have is the server side and on the green you have the client side We're gonna be starting with the server because we do want to serve some connection So we start by importing the library socket that gives us access to the low-level enter and interface for sockets That comes with Python. Oh And that is ugly finally You know what let me start again Starting again So we're gonna import the library socket that gives us access to the low-level interface And then we're going to be looking at the first primitive of socket, which is socket This here is a primitive that is pretty basic when you want to create a file handler. You do have to call open So with socket it's kind of the equivalent but for network communication path The parameter that are now optional in Python 3.6 because they assume TCP are what type of Net layer 3 protocol communication do you want? That's here representing IP. They have weird name. Don't worry they are all referenced and Here reference to TCP as a protocol for streaming information and you can have other protocol to stream like a CTP or I Don't know how those protocols, but that's where you would normally specify what what type of socket you want Which protocol you want to use for doing transport? Sorry. Yes, you can have UNIX socket. You can have Something else than IP and Bluetooth. I think yes, Bluetooth is an AF as well You can put any kind of socket in there that are not something local to the The file system. All right. Sorry. I got lost in my mind So we're gonna spin this socket Wait, why is there two already? It is running new My bad Why is there two sockets? Yeah, but I got here two different Process being started by Sigwin. Yeah, but the red the green is not started yet. So it's not there should not be oh Nice a bug in a presentation. That's gonna be nice What do I do? Because this is now sad. You're supposed to see only one right now. I want to see it disappear. Oh, it must be a The debugger is the inside the process as well. I did that presentation four time and I never had that problem Now I got it So probably something with the on my side to look at later. Anyway, sorry for the delay So the server side here we have it. No, it did something again. Oh Anyway, sorry guys. I'm gonna start Anyway So we created this socket and it's this one here This socket is now a file descriptor in our process But that's pretty much it about it. There's not much other information that I for getting Now the next primitive we're gonna be looking at is bind bind is basically telling the OS a I want to reserve those Resources explicitly. Can you give them to me? So in this case, I'm gonna reserve port 15,000 notice that there's nothing appearing for the connection yet and At that point the OS inside of it knows that port 15,000 is reserved So nobody else can already use that port even though nothing tells us about it Now the next primitive we're gonna be looking at is the important one that people often misunderstand Listen tells the OS that it should accept connection on your behalf and this is really important to understand because Connection are going to be accepted whether you want it or not by the OS and The number you pass it in here is the number of connection. You're ready to accept in parallel And now what do we see? We do see that our socket is now on listen We see the file descriptor for it that a net status reading and telling us Now I could go to the client side and so I'll connect work, but let's continue here We got an accept call which is the next primitive on the server side to accept and receive a new connection From the queue that listen has What it's actually going to do in the background is going to give us a file descriptor and The information of the address of the client that connected to us So There's also another thing to know about accept. It is a blocking call In that sense if I have no connection coming in my program will wait in for everyone to come in Now I'm gonna go on the client side because we do want a connection now and And what we do is we have We import socket we spin our own socket as a TCP as well And then we have the connect primitive and the connect primitive is pretty much for by the same as bind But it actually goes On the remote end points being specified. So bind is a local point connect is the remote point and Remember them remember this because it's going to be important All right, so now I connected we see that the prompt came back here and Now we're going to be exploring the receive and send command at this point the connection is established on both sides We can see two more file descriptors here Hmm one for the server side and one for the client side if we look at the At the file descriptor themselves We see this one is the connection that we just established while this one is the client cluster the client connection file descriptor They all live inside their own pit And I really wonder why the four is there and keep you sorry guys So I got a question here that maybe some of you master guru knows what this is The number in there. I've been curious. I've been bitten by it and I wondered what it was Does everybody has an ID here? It's actually a more Closer to what a file is than anything else While the file has an I node sockets also That is the I know number and as I am in a sig win Interface so a seemingly network a similar kernel The number you're seeing here is since the beginning of my kernels been up. This is the I know they have in my kernel Alright, so that does the point of the connection and how it works now We can receive that on it Receive is a blocking call though and the number you're you have to pass into receive. What's that number? Yes, the number of bytes, but will I receive 4,000 bytes? It's a what it's the up to exactly It's the number of bite the OS is allowed to return to you So basically if you go you receive a packet of 10,000 bytes Well, you're just gonna receive the 4,000 first if you receive a packet of one bite Well, you're just gonna receive one bite. So this is what the number is here Is give me up to this number of bite you try to play with this number for getting a high throughput on network connections and application that is their goal here and It's a blocking call as well So on the other side we have the send primitive in which we said we tell it what we want to send and Then finally we send the message. We see that it's being printed out it here So we got the send and receive being connected from one way and to the other Did I surprise anybody with that? All right I'm gonna close the listening file descriptor So what happened now? I? Closed my listening file descriptor, but the only the other twos are still active Do you think they still work? Not what oh Yeah, and we could Let's see Could I do this? Well seems to have passed and not control C. That's not what I want It still works even though the listening file descriptor is not there. Is this useful to know? kind of That's how in genics apache and other program due to reload their configuration without going down and closing your client connection They just closed the listening file descriptor and they open a new one with a new configuration Nobody saw it. Nobody knows and it still goes on as if nothing happened So that's about it and now we're gonna close the connection Well, sorry the primitive close, but I'm guessing you all understand what this would do right? Sorry. I assumed All right, so this does a turn at how the primitive works, so This is what we've just seen the socket is creating the new communication on endpoint the local communication endpoint bind does the reservation of the local address listen announced to the OS that you're ready to accept connection and Accept actually returns any new connection to your program so that you can interact with your client and Connect does the remote connection Finally the send and receivers is sending and receiving data on the on the file descriptors that are there and close Release the connection That's does a turn of all of what socket should be And you surprise and question at that point All right So the first trick that I've learned while doing a network library client I had to do my own mock-up of a server In a telecom binary network, so I had to learn how to utilize socket and One of the thing I would have loved to know about is the fact that Before doing a connect you do a listen. That's like creating a dictionary a directory for file When you do connect you do connect before accept because that's like opening a file for writing versus opening a file for reading and Send and receive send before receive because otherwise you're gonna be waiting a long time for your data that will never come So this is a basic trick what it ends up looking like is like this You spend the server socket You bind to your local address You start the listening and then with the client. Sorry. I'm gonna have to skip here with the client You connect to your remote endpoint and at that point you can do the accept which is the Equivalent of reading a new file and then you will have your new file. All right send before receive This is the first trick I wish I knew the next one is a little bit more important and I think it's not known enough Who here you do g-unit kind of test cases the g-unit format test cases The unit test is based on g-unit for example. Yeah the setup and teardown thing Where you can have a setup here being done and then you have the teardown that will do the cleanup And then you just specify the test you want to do with those setups Yes, sorry As of now, it's has no relevance. Can I tell you later? I'm really sorry. It's not that I don't think your question is important. I'm limited on time Okay, okay, so Yes, board zero. Sorry If I'm using that format though, I may want to be able to execute my test case in parallel but if I use this Bind with a specific port what will happen? Well, my port will be reserved and my net next parallel test case won't be able to execute What's the solution? Well, you could go and implement your own port incrementer that's going to do the bind For you and it's going to increment from the port a range of port. Is that a good solution or a bad solution? bad solution It's a no glee solution don't never do that, please I've seen this and I did this and Saw this too many time One of the things that you might not have known is that the socket API does expose a port that is called the port zero This port is a special port It tells the OS give me a free available port that I can use to do my shit my things and At that point the OS goes inside of himself look at what resource he has available and gives you one. Oh, yeah So at that point after that whenever you are having a library client mock like redis mongo or whatever And you're having a mock for that service Pass it port zero it will accept it and then you can go with your client and Fetch the actual port the OS return it to you This really works. It ensures really well. It keeps the flexibility for you Later down the this later down the road where you want to do parallel testing and it will speed up by a good factor your test cases By the way TCP is a protocol that has a source port or destination port But we only did connect on the client the DOS did you have my okay? I'm gonna tell the answer right away. I failed on this one so Connect inside of it has an implicit bind the implicit bind is on address 0.0.0.0 Port 0 which tells you else give me all the interfaces a Free available port and I'm going to do my request later on and this is really important to know because there are source port Exhaustions that are possible to happens All right, and finally my last trick. This is my last slide. Can I get five more minutes? All right Who here has been dealing with Docker containers? All right, who here has been dealing with Dockers containers stripped down Like no Python no pearl only bash A few I thought Bell were the only one All right Let's go here. All right With all of what we've learned tonight and only a bash shell I Will magically go and do a remote connection without any telnet or any other program I'm gonna first execute on a file handler that I'm gonna define by myself and this This scenario. I'm gonna define the number five Zero is standard in one is standard out to a standard error Five should be a good number I'm gonna redirect all the input of that file the script that file handler I should call it to a specific file the input and the output and That file will be on a specific Pat Pat, sorry That bash is aware of and I'm meaning bash here because It doesn't work with the old shell that is 20 years old and that's sometimes what you find in containers But at least this would work At this point what I've given Bash is I want to do a TCP connection. I didn't tell it which host I Think this one will be evident and Which port I wanted to do you think this is gonna work Yeah, I know I won't type is the cell as fast as that So, yeah, I did a connection to Google at that point if I wasn't able to resolve Google Then I would have had an error on an execution for example That should not exist So we can see that the DNS resolution did not work. It didn't know what that service was surprisingly enough so If you ever find yourself having no way of testing your connectivity inside a container This is a nice trick to know up your sleeve You're gonna be seen as the wizard to understand how sockets work and it will help you resolve operational issue where your gray log Doesn't connect to an elastic search or your log stash And that completes my presentation who here learns something new Alright, thanks a lot everybody to have have come And I will take question later Okay Yes Sorry On Google. Yeah. Yeah, I could Did not return anything to me. Oh, yeah, I need to do again. Okay. Okay You need a first line All right, so I did it the question was can you do port 80 and get the response from Google? Yes, I can Next question go on Yes, oh God I'm a beginner at this. I have normally program in Python The question is can you do a listening file descriptor in bash? I think you can I just don't know how I'm sorry. I'm really beginner at this Me neither but I've seen things way too weird to something. All right last question. Yes Okay No problem, did you miss the beginning of the topic? Yeah, okay, I can come and see you after we're gonna go back to you All right, so I'm still gonna answer your question. What is the application for which I did this I Had to work in a telecom world with the Belmobility where a lot of protocols are old binary And you have to kind of buy a library from somewhere else that costs a lot While it's super easy to understand if you just read the specification and know how socket works I've avoided a lot of Stupid contract for Bell by just proving that this doesn't make sense and I can do it by myself here and Then you save a lot of money to your own company because you understand how things works in the background So this is the application in which I've used the most of those and I've developed a test solution based on the socket as well Around you. Yeah To configure the ports Which part I don't get your question. Sorry Yeah, I mean because you are trying to open ports Read them up Well, if you just want to send receipts Namely thank you