 Good afternoon everyone, my name is Rafi, I am going to talk about debugging a live Glacier file system using a special directory called Meta, so before jumping into the agenda I will introduce myself, I am Rafi as a software engineer at Red Hat, I have been at Red Hat for 3 years, almost 3 and a half years, I have worked in Commonance Lake RDMA which is Remote Direct Memory Access and Snapshot and then Tearing and then Replication. This is pretty much about me, let us see what we have for today, so we will have a quick overview of Glacier and then the architecture of Glacier as in like an overview because it involves like a demo, most of the talks involves demo, so I will explain the architecture a little bit and then I will introduce about the Meta Translator and then the basic commands it provides and then more advanced options through the metallic frames information, memory information and the graph information and the translator like master translator and then finally we will end up our session with the future enhancement and then question answer session. So as everyone knows Glacier is a distributed file system, it is a software defined network attached to storage, it has multiple nodes which participates in a Glacier to provide one single name space that can be accessed via Glacier native claims or NFS claims or SMB or we have written around libraries called LibGFAPA, you can access Glacier volume using either of this access methods and it works either through TCPIP protocol or RDMA protocol. So Glacier has a very modular architecture, so how it is achieved is using something called as a translator, a translator is a module which gets an input and it operates on that input and it passes that information and the input into the next translator, so all of the Glacier process or binaries are working based on a graph, something called as a graph, a graph is nothing but a collection of translators. So I have a picture here, it is a picture for Glacier mount process and server process. So what it does is, let us say I have file system operation, when a file system operation starts, for example a re-operation, it starts from the kernel BFS layer and then it will be given to the user space fuse process which is this process here and then we have something called as translator which is this one. So this is the graph we have, so the graph is a representation of multiple translator, so this process works based on this graph, so let us say a read operation comes on from the VFS, it will be given to the first translator on the graph and then it does its operation. For example in the first translator is the IO cache which is nothing but like the page caching, so let us say read operation comes on an offset, so the IO cache translator checks whether there is any page hit on already cached contents. So if it has a page hit, then it serves from here itself and then will be given the data into the back to the BFS, otherwise what it does is it creates a page fault and then it will be given the read call into the next translator and it flows from here to here and this is a distribute translator which does the distribution for the glacier and we have a replica translator which does the replica. Let us say we have a forward distribution which means you will have a for child for the distribute translator, let us say we have a three way replication which means you will have a three child for the replica translator, that is how it works and then the client process will be given to the server and on the return path let us say you read the content from this disk then it will flow from here to all the way to the BFS and then let us say for this IO cache it creates its source that newly created page and it works on the next read call it serves from this one. This is the basic architecture for the glacier and this is the translator model. So I explain this because when we debug the glacier we have to look into the different different translators what options they have providing and how we can look into the options, the file system specific frames and latencies all those things that is the reason why I explain this big architecture level information. So now talking about the translator which is the main talk for this so it is it provides a dynamic information about a mount process, let us say we have a fuse mount process that accessing a glacier volume but it does is it provides a dynamic information like the memory information, what are the file system operation is ongoing and what are the internal operation that resulted let us say read operation or let us say rename operation which involves a log internally so what are the operation is ongoing through the glacier mount process although all those kind of dynamic information you can fetch through the this special directory. So it is very similar to Linux Prokofas you might already know about Linux Prokofas it gives the kernel level memory information and process wise like the open FDs all those information for a kernel for a Linux operating system. So it is very similar to the Prokofas file system what meta does is in the glacier route there is a hidden directory called dot meta it is not only hidden but it is also virtual because which means you cannot see that directly even if you do LS or LS minus or you cannot see this meta directly because we do not want to return any of the reader IP calls this dot meta cannot should not be returned to the application layer because if at all you are doing a CP or R minus RF that dot meta directly should not be exposed into the top layer so that is why it is a hidden plus virtual you cannot see if you are just doing CD to dot meta you can you can you can go to the into that directly but you want see that directly in any of this leasing. So so that is and yeah so as I said it is a translator so rating a new feature in Glacier is very simple you need to rate a translator and it put it into the appropriate position in the graph. So this translator but this is a special translator it is not actually present in the graph but it is automatically loaded into every mount process every mount process in the Glacier it is automatically added so you cannot see in the graph but it is automatically added and this is used for debugging purpose and as well as the monitoring because what you can do is you can see them latency or latency for each of the file system calls you can see those how much is the latency for each translator through the meta translator and the due information this usage of for each servers you can see those information using this translator. So the good thing about is that you do not need to do anything you do not need to alter you do not need to send any signals to the processing you can just access through a directory file directory structure. So like you do not need to put a any a system type or GDP to get all those information it gives high level information like what are the frames for frame information for each calls. So I will do I will start with the basic commands for the meta translator. So I will switch into so I will give you the I have a volume here this is a very simple volume it is the volume name is called patchy it is a distributed replicated volume and it is a two way replicated and two way distributed that is two cross two these are the servers the volume is started you can see here. So the volume is started all the volumes are running so what I am going to do is I am going to mount this process. So I am going to mount this as mount minus to Glacier surface I will just quickly discuss this. So I am going to mount this into slash a mentee I am going to mount a Glacier volume called patchy into slash a mentee these are the mount options the event history on is means the eventing for the file system operation eventing is turned on for this mount process. So I am sorry it is already mounted I will first and mount so it is already mounted sorry. So this is the mount point there is nothing in here so now if you do LL you cannot see this dot meta directory LS yeah sure sure sure I will also use the this is the box. So if you do LL you cannot see that even LL minus LRTA you cannot see that dot meta directory because it is a virtual directory. So but if you do CD to dot meta it is now moved into the dot meta if you do PWD you can see it is currently in the slash a mentee slash a mentee in dot meta. Now we can see what are the information dot meta provides so I will start with simple things. So if you do LL sorry if you do LL you can see okay I will do one thing I will make it LL. So you can do this command line arguments memory information process latency UUID of the process version. So I will start with version so version is very simple it provides the Glacier version as a JSON format actually there are I mean ages there actually. So the package version is 31010 so and then you can see the command line arguments so you can see the command line arguments which is this mount process is started the process is user local has been Glacier of us and the argument is the history so on and the whole file server which is which means the server host name and then the volume ID which is the volume name and then the mount point this is the process which is started and then the information about the log in. So in most cases we wonder where is the logging file is so if sometimes you can configure it through the command line when the process starts or you can configure it with volume options but if you do not know you do not need to you do not need to search for this volume file what you can do is you can just go to the dot meta directory then there is there will be a logging directory here so you can just go to here and then you can see the sorry okay so here you can see a log file which is nothing but a similar to the actual log file so if you want to see the log file you can just go to the meta directory and you can view it so whatever it is to the log file so now you can see the log file so if you do not know about the log where exactly the log file is stored but you can just go to the meta directory and see the log file and another important thing about the log is let us say you have multiple client process for a Glacier volume let us say thousands of client process are there for a single volume which means you can have a log level for your log file like debug log trace log and you know a critical logs but you can also change it configure it like through a Glacier commands you can change that log level into trace or debug or warning but that changes the log level for all the clients let us say you want to debug a particular client process you cannot just make it you cannot just change the log level for a single process of course you can restart the process with log level as a debug or trace but it requires a restart so how do you change log level of a single client process that you can do using this log level so there is a there is a file code to log level so you can see what it has so the okay sorry so it is locking the system okay so actually it is there I do not know how to adjust it so the log level is actually 7 right now you can change the log level like the log level there is a log level number for trace it is 8 for info it is 7 so something like that so to change the log level what you need to do is you have to rate the log level number into the log into this file log level 5 so let us say currently the currently it has the log level of 7 which is written there but it is not it is not it is playing it but I will just do a demo like changing the log level into 8 which is trace there is a documentation what are the numbers for each log level so what you can do is you can change the log level through this interface so now the data is written and you can see the log level 8 here and now what are the changes it has is now if you select log file and below you can see it is standard printing that debug logs okay so that is one information okay I will quickly move so that is about the logging other things yeah the process UID so the process UID is each client process has a unique identifier so that information can be fetched through this process UID process UID this is the process UID for this client process how this is helpful is let us say you have thousands of clients which means the information about a particular event on the client through in the server logs is logged through using the process UID let us say if you take okay I will just copy this one and if you take by the server logs is stored here in the as a brief see if you search that thing you can see the logging like the logging for this client like it says accepted client from this mount point so let us say you have thousands of client there will be thousands of lanes like this that the connection is established establishment has already happened or the disconnect has happened with all with the process UID so if you want to know whether the connection happens or disconnection happens you just need to take the process UID and search that process UID in the server graph and also when client makes a log locking into the servers if you want to know which when especially when you debug a dead log kind of scenarios or a locking issue you want to see which client is has taken a lock from the servers so in that case the locking has been with this process UID so you take the process UID and search in the log in the locking translator which lock in which client has taken the lock so you can use the process UID for that for this kind of information the thing is memory information yeah that is very important so you can see the memory information through this meta interface so if okay I will go to the front open through this one yeah so for a Glacier process for a Glacier client process you can see what are the memories this Glacier mount points as used using this meta interface and that is through the map info file so it says what are the arenas it is already allocated the blocks the map information and yeah the free blocks and also from this allocated spaces for the memory for this process you can also see what are the data stretches how the particular data stretches how much memory it is it has already used it like this this is that this is a data type data stretcher which is used in Glacier and how much that says it is it has already consumed let's say you have a let's say you have a memory leak problem in Glacier process and you are debugging it then you can look into this for you can look into this information and see what is the maximum says what is the maximum says allocated for each data types and you can see when there is a memory leak you could see one data type of one or two data type is using a lot of spaces and that's why you can understand which translate which data type is using this information this is a global this is a global data stretcher and it's there is also a translator specific data stretcher that you can see when we look into the graphs okay and yeah so now I will introduce about the I will talk about the master translator then because we don't have much time left so this is the translator for the fuse fuse bridge so all the information about latency so to see the latency information what we can what we can do is we can start some operation on the mount point then I will do go into a different terminal and go to here and dot metadata and then master translator so if you can see we are here at the in the mount point dot metadata and the master translator and here what we have is a history what history is what are the operation as going what is the operation is happening on the mount point if you see the history you can see what is the operation is going on a fuse mod so it says for example it has a time and what is the operation is going on like on an I know sending an invalidation operation and this is an opend on this directory because we did a we did a opend into that we did a opend in the history that's the last operation though and you can see the other f stat on the Linux so all those operations the fuse operation you can see the history through this interface that's one and yeah so and as I said as I already told you about the the profiling information so you can see how much information okay I think the profiling has turned on sorry so you can see a measure latency things so you can you can enable the measure latency thing by writing one into this one so then now it will take some time to measure the latencies but let's see if it is already have if it already turned on you can see what are the yeah but it it will take some time so basically you can see if you if you turn on measure latency into if you turn on that option you can see each latency through this interface currently it was turned off so it is showing 0 the other things is the options the options of each translator so you can see the graph information through this interface which is there will be a directory called graphs you can go into graphs and see you will have all the available all the graph is placed in the graphs let's say a graph means as I said it's a collection of translator so when you disable a translator it is like a new graph because you have removed one node from the graph let's say you disabled a caching translator which means you have a new you have a new graph so that information you can see from this what are the what are the graphs the graph history and everything and what is the active graph so I can show you a demo for that like lesser volume set performance there is a translator called a rate behind fit sorry there is a translator called there is a translator called rate behind which does the rate in the background so I turned off that graph that performance translator which means the graph has changed because this translator is now removed from the graph so you can see the old graph information through the meta so see now there are only two entries earlier just one this Rafi now you have to because this is an active one which is a simulant to the already existing active graph now you can see one more graph is the old graph so if you go into the if you go into the if you go into the active active graph you can't see the rate behind here but if you if you go to the if you go to the old old graph you can see that information it's so if you see here you can see rate behind here but you can't see that rate behind in the latest active graph so all those informations like what are the current graph and the previous graph all those information can be fetched through this interface yeah okay so the okay so I'm winding up the session with so I explained about the info memory information like this is an explanation of the each lanes what each lane means says and the number of allocation and the memory informations what is each lane means so you can refer into this one and I'll talk about the future and then yes so what we are planning as a future for this translator is we are planning to give a future like a free and a drop cash through this better translate let's say Glacier has a lot of translator for caching so each translator stores metadata informations or caches metadata information so you can see what are the caching memory how much caching memory is used and you can also do a drop cash or you can also do a drop cash to drop all the caching from the Glacier mount process that is we are planning as a future we are and yeah okay I'm done any questions we have questions for Afi I'm sure he'll be around at the Glacier booth thank you