 Okay welcome everyone to the Saturday issue of the software defined storage step room thanks for coming if you want to see more of that next year give some feedback then we maybe even get two full days and in the meantime give it up for Ricardo for the first talk. Thank you everyone so today I'm going to talk about the the work we have been doing on the SAF project about rewriting the wire protocol that that is used in that project. This is an outline of what I'm going to share to you. We will start just to give an introduction about what is the SAF messenger and then we will briefly talk about the messenger API and show what are the the the current wire protocol limitations and why did we decided to implement a new one or design a new one and then I will give some more details about the design and implementation of the of the new wire protocol. Alright let's get started with what is the SAF messenger basically so it's a wire protocol specification but it's also the the corresponding software implementation of that of that protocol so when we talk about SAF messenger we are talking about implementation and the the the protocol specification. Normal users don't hear about this it's invisible to users right and if users hear about about this it's because something's not working properly okay and we don't want users to hear about the SAF messenger. Another important thing is that SAF messenger is not does not know anything about what happens with the the SAF with the other SAF components so the the the SAF messenger doesn't implement the algorithms used for for other components to to that perform like the OS the operations in OSD is months. SAF messenger is just the communication library used to for these components to talk between each other so where can we find it in in in SAF well basically we we can find it everywhere so every every component every demon it has this small code that is used to communicate between each other and also the libraries the client libraries also use have this code too so that they are able to talk with other components right so as I said before the messenger is the is the small communication library that is used to for the components to talk to each other it it can be used both for as a server or as a client it's it's the same it's the same code for both and it this this communication library abstracts the the components that use them abstracts the transport protocol that is used so it this this runs on top of POSIX sockets RDMA or DPDK and it it gives you it going to use a reliable delivery of messages and it it also gives you an exactly one semantics and besides that it also helps it also abstracts part of some temporary connection failures so if when there are there are failures at the tourist park level it it takes cares of reconnecting to the peers and making the more transparent as possible for for the for the the applications that use it okay so the API the API is very extensive but we can focus only on what's the important so the programming model for the the applications that use that use these small library they basically they connect to to another peer by giving an address and you receive the connection so very simple and then you can use this connection to send to send a message basically and you and you can also tear down the the connection you can also register a dispatcher where where you can implement some callbacks that will give you that you need to receive message so you you implement the callback to that that is called when you receive them a message so it which we call dispatch and you can also have a callback when you receive the new connection from another peer so this is a high-level abstraction over a communication channel okay so you and you also in the API you also take care of you have callbacks for taking care of the of the authentication of between the peers so until now we so safe has a has a wire protocol since the beginning right the components need to talk to each other and the current protocol was it is still the first one that was developed since the the beginning so it's it has a few years now and one of the the problems was it's it's not easy easy to to accessible to add additional states to to the protocol at least in an early stage so it's very hard it has a in the beginning there's no flexibility to change it it doesn't provide dot that authenticity so the payload of the of of the data that is sent through using this protocol is not possible to grantees authenticity there's also no encryption supportive and it's also limited in the support for different authentication schemes and and finish it's it's it also doesn't doesn't have a strict structure on on its own internal messages of the protocol so it's a bit ad hoc in the way that well it has evolved it's a the result of an evolution of many years of patches and yeah you know all these things work so so it was time so these limitations we we we were not satisfied and we wanted to design something from scratch that would that would could fix these limitations and so we started the design of the of the v2 protocol and so this this new protocol it's a completely it's a completely different protocol will will be available in in a different part so and we will still keep the v1 protocol so when we when we have this new protocol components will basically talk both protocols in different parts this is being this new protocol is being developed for the next safe release called not to us and only the user space libraries will will talk this new protocol for instance the kernel modules will still talk v1 and as as I said so not to us has not been released yet so this is is currently under development and we hope that we can finish it on time so the the protocol is a complete redesign in implementation it's it's designed from since the beginning to be extensive extensible in the way that we can change the protocol behavior see right from the beginning well almost right from the beginning and we didn't want to impose any limitations on the authentication schemes that we want to use with this with this protocol and we wanted to to fully support encryption on wire so what I'm I'm going to show you is it now it's a very high-level idea of how this protocol these new protocol works so to as a start we will have two actors the connector and the acceptor where the connector is always the entity that starts the connection and the acceptor is the one that is accepting the connection and and the protocol has four four phases it has the banner exchange which is the first one then the second the second phase is where the authentication between the peers is performed and the third is after the authentication they the protocol is stable is a session between the peers and and then as soon as we have a session between peers we can both peers can start exchanging messages so one important and sorry about specifying this with code but it was so much easier one of the units of that of this of these new protocol is that all all internal messages of the protocol are our exchange in inside these structures that we call frames and the frame is is is actually a very simple thing is it has a length it has a type so there are many kinds of frames and and then you have a payload okay and all the the the things we send between each other peers are enclosing in frames and the same thing for when we start to encrypt the payload we will basically have an encrypted frame which it look it resembles it's still a frame right has a length a type and only the payload is is encrypted okay so the the first phase in the banner exchange the the peers both as soon as the transport layer establishes a connection for instance in in in pot with sockets you have a TCP as soon as the TCP connection is established and the acceptor as as received they both concurrently send the banner to the two to each other and the banner includes besides the string saying step v2 it it includes two very important things to feature masks that are called the supported features and the required features and with this information of what are the features that each other's support and require we can decide what to do from from from from that point onwards so as soon as the the banner exchange finish we can actually our protocol can change so in the future if you have new features on the protocol we can we can change the state machine from from from here in so after the banner exchange they also they also exchange the information about what kind of entity they are so if they are OSDs or mons or our clients or MDS or the manager and and they also tell each other what is the address of the of the peer I will not go into details why we need this information but that's how it works right now so after this we go to the to the second step the authentication and the authentication is started by the connector so the connectors says to the acceptor I want to authenticate and I want to authenticate with this method and this method can be can be none so I don't want to perform any authentication okay or can be using SFX which is what we usually it's the the system we use for for authenticating the staff components or or can be any other method that we have in the future like Kerberos or something something else we also specify at this point what what do we want what guarantees of security we want after the authentication regarding for instance if we want the messages the payload to be encrypted or just signed or we just want some integrity checks using CRCs so the connector sends this message to the acceptor and the acceptor acceptor looks at what the client with the connector wants and it might reject so if for instance if the connector wants no authentication the sir the acceptor can say I don't I don't support that and and give gives the connector what are the methods that I I support okay and and the connector has the possibility to send again an off request message to the acceptor and as soon as the acceptor receives and accepts the other request it it can ask more more information from the connector depends which is depends on the authentication scheme that is being on and this this new information that the con the acceptor sends the connector asks the connector the connector can also replies to the acceptor and the acceptor can again ask for more information so we can have several round trips at this point if the authentication scheme requires it okay and this was one of the the limitations that we had in the V1 protocol which only one round trip were but not it was not easy to extend to get to have more round trips so the last messages the the entity that said that finishes this is the acceptor by sending an off done message to the to the connector saying okay the authentication is done and and we can we can proceed okay and from from this point if if we have chosen to to that we want our frame payloads to to be encrypted then everything from here session establishment and message exchange is is is encrypted so session and check it it's actually really simple so the connector sends some information that identifies itself and the acceptor looks at it and if everything is okay it just replies saying it gives its own information and there's also the other case which is when some peer is actually reconnecting so imagine a case where the there was a failure while the the peers were communicating and the the client can the connector can actually reconnect to the acceptor and at this stage instead of sending the whole information he just sends oh I want to reconnect to a previously existing session and the acceptor will have the job to to see okay yes there I know this session so you I'm going to reconnect to you to this existing session and you will continue onwards okay so very very simply very explicit protocol so after the session is is established we go to the to the message exchange and at this point actually there's no more a connector in the next after both our peers both can send messages concurrently in any in any order and they need and then the protocol also handles the the hacks of the of the messages the acknowledgments so you send they send messages the the acceptor some can piggyback acknowledgments in other messages and if there's if the peer that is not sending any message we we have a special acknowledgement message to say I already received up to that point so this is the high-level overview of of how this new protocol works so what so we for in this protocol we need to guarantee we we give the guarantee for we can give integrity authenticity and confidentiality on on the messages using the new one-wire encryption feature and in integrity is is given by CRCs in in the frame in the frame header and also CRCs in the in the message payload authenticity and confidentiality are given you only get this for the payload of a frame so tags the kinds of the frame length and the the the frame tag are not signed or encrypted and authenticity is given by by message authenticated codes using shot 256 at least for now but actually this protocol we can that we can specify multiple authenticity schemes and in protocols or even multiple encryption modes and confidentiality currently is is given by a yes encryption so the the source code if someone if someone wants to to look at it it's currently so you have to clone the project and you have in this file it's the the implementation of the of the whole protocol you also have a file called protocol v1 where you can see the the v1 protocol and you you can also see oh it's good but you can have also read the specification drafts of this protocol in the in the staff docs page so future features for this well we will like to add more kinds of authentication protocols supported especially carvers one thing that we have been talking but still don't have plans for it is the connection multiplexing so if you want to collocate for instance many OSDs in the same process we want to avoid having actually physical many physical connections per each OSD for the other peers and we want to reuse the same connection to the other side so for that we need to multiplex it and well this is an extensible protocol so new ideas and and contributions are are welcome basically and yeah we're we're done yeah so the question was what types of AES encryption do we currently support that's a good question because I don't clearly remember what we are currently using maybe CBC no yeah I'm just going to repeat briefly what Sage did said so yeah we initially were supporting CBC but now we're we're changing it to AES GCM which takes care of signing an encryption right yes so the yeah so the question was how does the the safe messenger actually is is currently working it's like you have a central right right so so in this here what to do we we have a we only we use non-blocking sockets non-blocking connections and we we rely on the on the operating system event system to be notified when when we are allowed to read when there's something to read on the on the sockets or when we can actually write so it's and so every every demon that's when he initiates the the messenger it's instantiates these these mechanism it creates the if it's binding creates a socket for for accepting connections and it changes to non-blocking mode asks the operating system to start to receive notifications for for that connection and that's how it works well I oh yeah sorry so the question was if with this new protocol the the safe messenger is is now a good fit to be used in a public network for for the components to communicate in a public network I really don't know what to answer to that question because I I don't know what are the the the threats that that that was not the target okay yeah and yeah I don't know if I would I don't know okay it's I don't have enough knowledge to to answer that question from this point of view yes well that's so the the question is why adding the overhead of encryption if we cannot use the safe mention safe messenger on the public network well you might have some requirements security requirements that all data that is so imagine you're you're storing user data in in the data center and you have some requirement that the that there must be always confidential because you might have some very curious sees admin that starts seeing science leaving packets in the in the in the network so inside people and and you have the requirement that you must guarantee that users data is actually it's confidential and no one can actually see so for the only way to achieve this is if you actually encrypt all that even in a private network so the the question is if the messenger can can be configured to to the to to configure it in a way that we can specify if we want a secure connection only for some links right the problem is how do you characterize a link that's where I'm struggling you right yeah yeah so but overall the thing is that currently we can only distinguish between type of peers to change the configuration so you can say that between yeah between your OSTs you have this configuration and but it's with the type it's not individual and instance of the demands which one I don't know okay yes and and that you that you can do because you can you can say so every client is required to it only works if we encrypt if it accepts to encrypt the messages and for other other types you you say no you I also allow to not to do that hello so the the question is why didn't we decided to to also encrypt the frames text in in the protocol it's a good question because why didn't we do it okay all right so sage is saying because I was not a present when the protocol was was designed but at the time we were thinking of implementing the move the connection multiplexing which I talk about in the future features that was supposed to be implemented in with the new protocol and at that point we we needed to there was no way to if we prep that information we probably wouldn't be able to distinguish some some messages so yeah they need they need to know who from from home comes right unless you do encryption on top of encryption but that way we are you are actually doing a SSH tunnel so the so the question is if we if we have in the future when we have support for more authentication protocols like Kerberos if we have some guidelines to help the admin to choose which one to use I'm sorry but I'm not I'm not that I don't know so so much about these these security protocols to you to answer at this moment that question well actually suffix is it was based in Kerberos so I don't think there there's too much of a difference between choosing the current effects and Kerberos it's just a matter of the if you already have the infrastructure everything use Kerberos and then you can actually leverage it in in the set instead of having one more security protocol to manage and manage keys and manage access so the the rationale for for having Kerberos as as an x-step is because if you have that infrastructure already it will ease you a lot right because you already have the infrastructure to manage it so