 Good afternoon everyone. My name is Noa Jarr and my name is Aaron Beer and we're both part of the Tech Lab team at PostFinance. PostFinance is the post bank of Switzerland. And yes, let us share with you today our insight into a production-ready hyperledger fabric certificate authority. Yeah, to begin let's look at some fundamentals first. In the upper part, in the yellow boxes we have some concepts related to public key infrastructure and underneath in blue and the implementation hyperledger fabric and now we go step by step through those terms. First one is identity. In hyperledger fabric it's very important because it's a private permissioned blockchain so every entity participating in the network has to be identifiable. This identity is used to produce signatures. They are to ensure the origin and integrity of a message and also for TLS transport layer security to answer the question who is my communication partner. Continuing with asthmatic cryptography also called public key cryptography. This term consists of a key pair which consists of public and the private key. This key pair can be used for encryption which we will not go further into in this presentation but very important digital signatures and there we have design. We do this with the private key of the key pair and this can only do the admin or the owner better said of the key pair and the verify procedure of digital signature can be done by every participant of the network with the public key. Now it comes together in this public key infrastructure term also called PKI for short. This is like the assignment of an identity to a public key and the identification is done by trustworthy authority which issues digital certificates so we call this authority the certificate authority the CA. The digital certificates contain information about the identity attributes, the public key and the signature of the CA. By ITU there is a standard set which is called X.509 which we will use in our solution. There is also an important part the hierarchy of certificate authorities which can PKI can be built on and this is like built up by a root CA and intermediate CA and operative CA's which builds those CA layers and those layers represent the chain of trust. And now the implementation of hyperledger fabric and there's the hyperledger fabric network which consists typically of organizations which are companies participating in the network and they belong and there are identities which belong to those organizations and they can be type of peers, orders, clients and admins. Next we have the crypto material this term includes all cryptographic artifacts of an identity containing the key pair consisting of public and private key, the digital certificate and the chain of trust which contains all CA certificates up to the origin which is the root CA and those two combined like the PKI version in the implementation of hyperledger is the hyperledger fabric CA which handles the whole life cycle of identities and crypto material so issuing with registering and enrolling identities and also renewing and revoking of certificates is included. And there is the global MSP. MSP stands for membership service provider and this is a directory structure which is built and the name is very misleading because this MSP doesn't service or provides anything actively and it's really only a folder and certificate structure. The global MSP contains all issued crypto material for an organization and the global MSP contains all local MSPs. Local MSP contains all crypto material for a single entity. This local MSP can later be distributed to the network's nodes and now some definition to improve the understanding of the rest of the presentation. So when we talk about an intermediate CA we mean every CA except the root CA. When we talk about an operative CA we mean an intermediate CA on the lowest layer and this CA manages the entity crypto material and when we talk about signing CA we mean an operative CA which only issues certificates for the usage digital signature. Yes so now let's dive right in and see how we got started with Hyperledger Fabric Certificate Authority. We started off with a simple fabric blockchain network with which we wanted to go into production mode. Now we had two peers and three orders running and these ran on Kubernetes and communicated with mutual TLS with each other. The cryptographic material was generated by Cryptogen. Now you may ask what is Cryptogen? Well Cryptogen is a tool developed by Hyperledger Fabric to generate initial key material to then start a Hyperledger Fabric Network. The tool consists of a binary and a config file. You see the config file on the right on the screen. This is a very simple config file in which we can configure things like for how many peers or how many users we want cryptographic material and that's where our problem arised. Now we wanted to go into production and in the read the docs it says for Cryptogen it would normally not be used in the operation of a production network and the tool is provided for development and testing only. So we wanted to go into production but had Cryptogen in use. Big problem. Now Cryptogen is like a one-way ticket. It can easily issue crypto material but there are many other features that are necessary for a production network that it can't provide. Yes then we had a closer look at what the PKI has to provide because Cryptogen plays this part in our network and like the iron said the one-way ticket is the issuing of a certificate. We see this in this a little bit complex illustration of the life cycle of identity and certificate. So now Cryptogen doesn't meet our requirements and so we built this diagram to have a closer look at what our solution has to provide. So what is this in detail now? Very important is that the identity is connected with the certificate and because of Cryptogen we only issue certificates itself. There's no identity part and also the rest of the whole life cycle of certificates and identities has to be covered. So we searched for product that fulfilled this criteria and we found out that Fabric has his own CA solution which is called the Fabric Certificate Authority which manages the network's identities and can issue revoke and renew crypto material. So there are also two binaries that come with Fabric CA. The first one is the Fabric CA server which can run a CA server with different options that we can configure in different ways and has a database connected to persist identities and crypto material related stuff and then we have Fabric CA client which is the interactive command entry and to communicate with the Fabric CA server. Now if we compare those two solutions Cryptogen and Fabric CA we can see that the features included in Fabric CA are like we expect that in the production network but we already thought that this would require more knowledge and more work to implement this solution due to the higher complexity. So nevertheless we decided to implement Fabric CA and while researching we came across many clever concepts but however we had to overcome many obstacles during our implementation. This was mainly because detailed examples of the implementation were not provided on the web and especially regarding the Fabric CA hierarchy we wanted to build. There was not that much like tutorials or detailed examples so we want to share our solution now with the community. We found several options regarding the hierarchy in the read the docs and we discussed them as a team and I will present to you them now. The first one is like how we built the layers of the hierarchy. As a first option we saw the possibility to just run a Fabric CA as a root and at the same time as an operative CA so every entity crypto material is issued by the CA. The next option in the middle is a root CA which issues only the crypto material for the CA's underneath and can be offline afterwards after issuing those crypto material and the operative CA underneath which issues the crypto material for the rest of our network. There's a third solution this is quite similar to the second one but there are one or multiple layers in between the root CA and the operative and with this we can extend like the chain of trust but our team committed for the second option because it's like the multi-level architecture or hierarchy we wanted to build to generate the chain of trust in case of compromised net to a crypto material we would handle this way faster and didn't have to have to exchange the all crypto material and this is not that complex compared to the third option which is similar but this is like would be an overkill the third one. Then by separating TLS and signing CA was also another decision we see this in option two we would build two operative CA one for TLS and one for the signing certificates and the other option would just be provide those two different certificates in the same operative CA and again we vote as a team for the second option because with this we can build a separate chain of trust for TLS and signing and handle the certificate types independent so in case of using an HSM a hardware security module and this can be done in the configuration far easier and another decision we took by choosing one of those option the first one is to provide a signing CA and the TLS CA for our whole network and the other one is to separate those peer and order stuff and providing a separate CA for each of them and as a team we decided to go with the option two because we thought this was more reasonable and then to our setup in detail and first we use open SSL to generate a route we do this by generating a self-signed certificate which comes with a private key of course then we have this second part where we issue the CA crypt material for all our operative CA's and this is also a digital certificate and a private key and in the addition to this we also have a chain file which represents the chain of trust which consists at this part only of the root CA certificate to keep our illustration a little bit simpler in the following slides and we mark this with the red folder so now if we look at the total or the full setup that we generate with open SSL we see that next to the root crypt material we have the CA crypt material for all our operative CA's which are listed here and then we can continue with the CA fabric CA servers so for this we prepare first the TLS CA server and because if we choose to split signing a TLS CA we first have to bring up the TLS CA and we do this by mounting the crypt material from open SSL into the container and configure this and afterwards we can launch the fabric CA server and the TLS certificate with the private key is generated automatically by the server again for the next step we use another color this time we use this orange folder to mark the TLS crypto material with the TLS CA server running we can continue with the signing CA server there again we just prepare the server by configuring or mounting the CA crypto material from open SSL then we enroll the identity for the signing CA server we register the identity first of course so that we can enroll it on the TLS CA and we get from this TLS CA server the TLS certificate with the belonging a private key so when we have this we get we have to configure of course the TLS section also and then we can finally launch the signing CA server so now to get a quick overview after those detailed steps we look once again at the open SSL part where we issue a self signed certificate then underneath the the CA crypto material containing the chain file that sit digital certificate and the private key for every operative CA then we continue by configuring the TLS CA also of course for the peer and order like we have chosen in our decisions there first we mount the open SSL CA crypto material and configure it then we can already launch the TLS CA server and the TLS crypto material is generated automatically then we continue with the signing CA's also for peer and order there first we also mount the open SSL CA crypto material register an identity for the signing CA on the TLS CA and then roll it so we get the TLS crypto material and last but not least we are ready to issue identities and issue the belonging certificates to the rest of our fabric network yes and how that works let's look at it step by step first of all we have a fabric tools container that's a simple container which contains the fabric CA client binary and with this container we communicate to all of our CA servers this container also includes our global MSP and all of our local MSPs included in the global MSP of course you see here two examples of local MSPs there will be all the others as well now it is very important to mention that we have one single database for both TLS and signing CA that means we generate a crypto material for the same identity for both TLS and signing which also means that it is much easier to manage for us in case of compromised crypto material now let's look at the step of issuing and entity crypto material step by step first of all we have to register an identity we take as an example a peer so we register a peer identity on which CA we do this doesn't matter because once again the CA's share the same database so we can do this on the TLS CA for example then as the next step we enroll this created identity on the TLS CA then the TLS CA persists that identity in its database and that's the next step we can enroll this identity yes I was in enrolling sorry so the TLS CA doesn't it already has persisted its identity in the database so now it persists the crypto material from this identity and in the fabric tools container we get a copy of this generated crypto material as the next step we enroll the exact same identity on the signing CA the signing CA persists that crypto material that was generated in the database and once again in the fabric tools container we get a copy of this generated crypto material now it is important to mention that we have the TLS crypto material saved under slash TLS and the signing crypto material saved under slash MSP the next step is to organize and rename all of our files contained in this local MSP how we do this in detail I'll explain I'll explain later so now we are ready to configure our peer to mount this local MSP which is ready and when it is mounted the peer can be launched it is important to mention that the same exact process works for the orders as well the peer is just an example so now let's look at the organizing and renaming part of our crypto material as an example we have this fabric CA client command to enroll the peer one identity on the CA peer the m flag defines under which path the the generated crypto material is saved in the fabric tools container now when we execute this command we get the following file structure we have the sign search folder with the generated certificate we have the key store folder with the corresponding private key then we have the intermediate search folder which contains the chain file and the CA search folder which contains the CA root certificate now we do some renaming to make things easier for us first of all we rename the private key to a name which isn't that cryptic anymore we add a minus chain to the name of the chain file so we know it is a chain file and we remove the ports from the file names so because we think they are unnecessary for our case at the right you see the the final local msp for the signing crypto material yes now while setting up this structure we presented we had a lot of learnings a lot of important learnings and the most important ones we want to share with you right now first of all we noticed that Fabric CA has a configuration hierarchy for almost all config properties we can define them in three ways as an example when we want to define the CA certificate for the CA server we can do that first of all by providing a flag in the Fabric CA server start command we can also do it by setting an environment variable in the Fabric CA server container or we can define it in the config file of the Fabric CA server and now there is the hierarchy that is important to detect the flags overwrite the environment variables and those override the config files so when we specify a CA certificate when starting the server in the with a flag all all the other prop the same property in the other ways to configure it are overridden yes another learning now in the Fabric CA server config file we are here in the signing section of this config file and it's about the attribute max path length this attribute is specified by a number and it says how many CA certificates can be issued below this layer of CA that means a number of one in the max path length attribute would mean that no CA certificates can be issued underneath the CA this CA can only issue end entity certificates now a simple example when we have a Fabric CA hierarchy with two layers a root and an operative layer we would choose the value of one in the root layer and the value of zero in the operative layer and in our example because we have no Fabric CA server as the root we generate the root crypto material with open as a cell and we only have one layer of operative CAs we dare choose the value of zero for for the property max path length now another attribute in the same signing section we have these different profiles we can enroll certificates with we see here the default profile and the CA profile and there is this attribute called is CA and this attribute defines if the profile and enrolls a CA certificate in the profile CA there is this attribute default set to true which means when enrolling an identity with the CA profile it enrolls a CA certificate but that's exactly where we made a big mistake because we thought when enrolling with the CA profile we get a signing certificate which isn't true for getting a signing certificate we have to enroll with the default profile or no profile at all because it is the default one and then we get a signing certificate because the usage of the default profile has the digital signature and one last learning which was very important for us is the usage of the chain certificates now we never really knew where to use the root CA certificate and where to use the chain file and it turns out that everywhere in the order and peer conflicts where there is a root certificate wanted you have to specify the chain file and not the root CA certificate in the documentation that isn't that clear that clearly mentioned so it is important to take away and now let's take some conclusions over all of this first of all we asked ourselves if this was really worth it and we want to be honest for us the step from crypto chain to fabric CA at the moment has hardly paid off we invested a lot of time and hard work and at the moment our network just runs the exact same like it did before like there's no big difference evil but and this is very important we no longer only have a one-way ticket so when we are looking into the future we experienced fabric CA as a reliable technical mature solution so in case of an incident with our crypto material we are now ready to react as of before we weren't so and that's worth very much we have to say and for all of you it is definitely worth it because the step for you now has just gotten a little bit smaller but talking about the future there is one important topic to mention that has to be studied before you can go into production mode with fabric CA there are many other yes the topic is managing the entire crypto material life cycle in this presentation we covered the issuing part of the certificates in all detail but there is also the part of revoking certificates generating certificate revocation lists making channel updates and re-enrolling certificates which really had have to be studied but we have no time in our presentation to cover these as well yes we just presented to you our whole journey and what is what it all was worth it so we asked ourselves what can all of you take away from all of this and we thought well there are many things but nevertheless we want to tunnel down on one specific key takeaway because while we were working on this presentation we came across a fundamental insight for us that explains most of the problems we faced while working with fabric CA and that is most of the problems when working with fabric CA are linked to other topics and concepts just like at the beginning of the presentation in the fundamentals part we had to explain to you the different concepts so you would understand the rest of the presentation when working with fabric CA you have to have deep knowledge and understanding these topics to not run into problems we experienced that we can't just read the docs and be good to go and that's exactly where we lacked because most of the time when we understood the topics and concepts all of our problems were almost solved immediately yes and we have decided to contribute we believe a contribution to the reader docs doesn't fit as well because like mentioned information there is really scattered and like mentioned in our key takeaway there are many other topics that should be known but have no place in the reader docs so what we want to do is provide this presentation as like a full on cookbook for the topic production ready hyper ledger fabric certificate authority server hierarchy yes and we are clarifying if we can contribute to a simple version of our fabric network to the fabric samples github or a similar github page yes that's it thank you for your interest and we are happy to try to answer your questions so are there any questions we have like 10 minutes left yeah alex yes alex asked if we combines those servers in one server instance or we have those running like parallel and it's the second option we decided to run them by using single server instances for every ca yes yeah the question was this time why we don't want to contribute to the reader docs um a tutorial would also be useful and we think that is right we tried to to search for those like parts in the reader docs and we found multiple very good examples for the ca a fabric server and fabric client binary how they can be used but like for a tutorial of fabric ca we struggled to to find like the space in the reader docs because fabric ca's when we run this in a single instance it's very well documented we thought that we can develop or at least work with this documentation very well but like the hierarchy is it's like never mentioned clearly in the reader docs so we decided or we thought that in the samples it would be more accurate but if you can show us the section where you think this would be accurate okay that's good okay nice to know and we are happy to contribute at all yeah yes so this exact solution we presented we have now running for half a year or so so not that long but we we researched a lot of stuff about re-enrolling generating certificate revocation lists and what to do in case of compromised material so we are ready in a case of anything would happen but yes the struggles we never really experienced until this point yeah we can say we have done a cron job running on Kubernetes to find out if a certificate will expire soon so not a single certificate is expired by now but we can test the whole exchange with revoking a certificate and we have done this and all the pain which comes with like Aaron said in the presentation with the renaming stuff at the very end but before like the revoking the creating the revocation list and channel config updates like we mean we managed to do this by now but yeah the pki is it's not completed anyway soon it's running to the end of the life of this solution yeah yeah the following question yeah the topic at this time was the connection of identities related to between Fabric CA and Hyperledger Indy and yeah we're pretty new to Indy so we like implemented a solution with Fabric CA so we're kind of new and if we want to discuss with some other contributors we're open to do this yeah okay they have a production ready Indy in Geneva so that we're located in Bern it's like not that big distance yeah Alex once again the question was if we came across using the private key as a password with a password exactly and this is a topic we didn't cover actually and I heard it for the first time so I never I've never seen anything like that in the Fabric CA realm yeah it's important to mention that we issue identities and the crypto crypto material for like technical users so all our peers and orders running they use our crypto material and like all client stuff we handle with our partner which provides the front end network and we only handle the fabric network so we issue like technical identities well we issue also identities for the clients which communicate to us but these clients are applications and not end users yeah yes we got it to work but the question was if we ever got re-enrolled to work and we got it to work but we didn't cover it in this presentation because we had no space for it yeah we did it by revoking a certificate and expiring certificates we we had not until yet because we're running this since half a year yeah if that's it we're happy that you were here and thank you for interest once again thank you very much