 I will present Article Privacy, Preserving Classification on Deep Novel Network. Today, machine learning is widely used in some many fields. For example, you can be found in medical diagnosis, financial analysis, or for sure, recognition. For example, I could manipulate sensitive data. And this is why today you would like to continue your machine learning while preserving privacy on Deep Novel Network. Today, these articles are used to preserve privacy on a Deep Novel Network. First, I will define what is a Novel Network. A Novel Network is composed with a perceptron. A perceptron takes some binary inputs and gives one binary output. Like you can see in the picture on the left, each coordinate is associated with a weight WJ. During the training phase, this weight is learned using a gradient descent processing. Each Novel Network has two phases, a training phase followed by a classification phase. And during the classification phase, you will assign a input for a new input X. And during the classification phase, you will apply perceptron rules. And the perceptron rules is a linear combination between the weight WJ and the coordinates of X. Then you will see if this linear combination is greater than or less than a certain threshold. Then you can see with this method, you only perform linear classification. In practice, you will use a multilayer perceptron. A multilayer perceptron is composed of several perceptrons. And in order to solve a linearity problem, you will introduce a nonlinear activation function, a sigmoid or rectify linear function. This function is applied on each wire. Then a convolutional network is just an extension of multilayer perceptron, which added some other layers. You can be found in the convolutional network, a convolutional layer, a weight sum. Then a maximum pooling layer, which computes maximum or average values of some of the components of feeding layers. And you can be found also an activation function like a multilayer perceptron. Then you place yourself in the cases where a convolutional network is already trained, and some parameters are fixed. Then the problem is used with 20CNN to preserve privacy during the classification phase. More formally, a privacy-preserving classification is when a client wants to be performed a classification on the cloud. And you would like this cloud doesn't get any information about the client's data or the resulting of the classification. In addition to privacy, you have to deal with two other aspects, efficiency. The running time to obtain the classification needs to be low for applied in the real world. On the accuracy, the classification performance needs to be closed, the classification performance without privacy. Then, first of all, the problem of privacy, a natural solution is to use homomorphic encryption. Homomorphic encryption ensures privacy thanks to the encrypted data and enables to perform several operations on encrypted data. In practice, the homomorphic encryption, the most costly operation in the homomorphic encryption is the multiplication. The efficiency, the complexity of a homomorphic function is equal to the number of multiplications you compute during this function. In the cases of convolutional network classification, efficiency is approximately equal to the number of nonlinear layers times the multiplicative layers. Because in the CNN, only the nonlinear layers are the multiplicative layers. Then, like I said previously, it multiplicative layers can be efficiently compute with a full homomorphic encryption. And CNN have two nonlinear layers, rectify linear function and the max pooling layers. Recent paper, Kryptonet suggests to replace the rectify linear function by a nonlinear low-degree polynomial function. The natural solution is the square function. And replace the max pooling layers by a sum pooling layer, which have a null multiplicative death. Then, to test your solution, Kryptonet use unwritten digital dataset MNST. In practice, Kryptonet has a good performance on MNST for a light CNN. The CNN is composed with nine layers with two nonlinear layers with a multiplicative death equal to one. They obtain accuracy equal to 98.95% on preserving privacy using full homomorphic encryption. And Kryptonet is able to perform 41 classification per hour. Then, like I said previously, Kryptonet has a good performance on the light CNN. But what happens when you're dealing with more layers? More layers are just a deep neural network. And for deep neural networks, the state of art teaches that in order to have good accuracy on the deep neural network, you need a rectify linear function. But the problem of the Kryptonet is replace this function by a square function. Then, the problem is the square function approximates the rectify linear function only as a small interval. And in addition, due to a central limit theorem, the input of this activation function has a normal distribution. And like you can see on this graph, the half of the input of the square function doesn't well approximate the rectify linear function. For a linear function, the square function needs to be within the interval 0 and 1. Then, you propose is not replace the square function but approximate this rectify linear function by a low degree polynomial and add a batch normalization before each activation layer. This batch normalization will stabilize distribution of the activation function input. And concentrate most of the input on the interval where the activation function is well approximated. Then, the accuracy of our solution is strongly linked to the quality of the polynomial approximation on the normalized distribution. In practice, to compute a polynomial approximation, you take a several number of a couple xa ya where xa follows a normalized distribution and ya equals to max 0 xa ya. In other words, you applied rectify linear function. Find a n-degree polynomial pn which minimizes the equation. On the picture, you can see the distribution of x when you apply the batch normalization. And you can see x follows normal distribution in 0 on the elements within the interval of minute 3 and the frame. On the picture, you can see several low-degree approximations of the rectify linear function. And you can see, with this method, you all use a low-degree approximation within the well approximation of the rectify linear function within the interval of minute 3 and 3. Then, in practice, in order to respect the accuracy, during the training phase, we will use a rectify linear function to apply the batch normalization. And then, you can train again in order to have better accuracy during the classification. We will use an approximation of value function on batch normalization. And then, during the classification, we will use an approximation of rectify linear function on the batch normalization. Now, to test our solution, we will use MNST2. And for a light CNN with nine layers and single nonlinear layers, you option accuracy equal to 97.95%. You can see this result is slightly worse than a Kryptonite, but in practice, for a more complex CNN or deeper neural network, for a deeper neural network with 24 layers and six nonlinear layers, you option accuracy equals to 99.30%. You can see your accuracy is too close to accuracy without privacy. Efficiency, you're able to compute 17 classifications per hour. And to conclude, the most important is that this work allows to provide a privacy-preserving classification on deep neural network because a deep neural network gives a better accuracy than a light CNN. And your accuracy performance is similar than a non-secure protocol. And the low multiplicative depth of our solution leads to practical efficiency in the real world. Now, do you have any questions? No. Thanks for your talk. On slide six, if you can quickly flip back. Yeah, so you have your list of things you need to satisfy here, and you have privacy in one direction but not the other direction. And it's actually tricky to define what you would mean by privacy in the other direction, right? So you have no requirement that the client doesn't learn the model, for example. So a trivial solution is just send the classifier to the client and have the client do it. Obviously, you don't want to do that. But if the client is allowed to give data and get classifications out, the client can learn the model itself. So what actually is the privacy requirement in that direction? So can you repeat again a question? Okay, so here's a simple protocol. The server just sends the client the model and the client just runs it itself. Yes. Right? Why is that not what you want to do? So I think they also want to hide the model. Right, so I mean that's not a stated requirement here. But if you want to hide the model, you have a problem, because if the client can just submit arbitrary things and get classifications out, the client can learn the model itself. Right? Just by using... So what actually is the un-stated requirement is my question. So you mean by adaptively querying many times? Yeah. Yeah, that will be dependent. I guess that's... it's an NPC definition and it doesn't... Thank you. Try to understand, right? So whatever you learn from the outputs, many outputs is fair game. Right. Yeah. Yeah. Right. But the IO is the function, right? Yeah. Okay. In the last example that you showed where you can support 17K queries per hour, what's the depth of the network that you get that you need to evaluate and what parameters did you use? What homomorphic encryption parameters did you use? A parameter is written as... So it's not my hat to go on the... It's not your hat. Yeah. Okay, so I think we have to take the questions offline because the authors are not here. Ah! We got to the bottom. Yeah. Oh, from this way. No worries. Okay. Yeah. Well, thank you. Thank you. Thank you. Thank you.