 Good morning everyone, welcome to my talk, Noble evaluation metric for deep learning based central analysis and its extended application to imbalanced data. Before I go into detail, I first want to give our motivation. Deep learning paradigm has received a significant amount of attention and shared great potential recently. However, deep learning method is at first new to solve the classification problems. As previous works pointed out, such an analysis and classification problems do not share the same objective. Also some work have noticed that there is a gap between such metrics like guessing entropy, success rate, and deep learning metric accuracy. Besides, recent works show deep learning based such an analysis suffers from imbalanced data. In this work, we want to explore the relations between deep learning and such an analysis and try to give a better evaluation metric for deep learning based such an analysis. In this process, we also propose a good loss function which solves the problems of imbalanced data. First, I give some basic definition useful for this formula. We define cross entropy between the true distribution and the predicted distribution given by the model as follows. Here x denotes the leakage and y denotes the labels. In practice, when we use cross entropy loss, we are using the estimation of cross entropy on a sampling set. Let t be the training date with label and m set as a model. Then the cross entropy loss is defined over the training set and the model. Since cross entropy loss is an estimation of cross entropy on a sampling set, by the law of larger numbers, we have cross entropy loss converges to cross entropy in probability. When deep learning based opinion attack penal procedure is like this. We first collect our such an leakage or traces. Then we calculate the sensitive intermediate with the key and public variable. Transforming the sensitive intermediate to a label using some labeling function f. Next, we feed our model with the traces and the labels starting the training process. After training, we use maximum likelihood method calculating the scores for all key high processes and discriminate which key is most likely to be the right one by comparing the scores. This equation shows how to calculate the maximum likelihood score. Here we can see why accuracy is not a good metric for such an analysis. Accuracy is designed to show the performance of the models are a single prediction. However, in profane attack, we use maximum likelihood method where a set of traces related to the same key are given. Accuracy feels to tell the underlying mechanism and concentrate only on the labels. There's also where the objectives of classification and profane attack have conflicts. Because when conducting profane attack, we are not trying to find a model giving excellent results on a single input. But we hope the overall results on the whole attacking set can help us retrieve the right key. Deep learning method designed for classification does not know this and thus it may not be an optimal solution sometimes. The case can turn worse when data becomes imbalanced since high accuracy is no longer our primary purpose in such an analysis. When we talk about cross entropy in deep learning, it concerns with the labels. However, in such an analysis, labels are generated from key hypothesis. Before I give a new metric, I will first redefine cross entropy in a slightly different way. Here comes the definition. Let Lk be labels generated with key hypothesis k. Then the cross entropy between the two distributions and the predicted distribution with respect to the key hypothesis k denoted as Cek defined as follows. It's clear that the definition is compatible with the original definition of cross entropy if k is the right key. Also notice that we expand the definition of cross entropy to all possible key hypotheses. Actually, there is a tight relation between the maximum likelihood score and the cross entropy for key hypotheses. Thanks to the law of large numbers, the expectation of the opposite of the maximum likelihood score now always equals the cross entropy for the corresponding key hypotheses. That's why we redefine cross entropy. With this tight relation, we can define our new metric from the perspective of such an analysis. By considering both cross entropy of the right key and the wrong keys, which have the direct relation with their maximum likelihood scores, we define a normal metric called cross entropy ratio. Let k star be the right key, the cross entropy ratio is defined as follows. Here, the numerator is the cross entropy for the right key and the denominator is the expectation of the cross entropy for the wrong keys. Regarding the relation between the maximum likelihood score and the cross entropy, our metric considers both the scores of the right key and the wrong keys. So our metric is directly linked to the attacking process now. The denominator of the cross entropy ratio can be further simplified with two reasonable assumptions. The first assumption is that when we're generating labels, the output of the cryptographic primitive are computationally indistinguishable from random outputs. The second assumption is when k not equals k star, then k is independent of the modal m system. With these two assumptions, by substituting the output of cryptographic primitive G with a uniformly distributed random variable u, we have equation 7. This equation shows the expectation of the cross entropy for the wrong keys, exactly the same as the cross entropy for any wrong key. This is true since the wrong keys are shadowed by the cryptographic primitive G and the deep learning model knows nothing about it. In this way, cross entropy ratio metric can be simplified as the ratio between the cross entropy of the right key and the cross entropy of any wrong key. Combining equation 5 and equation 8, we have some interesting results. Equation 5 tells us, given a key hypothesis, the expectation of the opposite of the maximum likelihood score is exactly the cross entropy. Equation 8 gives us the definition of cross entropy ratio. Combining these two equations, we have some interesting results. Let's cross entropy ratio equals gamma if we make a difference between the expectations of the score for the right key and the expectations of the score for any wrong key. The result is equal to 1 minus gamma multiply the expectation of the minus score for the wrong key. Here we see how cross entropy ratio affects the taking process. Since this part is always positive, if cross entropy ratio is less than 1, it means the expectations of the score for the right key is greater than the score for the wrong key. In this way, hopefully we think with sufficient side-chain matrices, the attack will be successful. Actually considering a widely accepted evaluation metric for side-chain analysis, Gaussian entropy, and the success rate, the following proposition holds. For any attacking set as a, drawing uniformly from the joint distribution under key k star with n-traces and the attacker with n is to retrieve the right key, because the cross entropy ratio is less than 1 implies when n approaches infinity, Gaussian entropy and success rate will be 1. Platforms proposition would also have some observations showing why cross entropy ratio is a good side-chain metric. According to equation 9, with the smallest cross entropy ratio, the difference between the expectations for the scores gets bigger, which means it's easier to retrieve the right key, and thus the attacker might need less traces to mount a successful attack, or with the same amount of traces, it might get lower Gaussian entropy and a higher success rate. However, these haven't been proved yet, but we will verify it with our late experiments. Equation 9 also implies that on average since the right key and the wrong keys will be more distinguishable with the bigger cross entropy for the wrong key, which in turn verifies the definition of cross entropy ratio is reasonable. Cross entropy ratio considers both the cross entropy for the right and the wrong keys, and here we see how the cross entropy for wrong keys affects our attacking results. Before I show you our experimental results, I first give a brief overview on how to estimate cross entropy ratio like traditional side-chain metrics. First, let me introduce the lemma. Let sequence A and sequence B be two sequences of random variables. Sequence A converges to a constant alpha in probability. Sequence B is no less than 1 and converges to a constant beta in probability. Then the fraction of sequence A and sequence B converges to alpha divided by beta in probability. This lemma can be proved using the definition of convergence in probability. The condition sequence B is no less than 1 can be relaxed. This lemma shows the convergence property of the ratio remains if both the numerator and the denominator have the convergence property. With this lemma, we only need to consider the estimation of the numerator and the denominator. Thanks to the law of length numbers, we know that cross entropy loss converges in probability to cross entropy. Let Sb be treated from the right key with labels. Then we know the cross entropy loss or estimation of the cross entropy converges in probability to the cross entropy of k star. The thing that Spr be a randomized test set where you are uniformly distributed, then we also have the convergence property. Actually, the randomized set can be generated by just sampling the labels of the normal test set Spr. It has the same function. Regenerates the labels is not needed. Combining these two convergence property equation 10 and equation 11 with the lemma, we can give an estimation of cross entropy ratio with the convergence property holds. Now, I demonstrate some of our experiments on Ascot data set. Detailed results can be found in our paper. First, this is the figure about the packing process for multi-layer perceptron with hamming weight labels used in the original Ascot paper. Some parameters are shown on the top right of the figure. From the figure, we can see that C cross entropy ratio is closely related to the attacking performance. A smaller cross entropy ratio indicates the guessing entropy will drop sharply with the attacking traces increase. For example, the blue line which corresponds to the smallest cross entropy ratio always have the smallest guessing entropy. Also, for multi-layer perceptron model using the output of Ascot as labels, the results are similar. The curve with lower cross entropy ratio has better performance than the curve with higher cross entropy ratio. The experiments verify that cross entropy ratio can properly reflect the attacking results which means it's a good search and metric for profiling attack. To further investigate the effectiveness of cross entropy ratio with imbalanced data and show why deep learning metrics such as accuracy are not truthful in some occasions, we also conduct several experiments with different levels of the imbalanced training data. For Symbolic City, we use the least significant bit of the output of the Ascot as our label. The results are shown in the figure. It is evident that with the increasing of the level of imbalance, the accuracy of the model keeps increasing while the attacking performance becomes worse. On the contrary, cross entropy ratio still reflects the attacking results facefully without misleading. In this way, we conclude cross entropy ratio is more suitable than deep than deep learning metric accuracy with the data is imbalanced. Also estimating cross entropy ratio is easy and has similar computational complexity to the calculation of accuracy. In our experiments, time caused from competing cross entropy ratio and accuracy is close. Both consuming about 7 milliseconds and both are faster than the calculating of a guessing truth or success rate which needs to mark practical attacks. Since cross entropy ratio is a good metric derived from Syton's perspective, a natural thought is to adapt it to a new kind of loss function for the training of deep learning models. We define cross entropy ratio loss or simplified ratio loss as follows. Here SP is the training set and SPR is the training set with sharp root label and N is the constant defined by user to determine how many sharp root sets are needed. With the large N's estimation will be more accurate. Actually cross entropy loss is just an estimation of cross entropy ratio on the training set. Due to the good properties of cross entropy ratio metric when facing imbalanced data, the ratio loss also has the power of animating the influence of imbalance. The reason behind is the ratio loss is associated with the keys and take both the right key and the wrong keys into consideration. This loss function does not focus too much on the labels like cross entropy loss function whose purpose is to maximize the prediction accuracy. Instead, the ratio loss maximizes the difference between the right key and the wrong key which is exactly what we want because we use maximum likelihood method. So the imbalance problem will not bother the ratio loss since both the labels from the right key and the wrong keys are used and the influence is counteracted. We have abundant experiments on different deep sets with different deep learning models. All the models use hamming width or hamming distance labels which is imbalanced. First for escut, multi-day perception and synchronized traces will result in encouraging. Cross entropy ratio loss performs better than cross entropy loss in all conditions. Well, escut this set with many cheaters where we use CNN models, guessing entropy for the ratio loss corresponding to the orange line drops sharply sharper than the blue line which used cross entropy loss. For traces with random delays we will also use CNN as our model. The ratio loss function also performs better and for hardware implementations with multi-day perception model the ratio loss should show significant improvements. The secret key can be recovered using about 2,000 traces with ratio loss. However, cross entropy loss are not even powerful enough to mount a successful attack. All the results show ratio loss function is more efficient than cross entropy loss function with imbalanced data for software and hardware implementations for multi-layer perceptual and CNN models for synchronized and desynchronized traces and for both order and high order cases. So, we properly solve the problems of the imbalanced data in such a scenario. In conclusion, we propose a deep learning test such an evaluation metric cross entropy ratio which can be used to evaluate the performance of deep learning models for such an analysis. We show that cross entropy ratio is closely related to commonly used such a metrics guessing entropy and success rate both in theory and our experiments. Besides, it works stably with imbalanced data. We also adapted, adapt cross entropy ratio metric to new kind of loss function for deep neural networks properly solving the problems of the imbalanced data in such a scenario. Thanks for watching if you have any questions feel free to send me an email.