 It's been well established that algorithms can be biased towards individuals or demographic groups. In this work, we counteract these biases with a training algorithm that borrows ideas from adversarial robustness. To explain our paper, we will walk through an example of predicting whether a word has positive or negative sentiment given its word embedding. Training a neural network on this data, we obtain the following plot of logits. Blue regions correspond to words that have positive predictions, whereas red regions correspond to words that have negative predictions. Unfortunately, due to biases in the word embeddings that we use to train the sentiment classifier, white names on average are classified as positive whereas black names on average are classified as negative, even though the training data does not contain any names. This is troubling for applications like sentiment analysis of news articles. Someone's name should not affect the sentiment. Because classifications on average are different for white and black names, this neural network violates group fairness. Furthermore, any two names should have the same classification, but clearly based on the variance of the box plots. There are names with extremely different logits and hence predictions. Therefore, this classifier also violates individual fairness. The seminal work of Dwork et al. defines individual fairness in terms of an individually fair metric that encodes which samples should be treated similarly. In the sentiment example, the distance between any two names under the individually fair metric is close to zero. Most prior work focuses on group fairness. However, group fair classifiers can badly violate individual fairness. In contrast, we focus on individual fairness and propose one of the first practical algorithms to obtain an individually fair classifier. We have three main contributions. First, we propose defining individual fairness in terms of a machine learning model's robustness to sensitive perturbations. Second, we propose sensor, an algorithm to learn a provably individually fair classifier. And third, we propose practical data-driven approaches for learning the individually fair metric. Our first contribution is viewing individual fairness as a form of distributional robustness. A machine learning model is distributionally robustly fair if its loss is robust to perturbations of the data with respect to the individually fair metric. To illustrate this definition, consider the sentiment example again. First, we need a fair metric. Because any pair of names should be treated similarly, we define a fair metric on word embeddings such that the distance between any two names is nearly zero. Now we check if any of the sentiment words can be perturbed according to the fair metric such that their predicted classification changes leading to an increase in loss. This neural network violates distributionally robust fairness because if we move the positive label word from the base of the arrow to the tip of the arrow, its predicted classification changes from positive to negative. These two points are close in the fair metric. Equipped with this notion of individual fairness, we propose sensor, which finds the minimal loss classifier that is distributionally robustly fair. Mathematically, sensor finds the best classifier over the worst case distribution. If you are familiar with the adversarial robustness literature, then this equation likely looks familiar. The key differences of the perturbations of the data are with respect to the individually fair metric, not the L infinity norm. To illustrate sensor, consider the sentiment example again. We will use sensor to obtain an individually fair neural network. Sensor iteratively finds the best classifier and the worst case distribution. To find this distribution, sensor moves the training data to regions of the space that are one close in terms of the individually fair metric and two increases the loss of the classifier, which is what the blue arrow illustrates as we have discussed. The resulting neural network trained with sensor assigns nearly the same logits to each name, thereby achieving individual fairness, which is not true for the vanilla neural network. We compare sensor to state-of-the-art algorithms and other baselines and find that sensor is the only algorithm that obtains individual fairness and group fairness with respect to race or gender.