In this example, we will revisit the diabetes data (Efron et. al., 2004) used in the second example in section 4 of the course. As a recap, the diabetes dataset consists of in total 442 observations with the following features: age, sex, BMI, average blood pressure and six different blood serum measurements. The target variable is numerical and describes the disease progression one year after the original measurements.

We will redefine the problem a bit from the previous exercise and recast the regression problem as a classification problem. We split the observations into two classes based on the target variable, where the severe progression class has a disease progression larger than 140 and the mild progression class has a disease progression that is at most 140. In this example, the class split is done such that the final classes were approximately equally sized. (In reality, an expert in the field should preferably be consulted to define the split.)

You might recall that we excluded the categorical feature sex from the model the last time you used the diabetes dataset. You will exclude this variable from the model in this example as well. However, the extra variable will be used to analyze the model from a fairness perspective.

Let’s start by training the model. You will use a logistic regression model with only first-order terms. We numerically encode the target variable according to 1=severe progression (positive class) and 0=mild progression (negative class).

Run the code to train the model and evaluate the accuracy on test data.

It is time for the equality analysis. We use the sex variable to compute two confusion matrices over the test dataset, one for each sex (0 or 1). The test set consists of 42 diabetes patients of sex 0 and 47 of sex 1. In the first group, 40.5 % belong to the positive class (i.e. the severe progression class) and the corresponding number in the second group is 48.9 %.

Run the code below to print the confusion matrices.

In the following questions, you will be asked to compute performance metrics for the two sexes, using the confusion matrices obtained.

N.B. the plot_confusion_matrix function from scikit-learn shows a confusion matrix that is transposed (true label on rows, predicted on columns) compared to the one in the course book (true label on columns, predicted on rows).

Calculate the model accuracy for the two sexes, respectively. Use the code block below for your calculations.

Calculate the false positive rate for each sex.

Calculate the true positive rate for each sex.

Calculate the precision for the two groups.

Observing all of the calculated metrics, do you see any performance differences for the two groups? Which of the following statements are true?

The model seems to be worse at detecting severe disease progression among diabetes patients of sex 0 compared to diabetes patients of sex 1.

The model seem to perform equally well on both groups in all regards (accuracy, false positive rate, true positive rate and precision).

A mild progression person of sex 1 is approximately 2.3 times more likely to be predicted as severe progression than a mild progression person of sex 0.

The proportion of diabetes patients of sex 0 predicted to have a severe disease progression, that actually have a severe disease progression, is much larger than the corresponding proportion of diabetes patients of sex 1.

For this example, is it possible to make the model fair in terms of false positive rate, true negative rate and precision at the same time?

Fairness is an important topic in society. As a result, and since we want to be able to employ machine learning models in society, fairness is an important topic in machine learning as well. While it is often not possible to achieve fairness in every regard, we can strive to prioritize and achieve fairness in the regard most essential to the application. We could for example imagine that the model in this example could be used for giving extra medical attention to diabetes patients if predicted to have a severe disease progression. Hence, if there is a systematic difference between detecting severe disease progression between the two sexes, the risk is that one of the groups more regularly experiences negative implications due to inadequate treatment. Therefore, we might put particular value on that the proportion of detected severe progression diabetes patients is equal for both sexes (corresponding to equal true positive rates). We might not, for instance, care as much about wrongly predicting mild progression patients as severe progression, since the negative consequences are not as serious. Of course, some might think the opposite is true or that fairness in terms of precision is enough. Questions like these do often not have true, objective answers and they need to be considered case by case.