Fairness metrics

The fairness of the dataset and classifications were evaluated. We analyzed the fairness level of both self-rated and clinically-rated labels and focused the algorithmic fairness analysis on classifying clinically rated labels.

The number of subjects from different demographic groups and the selection rates (SR) in different groups were calculated. The selection rate was defined as the percentage of samples being “positive”, meaning clinically-rated MHC or self-rated depression, or self-rated anxiety.

Following the DP defined in [29], we used two definitions of the demographic parity ratio to measure the fairness level of the classifications. For sensitive demographic variable k with G different groups and a g* social-economically privileged group: The first demographic parity ratio (DPR) captured overall parity between any pairs of groups and was defined as:

and g ∈ G_k; The second demographic parity ratio focused on the parity compared to the privileged group and was defined as:

where g ≠ g* and S is the Selection Rate of the utilized classifier, i.e., the ratio of positive classification.

As privileged groups, we defined “male” for gender parity analysis, “white” for race parity analysis, “Older (≥40)” for age parity analysis, and “College or below (≤16 years of education)” for education parity analysis. Using classification results of the test folds in 100 repeated fold-fold cross-validation (detailed descriptions in Section Multimodal assessment of mental health conditions and [19]), DPRs of classifiers trained with features from different modalities were calculated. DPR being further from one means a larger disparity between the privileged and unprivileged groups.

Similar to DPR metrics defined in the above section, we followed EO definition proposed in [30] and defined the overall and over-privileged equalized odds ratios (EOR) based on false positive rate (FPR) and true positive rate (TPR). The first EOR was defined as:

The second equalized odds ratio was defined as:

where g ≠ g* and δ was set to 0.001 to avoid ratios being divided by zero. Similarly, EORs were calculated for each classifier. EOR of one means that all groups have the same TPR and FPR.