Machine learning protocol: performance metrics

The accuracy of the NR-SBINN was measured by computing the average relative prediction error and associated standard deviation over all data points in the testing set. For the C-SBINN, the accuracy of the network was measured by computing several performance metrics, including: the Cohen kappa score (CKS)^30,63, the Matthew correlation coefficient (MCC)⁶⁴ (shown to be the best measure of classification accuracy for imbalanced data sets⁶⁵), the area under the receiving-operator characteristics (ROC) curve⁶⁶, and the area under the precision-recall curve (PRC)^67–69 (the latter of which has also shown better performance for imbalanced data sets). The averages and standard deviations of the performance metrics were computed over the 10-fold cross validation. The ROC curves and precision-recall curves were computed using the Python scikit-learn functions⁵⁷. The G-mean⁷⁰ and the F1-score^71,72 were calculated and used to adjust the classification probability threshold⁷³ to maximize the MCC for the imbalanced data sets. A summary of the terminology and the procedures for calculating each performance metric is presented in Supplementary Table 4.

Lastly, in order to train and test the C-SBINN on the ex-vivo patient data, it was necessary for each sample to be labeled. In the absence of patient-matched clinical outcome, we developed a statistical approach that used k-means clustering⁷⁴ with the ex-vivo expression of granzyme and perforin, which are released by cytotoxic T-cell lymphocytes and natural killer cells to induce apoptosis in target cells, such as cancer cells. The full details of the ex-vivo labeling approach are presented in Supplementary Section F.