# Also in the Article

Evaluation indices of batch effect removal
This protocol is extracted from research article:
iMAP: integration of multiple single-cell datasets by adversarial paired transfer networks
Genome Biol, Feb 18, 2021;

Procedure

There exists an extensive list of batch effect removal evaluation indices in the literature [6]. Some widely used include kBET (k-nearest neighbor batch-effect test) [18], LISI (Local Inverse Simpson’s Index) [13], ASW (average silhouette width), and ARI (adjusted rand index). We argue that ARI and ASW are cluster-level indices and cannot reliably evaluate the mixture of cells from different batch at a local single-cell level (Additional file 1: Fig. S1a). kBET and LISI evaluate the batch mixing at a local level by comparing the batch distribution with kNNs of a cell with the global batch distribution. kBET has the advantage in evaluating the integration performance of batch-shared cell types, one drawback of which, however, is that when it measures the batch mixture, it is cell type ignorant. This may cause unfair results when the proportions of share cells types are too discrepant in different batches [13]. LISI could evaluate both the capacity of identification batch-specific cell types and the integration of batch-shared cell types, but it is hard to summarize all single cell-level LISI values into a simple statistic for comparing between various methods. kBET and LISI are nonetheless reliable metrics when appropriated employed. So, we first used these two kinds of metrics to compare different methods. For kBET, we computed the acceptance rates for each cell type separately and summarized the median value over all tested cells as the final output. For the “DC_rm” and “panc_rm” datasets, only those cell types appearing in all batches were taken into account, and since no cell type appears in all three sub-datasets of “cell_lines,” we computed the acceptance rates for the integration of “Jurkat” and “Mix” and the integration of “293 T” and “Mix,” respectively. One important parameter k, the number of nearest neighbors, has a large effect on the results of kBET, and following the kBET paper, a series of k values, which are chosen as 5%, 10%, 15%, 20%, and 25% of the total cell numbers, are adopted to run kBET. For LISI, we computed the cLISI and iLISI values for each cell, with the ideal cLISI equal to one. iLISI values of different methods are compared for each cell type separately, because the best values are cell type-specific, and determined by the number of batches having this specific cell type [13].

Considering that these indices all have their own limitations in terms of simultaneously evaluating both cell type and batch mixing, we propose two new indices to evaluate the batch mixture. Our evaluation procedure is also based on kNNs of a cell and divided into two successive steps (Additional file 1: Fig. S1b). Firstly, we classify all cells into “positive” and “negative” cells. “Positive” cells are those surrounded mostly by cells from the same cell type. Be default, one cell is assigned as “positive” only if at least 50% cells of its kNNs are with the same cell type label, otherwise “negative” (k is set as the minimum of 100 and the number of cells for this cell type). Then, those positive cells are further discriminated into “true” and “false” positive cells by a second dichotomous classifier. “True” positive cells are those surrounded by appropriate proportions of cells with different batches. We use the three-sigma rule of thumb to measure whether the observed batch distribution of one positive cell’s neighborhood is consistent with the global batch distribution. Considering a cell with cluster label y, the number of cells from cell type y in all n batches are N1, N2, ⋯, Nn respectively. We define pi = Ni/∑jNj for i = 1, 2, ⋯, n. Then, by expectation, if we sample k cells from cell type y, the number of cells from batch i is equal to kpi. We regard a positive cell as true positive if the numbers of its neighbors from different batches are all within the range of 3 standard deviation around the expectation. This is to say, suppose kNNs of one true positive cell have the batch distribution k1, k2, ⋯, kN, then $ki∈max0kpi−3kpi1−pikpi+3kpi1−pi$ for all i = 1, 2, ⋯, n. By these two classification procedures, we could automatically identify those cells that are not mixed well. We use the proportions of positive and true positive cells as the quantitative indices to evaluate the performance of batch effect removal of different methods. This two-classifier system also provides an effective tool for visualizations of the batch effect removal results.

Note: The content above has been extracted from a research article, so it may not display correctly.

Q&A