How to Train Custom Cell Segmentation Models Using Cell-APP

Background

High-throughput microscopy enables researchers to rapidly collect data on thousands of cells. Analyzing these data, however, can be prohibitively time-consuming. For this reason, researchers have developed many algorithms that partially automate microscopy analysis by segmenting individual cells within images—a process known as instance segmentation. Of these algorithms, deep learning models such as Cell-Pose, StarDist, LIVECell, and EVICAN comprise the current state-of-the-art [1–6]. These ready-to-use models, however, may underperform when applied to images that differ significantly from those on which they were trained. This is a common situation: numerous different imaging modalities and cell lines are used in research labs, yet only a fraction of these have been used to train existing models. This limited scope stems from the way training datasets are typically created: hand annotation, a prohibitively time-consuming process that requires researchers to manually outline each cell in the dataset. We therefore need more efficient dataset generation methods in order to build segmentation models that perform robustly across the range of used cell lines and imaging conditions.

To this aim, we developed Cell-APP (cellular annotation and perception pipeline), a tool that automatically generates high-quality datasets for the training of deep learning–based models that segment adherent cells in transmitted-light (TL) microscopy images. With Cell-APP-generated datasets, users can build models tailored to their cell lines and imaging conditions. The tool requires only two inputs—paired TL and fluorescence images—and these inputs have a minimal set of requirements. First, signals in the fluorescence images must correspond to individual cells and be spatially separate from one another; this is because the fluorescence images are used to locate cells. Second, cells in the TL image must be non-overlapping and have discernible boundaries; if they do not, Cell-APP may produce erroneous cell outlines (see Figure 1 for example inputs and outputs). Finally, Cell-APP includes an optional classification module that uses fluorescence-derived features to label cells. These fluorescence-based classes must correspond to visual differences in the TL image; if they do not, the trained cell segmentation models may not learn to classify cells accurately (see Figure S1 for the full process diagram).

Here, we provide a step-by-step guide for using Cell-APP to build custom cell segmentation models. The guide covers (A) acquiring, or repurposing, microscopy data for Cell-APP, (B–E) generating training and testing datasets, (F) training Detectron2 models on the generated datasets, and (G) results interpretation, evaluating the generated datasets and trained models.

Figure 1. Input-output and Cell-APP graphical user interface examples. (A) Inputs (top row) and outputs (bottom row) that satisfy the described input requirements (see General note 1) and output standards (see Results Interpretation). (B) Example of the final segmentation results displayed within the Cell-APP graphical user interface. Scale bar = 10 μm.