2.4. Point Mapping on the Dorsal Tongue Surface

The original photographs showing the dorsal surface of the entire tongue were mapped using points to indicate normal and abnormal regions by the following method: Photographs of the entire dorsal tongue surface of 80 patients who had not participated in previous VGG16 training were prepared. Each of them was labeled ‘abnormal,’ ‘normal,’ and ‘other’ regions using blue and green colors, as shown in Figure 3. The oral medicine specialist evaluated the images at various resolutions by freely enlarging or reducing the original photographic images in a graphics editing program (PaintShop Pro, version 9.01, Jasc Software, Eden Prairie, MN, USA). Detailed papillary structures were evaluated within the magnification range as well as the overall papillary pattern of the dorsal tongue surface. Regions on the dorsal tongue surface with normal mucosa were labeled in green, and those showing abnormal or pathologic changes were labeled in blue (Figure 3).

Examples of labeling abnormal and normal regions of the dorsal tongue surface with green and blue colors.

In order to apply the previously learned DL result for the classification of crop images, the original photographic images were divided into small-size images similar in size to the previously trained images. The boundaries of these images were partially overlapped to increase the area’s density to be analyzed. Three hundred eighty-four images (16 × 24) were cropped from a single photographic image of the dorsal tongue surface (Figure 4).

Small-size images were cropped from the original photographic images of the entire dorsal tongue surface. These small images were predicted to be abnormal, normal, or other.

Each crop image was resized to 96 × 96 pixels, and then the class was predicted by applying the previous learning results. A color dot corresponding to the predicted class (blue for ‘abnormal,’ green for ‘normal,’ and red for ‘other’) was placed at the position in which the cropped image was in the original photographic image. The point mapping image was achieved by distributing color dots according to the classes on the dorsal tongue surface of the photographic image, as shown in Figure 4. With this result, the dorsal tongue surface was segmented into abnormal and normal regions. The ground truth image was also mapped by color dots using the labeled image.

This point mapping segmentation was evaluated similarly to the metric applied to DL-applied segmentation or object detection. The number of pixels corresponding to the ground truth, the prediction, and the intersection of the ground truth and prediction was defined as G, P, and I, respectively (Figure 5). The number of union pixels indicated by the ground truth and the prediction was S = G + P − I. The number of true positives, false negatives, and intersections over union (IoU) was calculated as P − I, G − I, and I/S, respectively. We calculated the precision and recall as TP/(TP + FP) and TP/(TP + FN) for each predicted point mapping image.

TP (true positive), FP (false positive), FN (false negative), and IoU (intersection over union) are calculated from point mapping segmentation.

The average precision (AP) value was calculated by introducing PASCAL VOC evaluation metrics [27]. Abnormal, normal, and other regions were predicted on the point-mapping images of 80 test photos. The confidence value for each class was obtained by calculating the average values of the softmax predicted for the class in the prediction process using the VGG16 model for each test image. Each predicted class of point mapping region was sorted from high to low according to the confidence value. If the IoU value is 0.5 or more for each class, the prediction is correct; if not, it is false. Precision and recall values were calculated and plotted from these results to obtain AP values.