# Also in the Article

2.1 Patch Selection and Categorization
This protocol is extracted from research article:
Learning Medical Materials From Radiography Images
Front Artif Intell, Jun 18, 2021;

Procedure

The first component of the system is selecting and categorizing patches from the medical images so that every patch corresponds highly to its assigned category. Since images vary widely within medicine, such as the differences between X-rays and MRIs, it is important to normalize the images in such a way that the content and annotations are preserved while removing variations that may mislead the system.

Each specific image mode or dataset may use a different approach to patch generation depending on the nature of the source data. The following steps are used to generate patches of background, brain, bone, and tumor categories, but this system can be used to generate image patches in many different medical applications.

To generate the medical-category image patches used to evaluate the system, the first step is to invert negatives (images where the brightest regions indicate dark areas). Then, each image’s raw features are normalized to the range $[0,1]$, and Algorithm 1 is used to generate patches.

Patch categorization procedure

Some images may have expertly annotated masks—like a brain tumor in an MRI. Other images—like the knee X-rays in our experiment—may lack masks and labeling, but the categories sought to be analyzed are simple enough to be assumed. This reduces the detail of the dataset, but still yields useful categories for training which may even be applicable in other image modes. We call material categories that are expertly annotated (such as “tumor”) expert categories, while non-annotated material categories (like “bone” for the knee X-rays) are called naïve categories since the naïve assumption is made that the average brightness of an image region corresponds to its category.

A third type of material category, the null category, corresponds to a category that does not contain useful information, but when isolated can improve the model’s ability to learn the other categories. For the cases of X-rays and MRIs, the null category is derived from the image background.

We believe that brightness constraints are a useful way to extract naïve categories in most cases. Generally, extremely bright regions and dark regions lack interesting texture data—for example, the image background. Meanwhile, moderately bright regions may contain some textural information of interest.

For instance, in identifying brain tumors, gray matter tissue, which may not be annotated with a mask, is not as significant as tumor tissue. However, separating gray matter textures from the background, which is much darker, allows for a classifier to make more specific predictions by preventing it from learning that background regions correspond with gray matter. Additionally, when using multiple image modalities with distinct categories to build a dataset, separating the dark background prevents an overlap in each category’s texture space.

Although we use brightness constraints, other constraints could be used depending on the imaging modality. For example, with a set of RGB color images, a set of constraints could be created from the average value of an RGB color channel.

To generate a material patch from a selected region of an image, the first step is to calculate the average brightness of the region using Eq. 9, which is the sum of all the region’s normalized raw feature values divided by the number of raw features. The constraints $B¯min$, $B¯max$, $B¯0$, and T in Algorithm 1 can be altered at run time to create better-fitting categories.

For expert categories, like “tumor”, that are defined by a mask within the image, the patch generation process needs to ensure that a large enough percentage of the region is within the mask. This value is defined as the mask tolerance T, presented in Eq. 10. This value is included to avoid categorizing regions that are on the mask boundary, which may confuse the training of the system. We define a small value of $T>0$ since it allows for patches that intersect categories while still avoiding ambiguity. This increases the pool of eligible image patches, introduces variance to reduce overfitting, and allows for smaller masks (like for pituitary tumors, which are generally small) to be represented in the patch set.

For any expert category patch, at least $(1−T) × 100$ percent of the patch’s source region is inside the mask boundary. For any naïve category patch, at most $T × 100$ percent of the mask is allowed to be within the patch’s source region.

To further normalize the patches, we also introduce the average brightness constraints $B¯min$, $B¯max$, and $B¯0$. Since each patch raw feature is normalized to the range $[0,1]$, the average brightness constraints are likewise constrained to $[0,1]$. First, if a region has an average brightness $B¯, the region’s patch is automatically added to the null category. For another patch to be included in the dataset, its average brightness must fall within the range $[B¯min,B¯max]$.

Using the above constraints, for each iteration of Algorithm 1, a random image in the set is selected, and within that image, a random point $(x,y)$ from a set of points spaced p pixels apart is selected. For the selected point, patch $Pi$ is spliced from a $32×32$ section of the image below and to the right of $(x,y)$. This patch is evaluated against the constraints to determine if it is eligible to be included in the patch set and what category it belongs to. If the image has a mask, the patch is categorized into the mask or non-mask category based on the mask tolerance value. Patch $Pi$ is added to its assigned category set $Ci$ if it meets the constraints.

The generation process ensures every saved patch originates from a unique point, meaning there are no duplicate patches in the dataset. Additionally, different image types containing different categories may use different constraint values when generating patches. The final patch set is used to form training, validation, and test datasets for both of the CNNs in the following sections.

Note: The content above has been extracted from a research article, so it may not display correctly.

Q&A