Validation of the ML models

Both cross-validation during ML training and internal validation (test phase) after development of the ML models were done. Cross-validation is defined as dividing a dataset into k number of subsets. In one epoch, use k-1 subsets of data for training and use the remaining dataset to give an estimate of model skill while tuning model’s hyperparameters. For every epoch, validation dataset will be different, but it will be out of those k subsets of data. This is also referred to as k-fold cross-validation [13]. In this paper, fivefold cross-validation, was applied to the training set (4800 sample) and used to adjust the hyperparameters of each of the applied classifiers, where each fold consists of 960 samples.

Moreover, internal validation was done using the test dataset after the developments of the model. In internal validation, the test dataset is withheld from model training but is utilized to provide an unbiased evaluation of the quality of the final tuned model for comparing and selecting among different designed models.

Internal validation was tested on 1200 unseen samples using: accuracy and F1-score. Accuracy is the number of correctly predicted data points out of all the used test samples. F1-score is a way of combining the precision and recall of the model and is specially used to give a real evaluation for imbalanced data. It is defined as the harmonic mean of the model’s precision and recall, where precision refers to the fraction of correctly classified positive cases among all the estimated positive ones. Recall, also known as sensitivity, represents the fraction of samples classified as positive among the total number of positive examples [7].