To evaluate the model, we used the absolute error—first, we computed the mean absolute error of prediction MAEm(s) of model m for sample sS over individual genes gG as in Eq 1 where y(g, s) is the expression of gene g for sample s and y(g,s)^m is the prediction of model m for the same target.

For further evaluation, we treat individual samples as independent (which is close enough to reality—our dataset probably contains small groups of samples that might be somewhat dependent, for example, having the same treatment, but it should be negligible for our size of dataset). Thus for pairwise comparison, we compare error metrics over individual samples and not over individual genes that have ties to each other. The overall performance of model m is defined as:

To estimate the distribution of the MMAE, we employ bootstrap over MAEm(s), i.e., we resample the set of samples with a replacement to get a new set S which is then used for MMAE calculation in each bootstrap iteration. Pairs of models are not compared only in terms of MMAEs but also using pairwise differences. The mean difference of absolute errors MDAEm1,m2(s) for models m1 and m2 and sample s is defined as:

The MMDAEm1,m2 is defined as:

The pairwise nature of MMDAEm1,m2 and its distribution allow for an accurate comparison of two models even though their MMAEs are very close, and their confidence intervals (CIs) estimated using bootstrap on MAEs are overlapping. The distribution is estimated using bootstrap on MDAEm1,m2(s) in a similar manner as distribution of MMAEm is estimated using MAEm(s).

To complement the model comparison based on MMDAEs, we have used the Student’s paired t-test and the paired Wilcoxon rank test on MAEs of individual samples. These tests were used to test the hypothesis that the differences in MAEs for individual samples over all genes are significantly different.

Note: The content above has been extracted from a research article, so it may not display correctly.

Please log in to submit your questions online.
Your question will be posted on the Bio-101 website. We will send your questions to the authors of this protocol and Bio-protocol community members who are experienced with this method. you will be informed using the email address associated with your Bio-protocol account.

We use cookies on this site to enhance your user experience. By using our website, you are agreeing to allow the storage of cookies on your computer.