We used a combination of Watanabe Akaike Information Criterion (WAIC) [32, 33], Area Under the receiver operator Curve (AUC) [34], and leave-one-out cross validation (looCV) [31] to compare the importance of different effects on γ and p and assess goodness-of-fit (presented in Table 1, schematic of work flow shown in Fig. SM3.1). WAIC is a model selection criterion based on the posterior predictive distribution, and was used to compare fits of models. WAIC was not used to compare models with human population modelled on prevalence (p) because the data were different and thus the WAIC values would not be comparable. AUC is a measure of how well variation is explained by the model—in our case, the ability to distinguish a presence from an absence—and was used to assess how well a particular model explained the data (i.e. a measure of goodness of fit as in [35]). LooCV is a measure of the model’s ability to predict out of sample, and thus was used to compare predictive ability among models. When comparing predictive ability between models we used both AUC and looCV.
*Model used to estimate effects in Fig 2;
**Best predictive model, used to make predictions in Figs Figs44 and and5;5; · = intercept only, - = no effect, Pop = log human population size effect on prevalence parameter p, Distance = effect of distance to nearest infection, Neighborhood = local neighborhood infection density effect, N∙T = north-south by time directional effect, Season = factor with two levels (spring/summer: Feb.-Aug. and Fall/Winter: Sept.-Jan.), E∙T = east-west by time directional effect, -WAIC = Watanabe Akaike Information Criteria (section SM4.1 in S1 Text), AUC1 = Area Under the Curve for observed positive counts (section SM4.2 in S1 Text), AUC2 = Area Under the Curve for latent occupancy process (section SM4.3 in S1 Text), looCV = leave-one-out cross validation score (section SM4.4 in S1 Text), NA = not available; we did not display these WAIC values because they cannot be compared to the WAIC values from the models without Pop.
We calculated WAIC and AUC (Section SM4 in S1 Text) for fits to the full data as well as the looCV predictions. WAIC is preferable to DIC (Deviance Information Criterion—another Bayesian method for model selection) for model selection using hierarchical models [32] because it considers the posterior predictive distribution explicitly and penalizes for complexity of model structure, not just the number of parameters. Similar to DIC, lower values of WAIC indicate a better model of the data. However, in contrast to DIC [36], there is no standard quantitative difference between WAIC values from alternative models that indicates a significant difference between them (i.e., the only criterion is that lower is better). We presented two AUC scores: 1) AUC1 measured predictive ability of p using predicted y’s from posterior values of z and observed y’s, 2) AUC2 measured predictive ability of Ψ using the posterior values of z and the observed y’s transformed to binary data. For the out-of-sample predictions, we conducted looCV for each point in the data (using all other points as the training data) and presented means of AUC1 and AUC2 as measures of predictive ability.
Do you have any questions about this protocol?
Post your question to gather feedback from the community. We will also invite the authors of this article to respond.