3.3. Evaluation

A Bayesian phylogeographic reconstruction, as the one described above, provides a posterior distribution over possible root locations. We are interested in two properties of this reconstruction: (i) How far do we expect the reconstructed location to be from the true homeland? (ii) How certain is the model about its reconstruction? We measure these two properties using the following metrics:

We measure the Euclidean distance between every location in the posterior distribution and the true root to see how far the reconstruction is off. Doing this per posterior sample and taking the root of the mean squared error gives the RMSE metric

Here, X is the true location of the homeland and X^m are the reconstructed locations, taken from all posterior samples over multiple simulation runs. The RMSE is composed of systematic errors (bias) and errors due to variance (in the simulation and in the reconstruction). Since we are mainly interested in systematic errors arising from directed migrations we also present the bias of our reconstructions

As an illustration of the behaviour of these two metrics consider the examples shown in figure 2. With growing directional trend μ, the reconstructed root (red star) will be located further away from the simulated one (blue star), causing both the RMSE and the bias to grow. The difference between the two scores would mostly show in scenarios with low directional trend (μ close to 0), where the bias would approach 0, while the RMSE still shows errors due to variance.

We measure the Bayesian highest posterior density (HPD) region coverage of the root location for different HPD thresholds (80% and 95%). This entirely ignores the amplitude of the error, but shows whether the uncertainty expressed by the posterior reflects the observed error. If so, the HPD coverage will match the corresponding HPD threshold, i.e. the 95% HPD region should cover the root in 95% of the simulations.

In order to compare these evaluation metrics between the different scenarios, we want to measure directional bias in a unified way. In the case of migrations, the directional bias is introduced through an inherent trend parameter μ, in the expansion simulations it is controlled by the sector angle α (and in a real case study, we usually do not know the factors driving the directed migrations at all). To compare these scenarios, we measure the observed trend μ^, which we defined as the distance between the homeland of an expansion and the mean of the final locations (at presence)

Here, X represents the root of the expansion and x_n are the tip locations (contemporary languages). We use this statistic to make the varying levels of directional trend in the two simulation scenarios comparable: in the MigSim scenario, we simulated a total bias μ ranging from 0 to 6000 km (in steps of 500 km) and in the ExpSim scenario the sector angle α ranged from 2π down to 0.2π (in steps of 0.2π). In both scenarios, these settings resulted in an observed trend μ^ between 0 and 6000 km (figures ​(figures44 and ​and55).

The performance of phylogeographic reconstructions of the root based on the MigSim simulations with varying levels of observed trend. (a) The RMSE (blue) and the bias (orange) of the reconstruction. The dots represent single simulation runs, the lines interpolate between the average results for a specific setting for μ (trend). (b) The empirical coverage of the 80% (blue) and 95% (orange) highest posterior density (HPD) regions.

The performance of phylogeographic reconstructions of the root based on the ExpSim simulations with varying levels of observed trend. (a) The RMSE (blue) and the bias (orange) of the reconstruction. The dots represent single simulation runs, the lines are averages across all runs with a specific setting for α (sector angle). (b) The empirical coverage of the 80% (blue) and 95% (orange) highest posterior density (HPD) regions.