Model evaluation system

The evaluation system for 2019 challenge (model-compare.emdataresource.org) was built on the basis of the 2016/2017 Model Challenge system¹¹, updated with several additional evaluation measures and analysis tools. Submitted models were evaluated for >70 individual metrics in four tracks: Fit-to-Map, Coordinates-only, Comparison-to-Reference and Comparison-among-Models. A detailed description of the updated infrastructure and each calculated metric is provided as a help document on the model evaluation system website. Result data are archived at Zenodo⁵⁴. Analysis software versions/websites are listed in the Nature Research Reporting Summary.

For brevity, a representative subset of metrics from the evaluation website are discussed in this paper. The selected metrics are listed in Table ​Table22 and are further described below. All scores were calculated according to package instructions using default parameters.

The evaluated metrics included several ways to measure the correlation between map and model density as implemented in TEMPy^16–18 v.1.1 (CCC, CCC_OV, SMOC, LAP, MI, MI_OV) and the Phenix²¹ v.1.15.2 map_model_cc module¹⁹ (CCbox, CCpeaks, CCmask). These methods compare the experimental map with a model map produced on the same voxel grid, integrated either over the full map or over selected masked regions. The model-derived map is generated to a specified resolution limit by inverting Fourier terms calculated from coordinates, B factors and atomic scattering factors. Some measures compare density-derived functions instead of density (MI, LAP¹⁶).

The Q-score (MAPQ v.1.2 (ref. ⁸) plugin for UCSF Chimera³⁸ v.1.11) uses a real-space correlation approach to assess the resolvability of each model atom in the map. Experimental map density is compared to a Gaussian placed at each atom position, omitting regions that overlap with other atoms. The score is calibrated by the reference Gaussian, which is formulated so that a highest score of 1 would be given to a well-resolved atom in a map at an approximately 1.5 Å resolution. Lower scores (down to −1) are given to atoms as their resolvability and the resolution of the map decreases. The overall Q-score is the average value for all model atoms.

Measures based on Map-Model FSC curve, Atom Inclusion and protein sidechain rotamers were also compared. Phenix Map-Model FSC is calculated using a soft mask and is evaluated at FSC = 0.5 (ref. ¹⁹). REFMAC FSCavg¹³ (module of CCPEM⁴²) integrates the area under the Map-Model FSC curve to a specified resolution limit¹³. EMDB Atom Inclusion determines the percentage of atoms inside the map at a specified density threshold¹⁴. TEMPy ENV is also threshold-based and penalizes unmodeled regions¹⁶. EMRinger (module of Phenix) evaluates backbone positioning by measuring the peak positions of unbranched protein C_γ atom positions versus map density in ring paths around C_ɑ–C_β bonds¹⁵.

Standard measures assessed local configuration (bonds, bond angles, chirality, planarity, dihedral angles; Phenix model statistics module), protein backbone (MolProbity Ramachandran outliers²⁰; Phenix molprobity module) and sidechain conformations, and clashes (MolProbity rotamers outliers and Clashscore²⁰; Phenix molprobity module).

New in this challenge round is CaBLAM²² (part of MolProbity and as Phenix cablam module), which uses two procedures to evaluate protein backbone conformation. In both cases, virtual dihedral pairs are evaluated for each protein residue i using C_ɑ positions i − 2 to i + 2. To define CaBLAM outliers, the third virtual dihedral is between the CO groups flanking residue i. To define Calpha-geometry outliers, the third parameter is the C_ɑ virtual angle at i. The residue is then scored according to virtual triplet frequency in a large set of high-quality models from PDB²².

Assessing the similarity of the model to a reference structure and similarity among submitted models, we used metrics based on atom superposition (LGA GDT-TS, GDC and GDC-SC scores²³ v.04.2019), interatomic distances (LDDT score²⁴ v.1.2), and contact area differences (CAD²⁶ v.1646). HBPLUS⁵⁰ was used to calculate nonlocal hydrogen bond precision, defined as the fraction of correctly placed hydrogen bonds with more than six separations in sequence (HBPR > 6). DAVIS-QA determines for each model the average of pairwise GDT-TS scores among all other models²⁷.

Residue-level visualization tools for comparing the submitted models were also provided for the following metrics: Fit-to-Map, Phenix CCbox, TEMPy SMOC, Q-score, EMRinger and EMDB Atom Inclusion; Comparison-to-Reference, LGA and LDDT; and Comparison-among-Models, DAVIS-QA.