2.2 Evaluation measures

We considered a named entity as successfully extracted if its text offsets (character positions in a text-string) were correctly identified (tp: true positive). We considered two modes for tp: exact matching (two entities match if their text offsets are identical) and partial matching (text offsets overlap). Any other prediction was considered as a false positive (fp) and any missed entity as a false negative (fn). Partial matching is more suitable to evaluate NL mentions lacking well-defined boundaries. For instance, in finding ‘[changed conserved] glutamine at 115 to proline’, we did not distinguish solutions with and without the words in brackets, because we focused on the extraction of the mention not on that of additional annotations (here ‘conserved’). We computed performance for all cases and for the subclasses (ST, SST and NL). A test entity of subclass X was considered as correctly identified if any predicted entity matched. We then used the standard evaluation measures for named-entity recognition, namely, precision (P: tp/tp + fp), recall (R: tp/tp + fn) and F-Measure (F: 2*(P*R)/(P + R)). Within a corpus, we computed the StdErr by randomly selecting 15% of the test data without replacement in 1000 (n) bootstrap samples. With <x> as the overall performance for the entire test set and x_i for subset i, we computed:

Across corpora, we did not merge documents. Rather, we computed the mean of P, R and F between the considered corpora, and computed the StdErr of the mean without subsampling.