Explanation algorithms

The algorithms used for modeling include rule learning, which generates a directly interpretable representation, as well as random forests and neural networks, whose interpretation requires the application of additional explanation algorithms. In the following, we provide a brief overview of these explanation approaches.

Rule models Both CORELS and CBA algorithms involved in our study generate rule lists. A rule list is an ordered collection of rules, where each rule is associated with a distinct priority value. A rule has a form of the antecedent→consequent. The rule consists of a set of conditions (antecedent). The consequent contains the predicted value of the target class. To classify (predict a class) for a particular instance, the evaluation algorithm processes the rules in the rule list in the order of priority, highest to lowest. Once it finds a rule with all conditions in the antecedent matching the current instance, the consequent of this rule is used as a prediction for the instance. Rules with lower priority are not processed.

The principal difference between CORELS and CBA is the type of models they produce. The CORELS algorithm tends to provide very condensed models, often containing only one if-else rule. In this respect, the application of the CORELS algorithm on real data for parole and bail decisions received significant attention. In a paper titled Stop Explaining Black Box Machine Learning Models for High Stakes Decisions and Use Interpretable Models Instead (Rudin, 2019), some of the authors of CORELS have argued that a rule model composed of several rules generated by CORELS performs comparably in terms of accuracy to the black-box COMPAS model widely used for actual bail decisions in the U.S. Angelino et al. (2017) have also shown that CORELS is competitive against multiple other machine learning models, including C4.5 and CART decision trees.

CORELS outputs very short models, which is not always desirable. As also demonstrated in our experiments, CORELS models can have lower predictive performance. To complement CORELS, we used CBA, which is one of the most commonly used rule learning algorithms based on association rule learning. Unlike CORELS, CBA results in models that contain a higher number of rules. The advantage of this approach is that it provides better insight into the data since the individual rules correspond to local patterns and could thus be used as a tool for descriptive data mining or explanation. The disadvantage is that CBA-generated models may contain too many rules for the user to be able to manually review. To present these rules in a concise way, we applied grouped matrix clustering, which is a rule clustering technique recently proposed by Hahsler and Karpienko (2017). To communicate the shared elements between the rules, we used the graph-based rule visualization, also adapted from Hahsler and Karpienko (2017).

Random forest and neural networks Both these algorithms belong to the group of “black-box” approaches, which are characterized by the opacity of the internal working of the generated models.

Models generated from random forests cannot be directly interpreted due to the number of trees, their complexity and also the fact that multiple trees can take part in the decision. However, the random forest learning algorithm was designed so that estimates of feature importance scores are readily provided (Breiman, 2001). In our work, we adopt the original method for computing the feature importance scores of random forests, which is based on Mean Decrease of Impurity (MDI). For this method, it has been shown that the MDI importance of a relevant feature is invariant with respect to the removal or addition of irrelevant features and that the importance of a feature is zero if and only if the feature is irrelevant (Louppe et al., 2013).

For neural networks, a number of feature importance methods have been proposed, but as has been recently shown, many of these methods do not provide stable results (Ghorbani et al., 2019). In addition, there is a number of model agnostic methods by which feature importance for models like random forests can be computed. In our work, we adopt Shapley values (Lundberg et al., 2020) and LIME (Ribeiro et al., 2016). Unlike the MDI method for Random Forests, which generates global feature importance scores, these algorithms provide local feature importance values for a particular test instance.

SHapley Additive exPlanations (SHAP) value emerges from the Shapley concept from game theory (Rodríguez-Pérez & Bajorath, 2020). The SHAP values allow global interpretation. Each observation gets its own set of SHAP values so it is possible to also interpret it locally.

LIME (Local Interpretable Model-agnostic Explanations) shows which feature values contributed to a particular prediction and how. This explanation is only approximate since the LIME model is learnt by modification of the explained instance by perturbing the feature values and collecting the resulting impact of each individual feature change on the prediction. The explanation is obtained by locally approximating the explained model with an interpretable one.