Data analysis

For each included DRD, the final recommendation was converted into a binary outcome variable coded as positive if the recommendation was to ‘list’ the drug (i.e., include it in a participating publicly funded drug benefit plan) with or without conditions and negative if the recommendation was to not ‘list’ the drug. Factors were converted to categorical variables characterizing the submissions, including the type of submission (new or resubmission), prevalence of the condition (orphan or ultra-orphan) and type of drug (alimentary tract/metabolism product, antineoplastic/immunomodulating agent or other) were created. Four binary variables (‘yes’ or ‘no/ not measured’) were created to describe the presence or absence of meaningful improvements across efficacy and effectiveness outcomes: 1) differences in clinical outcomes, 2) differences in biomarker/surrogate outcomes, and 3) differences in patient reported outcomes (PROs). Classification of the outcomes was based on the definitions described in the “Final recommendation” documents. The following binary (‘yes’ or ‘no’) variables were also created: safety issues, bias in outcome measures, consistency between the patient population in trials and indication(s) for which a reimbursement/listing recommendation was sought, availability of direct comparative data, availability of long-term data, and presence of other methodological or study design issues. A detailed description of these variables is provided in Table 1.

Description of variables included in the analyses

0 if negative

1 if positive

• Negative: do not list

• Positive: list, list with conditions, list with criteria, list if price reduced or cost-effectiveness improved

0 if new submission

1 if resubmission

0 if no

1 if yes

0 if alimentary tract & metabolism

1 if Antineoplastic & immunomodulating

2 if other

0 if cancer

1 if non-cancer

0 if ultra-orphan

1 if orphan

• Ultra-orphan: < 1 in 100,000 people

• Orphan: < 1 in 2000 people

0 if no or not stated

1 if yes

0 if yes

1 if no

0 if no, inconsistent or not measured

1 if yes

• Biomarker is “a defined characteristic that is measured as an indicator of normal biological process, pathogenic process, or responses to an exposure or intervention, including therapeutic interventions” [17].

• Surrogate outcome is “an endpoint that is used in clinical trials as a substitute for a direct measure of how a patient feels, functions, or survives” [17].

• Meaningful improvements defined as statistically significant differences or non-inferiority in biomarker/ surrogate outcomes (e.g. weight, 6 min walk test, progression-free survival)

0 if no, inconsistent or not measured

1 if yes

• Clinical outcome is “an outcome that describes or reflects how an individual feels, functions or survives” [17].

• Meaningful improvements defined as statistically significant differences or non-inferiority in clinical outcomes (e.g. survival, transplantation)

0 if no, inconsistent or not measured

1 if yes

• PRO is “a measurement based on a report that comes directly from the patient about the status of a patient’s health condition without amendment or interpretation of the patient’s response by a clinician or anyone else” [17].

• Meaningful improvements defined as statistically significant differences or non-inferiority in PRO (e.g. QOL, rating of pain intensity, SF-36)

0 if no

1 if yes

0 if no

1 if yes

• Present when ‘final recommendation’ document stated that data from trials included all subgroup of the indicated population

• Not present when for example submitted indication includes mild, moderate, and severe forms of disease but trial data limited to mild-moderate forms of disease

0 if yes

1 if no

• Present when indicated in the final recommendation document

• Bias in outcome measurements (e.g., subjective outcomes classified by non-blinded investigators)

0 if no

1 if yes

• Presence of long-term data where long-term data is important given the course of disease

• Present when indicated in the final recommendation document

0 if yes

1 if no

• Concerns over other aspects of study design (e.g., small sample size, carry-over effects associated with withdrawal trial methodology)

• Present when indicated in the final recommendation document

ATC Anatomical Therapeutic Chemical (ATC) Classification System, CADTH Canadian Agency for Drugs and Technologies in Health, ICD International Classification of Disease, ICER Incremental Cost-effectiveness Ratio, ICU Intensive Care Unit, PRO Patient- reported Outcome, QALY Quality-adjusted Life Years, QOL Quality of Life, RCT Randomized Controlled Trial

First, a series of two-by-two or three-by-two tables were constructed to examine the percentage of positive recommendations for each variable extracted from the “Final recommendation” document. Data were tabulated for all included recommendations and stratified by type of condition (i.e. cancer and non-cancer) to examine whether the frequency of positive and negative recommendations for each factor (i.e., independent variable) varied with type of condition. Pearson’s chi-square or Fisher’s exact test were used to test the statistical significance of differences in such percentages. This step was also used to check for any errors and spot complete and quasi-complete separation of data (i.e. recommendations were almost perfectly predicted by the independent variables).

Next, factors potentially associated with recommendation type were further explored through multiple logistic regression- a statistical analysis that allows for the assessment of the association between multiple factors and a dichotomous outcome (in this case, positive or negative recommendation) [18]. Two methods for building regression models were used and the results compared: 1) purposeful selection and 2) stepwise selection.

In purposeful selection, covariates whose univariate test had a p-value < 0.21 were first identified [18, 19]. A multivariable model containing these covariates was constructed, and variables with p-values > 0.21 were excluded. Each variable not selected initially for inclusion in the multivariable model was then added one at a time. If its p-value was > 0.05 and none of the coefficients in the model changed by > 20%, the variable was excluded. The resulting model comprised the main effects model. Finally, two-way interactions among the variables were added to the main effects model one at a time and checked for statistical significance. Those with p-values > 0.05 were excluded. To assess the fit of final model, the Hosmer-Lemeshow goodness-of-fit test was used [20, 21].

In stepwise selection, each variable was entered into the model step by step (SAS® Stepwise Logistic Regression). The significance level for entry and stay were set at 0.2. The results were identical with purposeful method.