2.5. Validity checks

A major disadvantage of the RDD is the necessity of larger sample sizes than experimental research (Schochet, 2009). To evaluate whether the RDD in the current study suffers from low statistical power, we calculated the statistical power for the treatment effect by using RDD power calculator (see Appendix E for results). Because the RDD power calculator did not allow us to estimate the statistical power for the interaction term, we focused on the power for the treatment effect. Our results showed that the statistical power for the treatment effect is around 0.61 to 0.88. One exception is the power in the full model, which is much lower than the power in the other models. Overall, although the power for the treatment effect did not reach the required statistical power for experimental studies (i.e., 1 – β ≥ 0.8) in most cases, using an established causal inference technique to identify the causal effect of trauma on mental health is a unique contribution to the gene-environment studies. Furthermore, a recent study proposes the importance of underpowered observational causal studies to obtain the precise pooled effect estimate in a meta-analysis (Hernán, 2021). Therefore, the power issue in our analysis is a limitation which increases the likelihood that the observed significant results may be false positive, but it does not eliminate the value of the analysis.

Furthermore, we also conducted two additional validity checks to test the RDD assumptions: (1) no treatment other than 9/11 and (2) exchangeability around cut-points which implies respondents could not sort around the threshold. First, we conducted placebo regressions for the CES-D score to test the significance of the observed discontinuity at other interview dates (August 22 to October 1) with bandwidths of 20 days prior to and after the cut-point. Fig. 2 presents the discontinuities (the solid line) and their 95% confidence intervals (the dashed lines). Fig. 2 shows the positive discontinuities around 9/11 are highest. Additionally, the positive discontinuities around 9/11 are the only discontinuities that remain at the 5% significant level for several days. Therefore, the results of the placebo tests did not show evidence to question the validity of the RDD.

Estimated discontinuities with the cut-off points other than September 11.

Note: the solid line shows the point estimators of the discontinuity, and the dashed line shows 95% confidence interval. We estimated discontinuities with the cut-off points from August 22 (equivalent to −20 on x-axis) to October 1 (equivalent to 20 on x-axis) without covariates.

The RDD also assumes that the treatment and control groups have similar characteristics. To test this exchangeability assumption, we estimated the RDD with a dependent variable of CES-D score and 19 items of the CES-D score in Wave I, neuroticism, extraversion, conscientiousness, age, sex, state of residence where at least 40 respondents in our analytical sample lived within the bandwidths of 50 days, family income, mother’s educational attainment, PVT score, the first 10 principal components, and mean and variance depression polygenic scores with bandwidths of 50 days. Out of 64 tests, the discontinuities of the proportion of living in state 27 (β = −0.04, p < 0.05), the proportion of mothers with some college degree (β = −0.09, p < 0.05), family income (β = 14.37, p < 0.001), and principal component 9 (β = −0.00, p < 0.05) reached a conventional 5% significance level. Most results showed no significant discontinuities and the effect sizes relative to their standard deviation are not as large as that of the CES-D in Wave III. However, the number of significant discontinuities was slightly larger than would be expected by chance. This may be due to postponements of an interview among those living close to the stricken areas. Nonetheless, the Add Health anonymizes the place of residence and does not provide information on interview postponements; therefore, we cannot empirically evaluate that speculation. Instead, we added state of residence as a control variable to account for this possibility. After controlling state of residence, the discontinuity of the proportion of mothers with some college degree was no longer statistically significant. Overall, we found no evidence to question the validity of the RDD in the balance check. Additionally, the non-significant discontinuities of mean and variance depression polygenic scores demonstrate the validity of gene-environment analysis, which assumes no gene-environment correlation.