Item design

Items were carefully generated using a consensus statement regarding the monitoring and reporting of negative effects [32], findings from a treatment outcome study of patients with social anxiety disorder that probed for adverse and unwanted events [42], the results of a qualitative content analysis of the responses from four different clinical trials [44], and a literature review of books and published articles on negative effects. This is in line with the recommendations by Cronbach and Meehl [45], advising researchers to articulate the theoretical concept of an instrument before developing and testing it empirically in order to increase content validity. Also, instead of restricting the number of items to be included in a final version, the concept of overinclusiveness was adapted, that is, embracing more items than necessary to aid the statistical analyses necessary for detecting those that are related to the underlying construct(s) [46]. Subsequently, 60 items were generated, characterized by interpersonal issues, problems with therapeutic relationship, deterioration, novel symptoms, stigma, dependency, hopelessness, difficulties understanding the treatment content, as well as problems implementing the treatment interventions. An additional open-ended question was also included for the investigation of negative effects that might have been experienced but were not listed, i.e., “Describe in your own words whether there were any other negative incidents or effects, and what characterized them”. Further, in order to assess the readability and understanding of the items, cognitive interviews were conducted on five individuals unrelated to the current study and without any prior knowledge of negative effects or psychological treatments, i.e., encouraging them to read the items out load and speak freely of whatever comes to mind [47]. Cognitive interviews are often suggested as a way of pretesting an instrument so that irrelevant or difficult items can be revised and to increase its validity [48]. In relation to the proposed items, several minor changes were made, e.g., rephrasing or clarifying certain expressions. In addition, the instrument included general information about the possibility of experiencing negative effects, and was comprised of three separate parts; 1) “Did you experience this?” (yes/no) 2) “If yes–here is how negatively it affected me” (not at all, slightly, moderately, very, and extremely), and 3) “Probably caused by” (the treatment I received/other circumstances). The instrument is scored 0–4 and contains no reversed items as this may introduce errors or artifacts in the responses [49].