Model conceptualization, calibration and validation

Prior to initiating the GMB sessions with the SD Modeling Task Force, the project’s primary SD modeler (second author) developed an initial computational (scoping) model of the HIV treatment cascade to represent the HIV CC for the purpose of demonstration and to initiate the modeling effort. The structure of the scoping model included the stages of the care continuum, from HIV incidence and undiagnosed status, to testing and diagnosis, linkage of PLWH to medical care, engagement in care and on ART, and achievement of viral suppression. This initial scoping model was parameterized using data from the CT Department of Public Health (DPH) 2015 HIV Surveillance Report [46] and calibrated to reproduce the subsequent two years of epidemiological patterns in the three-county area.

During GMB sessions, we distilled and refined Task Force members’ narratives and concept maps to specify and integrate services, programs, and other system contributors [39]. The structure of the initial HIV CC scoping model was expanded to incorporate the commonly available healthcare and social service resources (Basic Services, Modules B-E in Table 1 and Fig 1) and potential or hypothesized community intervention programs (Action Strategies, Modules F-I in Table 1 and Fig 1) that contribute to system dynamics and health outcomes. After modifying the initial scoping model and updating initial parameters, we chose start and end points for a 60-month time horizon (5 years; corresponding to calendar months t₀ = 01/01/2018 and t₆₀ = 01/01/2023). Stakeholder input and deliberation was elicited regarding the values assigned to initial parameter estimates of stocks and flows, as well as other ancillary variables or cofactors used to formulate model equations. To demonstrate effective SD model performance, we compared base case scenario simulated output for the prior calendar year (2017; t_-12 = 01/01/2017 to t_-1 = 12/31/2017) with reported trends over the same period in the catchment area (described more fully below). This technique was also used to resolve any computational anomalies due to initial parameter values [47].

SD model validation occurs through an iterative process of model conceptualization, calibration, and simulation, via cycles of deliberation, data input, review, and decision-making by key stakeholders [24, 45, 48–53]. Our iterative model development and revision process focused on four types of model validity.

Structural validation determines that the model has been formulated with accuracy and adequately represents the model developers’ conceptual description of the system. This was achieved during iterative GMB sessions for model conceptualization and revision, with support from the theoretical and empirical literature. Specifically, as we developed the model designs of each of the subsystems (modules in Table 1), which we derived from Task Force narratives and from known relationships supported by extant literature on TasP, we presented them at subsequent meetings for Task Force member deliberation and feedback on model components and the causal relationships among those elements. Our research team also iteratively rechecked variable parameters and formulations for inaccuracies, redundancies, and omissions to discuss and resolve them during weekly modeling sessions.

Behavioral validation involves assessing the model’s simulated behavior and assuring that it has sufficient accuracy for its intended purpose over the scope of its intended applicability. Behavioral validity supports the model’s credibility with stakeholders and others. This was initiated during the GMB process by comparing simulated reference modes (output graphs) with epidemiological trend data to assure correct initial calibration to the local context. We examined preliminary simulation runs iteratively in sessions with the SD Modeling Task Force to check and re-check the extent to which the model’s behavior (simulated output) conformed to key assumptions and sources of evidence guiding the study [39]. For example, we compared simulated output of new infection rates to historical 5-year trends in the catchment area, as well as state-wide trends in the epidemic as appropriate, both of which have plateaued into a steady state during the past 4–6 years. Other TasP outcomes calibrated to replicate effectiveness of the regional HIV CC included the numbers of PLWH engaged in and lost to care over time and the percent virally suppressed. Parameter estimates derived from these sources were evaluated by running the model. (We present and discuss this comparison of our base case simulation to epidemiological reference modes more fully below.)

Construct validation involves assessing the extent to which the model is supported by relevant theories, frameworks and assumptions underlying the problem of interest. In this case, we relied on a growing body of literature regarding the concept of TasP, which posits that infections at the community level can be reduced by identifying all PLWH in the community and providing them continuous ART to achieve and permanently maintain viral suppression, thereby reducing “community viral load” [54, 55]. This literature focuses on community, organizational, and individual level factors that facilitate moving PLWH quickly through all stages of the HIV CC (i.e., minimizing delays in achieving viral suppression), and retaining them in medical care over time [4, 8, 56, 57].

Data validation ensures that the data (and/or special knowledge) necessary for model formulation and calibration (capturing past trends), evaluation (predicting future trends), and application (virtual experiments and policy analyses) are adequate and reliable for the intended purpose of the model. Use of primary and secondary data, literature review, and stakeholders’ best estimates are common practices in the parameterization of SD model components [58]. This was assured through our use of key regional epidemiological and health services utilization data, including CT DPH surveillance data (e.g., state reportable viral load counts of PLWH in care, treatment cascade data, linked to care and late testing rates, etc. [46, 59]) and Ryan White health services data (e.g., case management, housing, and transportation needs, and gaps in substance use treatment and mental health services [60]). These sources generally use metrics commonly used state-wide and nationally [61–63], thereby increasing generalizability and potential applicability of the SD model beyond the geographic region in which it was developed.

We also analyzed qualitative and quantitative data from our concurrent study to identify variables for the model and parameters for some variables. For example, data from the study’s cohort provided estimates for some transition rates of PLWH through care continuum stages and global measures of perceived external stigma. Task Force members’ experiential knowledge of the service delivery process and other contextual factors that have no known or standard metrics provided a means to parameterize initial (starting) values of other variables, such as time delays, caseloads, missed appointment rates, and community attitudes like medical mistrust and HIV knowledge, among others.