Procedure

The IVRT training application has been built using Unity3D, a Unity Technologies game engine that allows developers to create and manage web-based gaming environments. Participants will engage in the IVRT training intervention using the HTC Vive Pro, a commercially available VR technology from HTC Corporation that places users in a fully immersive VR environment using the HMD headset, tracking devices, and controllers (Figure 1) [40]. The wheelchair joystick used for this study has been built by an engineering team using a 3D printer and specifications that closely match a real-world power wheelchair joystick. An HTC Vive Pro tracking device is attached to the top of the joystick to allow for accurate control of the power wheelchair within the IVRT environment (Figure 1). Henceforth, this joystick will be referred to as the “tracker joystick.” The user can also engage in hand-based activities (eg, turning on the power wheelchair) in the IVRT environment by moving a controller held in their nondominant hand.

An HTC Corporation tracker and IVRT joystick (A) attached to the Rifton Activity Chair (B) and in-use during gameplay of the training intervention (C). IVRT: immersive virtual reality technology.

In phase 1, participants will sit in a Rifton Equipment Activity Chair [41] that replicates the seating and joystick setup of a power wheelchair. The Activity Chair is a positioning chair that provides adaptable seating for a range of patient populations and has been slightly modified to accommodate an arm attachment for easy operation of the tracker joystick (Figure 1). In phase 2 and phase 3, participants will sit in their own power wheelchair, and the tracker joystick will be attached to their personal joystick controller stem. The tracker joystick is fitted to easily screw onto any standard-size joystick controller stem, allowing for users to participate in the IVRT intervention without the need to transfer to a new chair. During the IVRT simulation, users will progress through the intervention in a power wheelchair that replicates the natural movement of a real power wheelchair. The in-game power wheelchair movement patterns (eg, acceleration and deceleration speeds and turning trajectories) have been designed in collaboration with an occupational therapist to ensure that all movements are simulated with accuracy.

The IVRT training intervention has been designed to help users develop core power wheelchair skills in a motivating and engaging manner. The baseline version of the training intervention was created in collaboration with the research committee (consisting of power wheelchair rehabilitation experts, including a pediatric physiatrist and a pediatric occupational therapist) and software development team. The research committee identified power wheelchair tasks that were commonly taught to novel power wheelchair users as beginner to moderate–level skills (eg, moving forward, moving backward, and turning 90°) [42]. Then, the software development team integrated the skills into the IVRT system within an environment that was approved by the research team to be acceptable for the pediatric population (ie, containing age-appropriate graphics and characters). The baseline training intervention is intended to introduce participants to the potential of an IVRT system for pediatric power wheelchair training and gather expert feedback on how to improve the skills, environment, and overall user experience.

The intervention places participants in a colorful cartoon environment with robotic characters. Users begin the intervention in the main lobby, where the instructor of the game explains the instructions and wheelchair controls using audio and visual cues. Then, the participants are brought to the outside world to begin level 1, where they must use the tracker joystick to control their power wheelchair and move through specific areas in the game to complete each level. Users will be challenged to participate in tasks that integrate various power wheelchair skills, such as backing up in a narrow hallway, driving over a ramp, and following a figure-eight path (Figure 2). In addition, players are also provided with various opportunities to increase their scores during gameplay, including driving on the correct path and collecting fruits to power up their wheelchair. The inclusion of a reward system that allows for positive reinforcement has been shown to enhance a child’s attention and motivation when training within a VR environment [43-45]. Therefore, it is anticipated that these additional components will help to increase user engagement and improve performance outcomes.

Screenshots of the training application showing stages of the figure-eight task from three different camera viewpoints: follow camera (A), free camera (B), and first-person view (C).

The IVRT system offers a range of options to measure client performance, which will otherwise be difficult to capture in real-world training. In total, 4 different VR camera angles (overhead, follow camera, first-person view, and free camera) provide the operator with a variety of viewpoints to assess the user’s driving skills (Figure 2). Chart selections can also be accessed by the operator to evaluate in-game user behavior, such as client focus (measurement of attentional focus) and pathways (real-time charting of user’s driving patterns). Furthermore, client skills and goals can be easily tracked using the metrics options, which offer real-time speed, collision, and completion time statistics. Taken together, the IVRT system can provide an accurate and comprehensive assessment of driving performance without disrupting the in-game user experience.

The aim of phase 1 is 2-fold: (1) to use clinician feedback to develop a list of potential power mobility skills to be implemented in future versions of the IVRT training intervention and (2) to assess the usability and acceptability of the intervention from the clinician’s perspective. Phase 1 participants, comprising health care clinicians, will be exposed to a baseline version of the IVRT training intervention and participate in a training session during which they will progress through each of the levels, with instructions to carefully evaluate various components of the intervention. Participants will also be given the option to freely explore the game environment following the completion of each level, if they wish to do so.

Participants will be asked to assess the ability of the intervention to teach a range of core power wheelchair skills. Currently, there are no standardized methods that have been established among professionals to assess power mobility skills among children [38]; however, 4 main measures are commonly used: Assessment of Learning Powered Mobility Use [46], Powered Mobility Program [47], Power Mobility Training Tool [48], and Wheelchair Skills Checklist [49]. In the adult population, the Wheelchair Skills Test for Powered Wheelchairs is well validated for the evaluation of power wheelchair capacity [42]. Owing to the lack of an established measurement tool for the pediatric population, the research team identified core skills frequently listed in both the Wheelchair Skills Test for Powered Wheelchairs and common pediatric assessment tools to create a comprehensive skills list for clinician use. This list was developed in consultation with a pediatric rehabilitation specialist to ensure that all skills were appropriate for inclusion.

Immediately following the IVRT training intervention, participants will be provided with a list of 28 individual power mobility skills and asked to indicate their level of agreement with the following statement: “Based on my experience working with power wheelchair users, I believe the immersive virtual reality technology (IVRT) application can be used to teach children and adolescents to (insert skill here).” Consensus on the application’s ability to assess each item will be defined as ≥75% of participants indicating that they agree (score=4 out of 5) or strongly agree (score=5 out of 5) for a given item, leading to the inclusion of the skill in subsequent phases. If ≥75% of participants indicate disagreement (score=2 out of 5) or strong disagreement (score=1 out of 5) for a particular item, this will be deemed as consensus that the intervention is not appropriate for the development of that skill, and it will not be included in future phases. If a neutral response (score=3 out of 5) is indicated by ≥75% of participants, the skill will be refined for future versions of the training intervention using participant feedback. If an item is rated <4 out of 5, participants will be asked to provide a specific recommendation for modification of the IVRT intervention pertaining to that skill. If a participant believes that a skill cannot be feasibly modified, they will provide no response in the recommendation section.

Participants will also be provided with 3 separate questionnaires to assess perceived usability, acceptability, and overall user experience. Quantitative data will be collected from (1) the Presence Questionnaire (PQ) [50], (2) an ad hoc usability and acceptability questionnaire, and (3) an ad hoc user experience questionnaire. Qualitative data will be collected from (1) an ad hoc user experience questionnaire; (2) participants’ informal in-game comments and reports, as recorded by the researcher; and (3) a semistructured interview (round 2 only). A complete list of the assessment measures included in each phase is provided in Table 1.

Assessments by phase and round.

^a✓: the check mark specifies the assessment measures that will be used in each phase and round of the study.

The PQ and ad hoc usability and acceptability questionnaire will be provided as paper-based materials. The PQ is a well-validated assessment tool used to measure presence within a VR environment. In this study, the PQ has been adapted to include the 4 subscales most relevant to the IVRT training intervention: realism, possibility to act, quality of interface, and self-evaluation of performance [50]. Participants will be asked to report their experience related to multiple components of each subscale using a 7-point Likert scale rating. The ad hoc usability and acceptability questionnaire has been adapted from the Perceived Usefulness and Perceived Ease of Use scales [51] and System Usability Scale [52]. This questionnaire will be used to explore participants’ attitudes toward using the IVRT intervention as a training tool for the pediatric population. Participants will be asked to rank each statement on the questionnaire from 1 to 5, ranging from “strongly agree” to “strongly disagree.”

The ad hoc user experience questionnaire will be hosted on the secure web-based software platform, REDCap (Research Electronic Data Capture; Vanderbilt University) [53]. The web-based questionnaire format was chosen to allow participants to respond to open-ended questions by typing rather than writing; however, they will be provided the option to complete a paper-based questionnaire if it is preferred. This questionnaire will be used to assess participant demographics, user tolerance (a component of usability), and overall user experience. To capture user tolerance to the IVRT intervention, the presence of VR-induced symptoms and effects (VRISE) will be assessed during and after the intervention. VRISE includes symptoms such as nausea, dizziness, disorientation, and fatigue and can occur as a side effect of VR exposure [51]. VR systems using HMD have been found to increase the prevalence of VRISE compared with nonimmersive systems; however, our intervention’s length falls below the theoretical limit of exposure to VR for adults (55-70 minutes) [54,55]. Although our intervention session is approximately 20 to 30 minutes in length, it is anticipated that some participants may still experience VRISE symptoms. User experience will be assessed in this questionnaire using open-ended questions that have been developed to better understand participants’ experiences within the IVRT application, such as ease of use, appropriateness of tasks and graphics, and suggestions for improvement.

Following the completion of the study session, participant feedback data will be summarized, anonymized, and presented to the research committee. The committee will use these data to produce recommendations for a new iteration of the application, which will be implemented by the software development team.

Participants will be invited back to the laboratory to complete a second session during which they will engage in the IVRT training intervention that has been updated based on feedback from the round-1 sessions. We aim to have at least 60% (ie, >7 out of 12 participants) of phase 1 participants return for second round of testing, based on previous health care studies that have received a similar percentage of participant retention for multiround testing [56,57]. The second round of testing is intended to check for accuracy and ensure that participants are satisfied with the changes implemented based on feedback from the first session. Participants will complete the updated IVRT training intervention, followed by a semistructured interview designed to gather in-depth details of the user experience. The interview questions will be developed based on data from round 1 and will aim to capture feedback regarding the system’s new updates (eg, opinions regarding any new skills or levels added and updated graphics or audio) and address any potential areas for further improvement. Finally, participants will also complete the same PQ, ad hoc usability and acceptability questionnaire, and ad hoc user experience questionnaire as in round 1. Participant feedback will be presented to the research committee, and if any items in the IVRT intervention are found to still require significant changes, they will be updated as necessary by the software development team.

Phase 2 will assess the usability and acceptability of the IVRT system from the pediatric power wheelchair user’s perspective. Participants in phase 2 will be comprised of current pediatric power wheelchair users, who will test and evaluate the updated IVRT training system that has been adjusted based on feedback gathered from clinicians in phase 1.

During the IVRT trial, participants will be placed in the IVRT setup and provided with 5 to 10 minutes to freely explore and acclimate to the VR setting. Once the participants indicate that they are ready to begin, they will start the training intervention. The skills selected in phase 1 for inclusion will be integrated into the intervention, and participants will be encouraged to complete each skill as they progress through the levels. Following completion of the intervention, participants will be given the option to exit the system or continue exploring after a mandatory 10-minute break. After the break, participants may freely explore the VR environment for up to an additional 15 minutes at their own discretion. The mandatory 10-minute break has been included in the session to reduce consistent VR exposure and limit the potential of VRISE among children [58].

Immediately following the IVRT intervention, participants will complete the same 3 questionnaires as in phase 1, but with age-appropriate adaptations (eg, changes to wording or question structure). Age-appropriate adaptations will be approved by a child life specialist to ensure suitability for the pediatric population. All questionnaires will be asked aloud by the researcher, and the pediatric participant’s verbal responses will be recorded. As in phase 1, the questionnaires will explore the perceived usability and acceptability of the IVRT system for power wheelchair skill development, IVRT tolerability, and general user experience.

Participants will also engage in a semistructured user experience interview, in which questions will be asked aloud and responses will be audio-recorded for qualitative analysis. The semistructured interview intends to gather in-depth details on the perceived usability, acceptability, and experience in the IVRT environment (eg, most favorite and least favorite parts of the game and why and areas for improvement). The participant’s caregiver (parent or proxy) will also be encouraged to provide any additional details that the pediatric participant may not remember (eg, dates and early-life experiences).

Pediatric participants and their caregiver will also participate in a training methods questionnaire and interview during the session. The training methods questionnaire will use Likert scale questions (asked aloud by the researcher) to explore both the child’s and caregiver’s perceptions of previous power wheelchair training methods. Then, a semistructured interview will be conducted to further explore their experience with power wheelchair training (eg, most exciting or challenging parts of training and confidence in skills after training). This information will provide great understanding of past training techniques and experiences from 2 different perspectives. In all cases where pediatric participants cannot remember specific details, caregiver input will be sought to ensure completeness of the data set.

After all participants have completed their session, data will be collected, summarized, and presented in the same manner as in phase 1. The research committee will use this feedback to identify and implement changes to the application for the final phase.

Phase 3 of this study will also assess the usability and acceptability of the IVRT system from the perspective of current pediatric power wheelchair users. In addition, participants will complete a real-world trial and an IVRT trial to compare power mobility skill transfer between the VR and real-life environments. Both trials will occur over 1 study session, and the order of trials will be counterbalanced among participants.

All phase 3 participants will test the IVRT training system that has been updated based on phase 1 and phase 2 feedback. To compare and assess participant’s skill transferability, an experienced clinician will review in-game and real-world performance. It is anticipated that the IVRT intervention will be designed to closely resemble a real-life setting; therefore, the skills transferability assessment will measure the similarity of participant’s skills performance across both the VR and real-life environments.

A computerized recording of the IVRT intervention will be independently assessed by the clinician following the completion of the session to compare in-application versus real-world performance metrics. During the real-world trial, participants will be asked to complete each skill that has been included in the IVRT intervention. Skills will be performed in an environment within the hospital grounds and assessed by an experienced clinician. In-game and real-life performance will be assessed for capacity level (skill performed: “yes” or “no” and skill proficiency rating from 0-3) and time to complete each skill.

Participants will also be asked to complete the same questionnaires and semistructured user experience interview as in phase 2 to explore their experience in the IVRT environment and the perceived usability and acceptability of the intervention. Caregivers will again be encouraged to assist in the user experience interview to provide further details as needed and to complete the same ad hoc training methods questionnaire and interview as in phase 2. Exploratory analyses are planned to be conducted with phase 3 data to further evaluate the final version of the IVRT intervention.