Smart Physiotherapy Activity Recognition System Technical Development and Clinical Validation Study (Aims 1 and 2)

We will further develop SPARS to evaluate the physiotherapy exercise technique and validate the SPARS exercise type and technique evaluations in a clinical population undergoing physiotherapy treatment for rotator cuff pathology. The relationship between adherence and recovery and barriers to home physiotherapy adherence will also be determined in this clinical validation study.

We will conduct a prospective longitudinal cohort study of 120 patients who have been referred to Sunnybrook Health Sciences Centre for the management of rotator cuff pathology. The inclusion criteria will be as follows: (1) age ≥18 years; (2) diagnosis of unilateral rotator cuff tendinosis, shoulder impingement syndrome, or degenerative or traumatic rotator cuff tear; (3) planned conservative or operative management; and (4) capacity to participate in home shoulder physiotherapy. Exclusion criteria will be as follows: (1) upper extremity neurologic deficit; (2) bilateral symptomatic rotator cuff pathology; and (3) failed surgical management of rotator cuff pathology.

Each study subject will be provided with a smart watch to be worn on their affected extremity when performing the prescribed physiotherapy exercise. The smart watch must be worn for all physiotherapy exercise sessions conducted over the duration of treatment, up to a maximum of 5 months. This includes all supervised physiotherapy sessions at the Holland Centre and any physiotherapy exercise sessions conducted by the patient independently (home exercises). No alteration to the prescribed physiotherapy will be undertaken during this clinical validation study.

The smart watches purchased for this study will have a 9-axis IMU (3-axis accelerometer, 3-axis gyroscope, and 3-axis magnetometer). The inertial sensors will be sampled at 50 Hz and recorded to the internal memory storage on the smart watch. The inertial data will be encrypted using the Rivest–Shamir–Adleman public-key cryptosystem. The smart watch will permit up to 150 hours of sensor data recording to its 4 GB internal memory. The sensor data will be retrieved and removed from the smart watch when the patient attends supervised physiotherapy at the Holland Centre (2 times per week).

The researchers will attend a supervised physiotherapy session for each patient once every 2 weeks to record exercise type and technique labels synchronously with inertial data collection, based on the input from the treating physiotherapist. The technique will be labeled as a binary variable, indicating if the treating physiotherapist was satisfied that the exercise was performed correctly (as intended). The treating physiotherapist will also make a record of prescribed home exercises and exercise type and technique evaluations at each supervised physiotherapy session.

Color video recordings of supervised physiotherapy sessions will be made during 30 supervised inertial data collections. The videos will be used for the reliability analysis of the exercise type and technique labels assigned by the treating physiotherapist, and 2 blinded physiotherapists will review and reclassify the video data independently.

Patients will log their home physiotherapy activities in an adherence diary for 2 weeks, during the strengthening phase of their rehabilitation. Validation of the SPARS adherence measurement will include a comparison with these data.

Validated clinical outcomes will be collected to measure the relationship between home physiotherapy adherence and patient recovery. The following validated clinical outcome measures will be collected (1) pretreatment; (2) monthly through treatment; and (3) at 12-months final follow-up (items 1-3 only); these items are routinely collected by the Sunnybrook’s Working Conditions Program (WCP):

Work status (full-time, part-time, off-work, modified, or regular duties)

Numeric Pain Rating Scale (NPRS)

Disabilities of the Arm, Shoulder, and Hand (DASH) scores

Rotator cuff strength testing using manual muscle testing

Shoulder active range of motion measured with a goniometer.

The NPRS is valid and reliable for clinical practice [94,95]. The DASH has proven reliability, validity, and responsiveness in patients with shoulder problems [96-98]. Manual muscle testing and range of motion measurement using a goniometer are standard accepted methods for rotator cuff strength testing and shoulder range motion evaluation in the clinical setting [99], with established validity and reliability [100-104].

The following potentially significant adherence predictors [12,105-110] will be collected for each patient at recruitment: age, sex, body mass index, dominant side involvement, symptom duration, mechanism of injury (traumatic or degenerative), rotator cuff tear thickness and size, type of operative procedures, comorbidity (cumulative illness rating scale [111]), smoking status, alcohol intake, opioid intake, cannabinoid intake, the baseline level of physical activity, education, marital status, job demands, socioeconomic status (current income), perceived social support (Enhancing Recovery in Coronary Heart Disease Social Support Inventory [112]), patient self-efficacy (2-item Pain Self-Efficacy Questionnaire [113]), Patient Expectation Questionnaire [114], and Hospital Anxiety and Depression Scale [115].

The data for this project consist of the following (1) inertial sensor data; (2) video data; (3) clinical outcome scores; (4) adherence predictor variables; and (5) adherence diaries. Items 2-4 include personal information. Video data will be anonymized with face deidentification and audio removal. Data items will be linked with a subject identification number and stored on secure servers at Sunnybrook Research Institute, behind institutional firewalls, with restricted physical access. The database will be encrypted, and access will be limited to researchers carrying out activities related to the proposed study. Continuous data redundancy and versioning will mitigate the risk of large data loss.

When a patient attends supervised physiotherapy at the Holland Centre, their smart watch inertial data are uploaded to Google Cloud Storage. The retrieved data are immediately verified by a software tool that checks and reports the proper functioning of the sensors and correct watch usage to the researchers, ensuring that anomalies are detected promptly. Our research engineer will administer the study database, review the incoming data, and update the research team during monthly project meetings.

The ML algorithms underlying SPARS will be trained in a supervised learning framework to classify (recognize) different activities from the characteristic time series of inertial signals they produce in the smart watch inertial sensors. A detailed description of our implementation and its evaluation are presented in our previous pilot study [15] and our open-source software package for ML time series (seglearn) [116]. Briefly, sensor data are segmented into fixed-length windows (2-4 sec in length) that are learned by a deep convolutional recurrent neural network (CRNN) classifier. Once trained, the classifier can be used to reconstruct a record of activities from the recorded inertial sensor data.

The ML algorithms underlying SPARS will be trained and validated using the labeled inertial sensor data collected during supervised physiotherapy sessions. The trained SPARS algorithms will then generate a record of exercises performed from inertial sensor data collected during unsupervised home sessions and supervised sessions during which synchronous data labeling was not performed. Comparing the quantity and frequency of home exercises measured by SPARS with the prescribed exercises will serve as our measure of participation adherence. Participation adherence will be measured as a percentage of the prescribed activity over each 2-week interval of treatment.

A second supervised learning algorithm will be trained, validated, and optimized to classify the exercise technique from the labeled inertial sensor data collected during the supervised physiotherapy sessions. We anticipate that a CRNN architecture similar to the exercise type classifier will likely be suitable. The validated technique classifier will be used to evaluate the technique adherence from the inertial data collected during unsupervised home sessions.

Anticipated challenges with the proposed supervised learning approach to technique evaluation include class imbalance and recovery-related changes in exercise performance. It is anticipated that many more correct technique training examples may be available for some exercises and that an exercise may be performed incorrectly in ways not observed during the supervised sessions, leading to class imbalance. It is also anticipated that through recovery, the expectation for technique and performance of an exercise may change over time, creating a moving target for the technique classifier. The challenge of class imbalance may be mitigated using several potential approaches including (1) random over and undersampling [117]; (2) cost-sensitive learning [117]; (3) semisupervised learning [118]; (4) anomaly detection [119,120]; and (5) synthetic data augmentation [121]. The challenge of recovery-related performance changes may be mitigated by providing our technique classifier metadata related to the classification task (eg, treatment week, pathology, and surgery) [116], implementing patient-specific ML models using transfer learning, and periodically retraining the ML models through recovery.