Image-to-patient registration

Image-to-patient registration computes the transformation from the image space to the position of the patient. In this study, the image space is represented by a on rails Siemens SOMATOM CT scanner and the position of the patient is on the operation table prepared for surgery or in the biomechanical diagnostics laboratory. The most common approaches in the literature for image-to-patient registration is rigid transformations based on landmarks [3, 8, 9].

For the clinical qualitative assessment (“Accuracy evaluation method” section), registration was also performed rigidly through an application developed in Unity 2018.3 for HoloLens, with which a user can select positions on the directly the hologram and then sample them using the optically tracked and calibrated pointer tip (NDI Tool pointer 8700340, visible in Fig. 2). For the patient phantom, registration was performed twice by the surgeon. One registration was conducted for the patient’s hip (which was successively spatially anchored in situ) and a second registration was performed for the femur. The amount of time necessary for the registration was in total around 3 minutes. Registration was performed through singular value decomposition using seven target positions (similarly to [10]). The targets used for the patient phantom are metallic washers (six millimetres of diameter, also shown in [10]), glued to the patient phantom, which were visible in the CT scan and were segmented and clustered using fuzzy means classification. The metallic washers are replaceable by any hypo- or hyperintense markers, visible in CT (e.g. electrocardiogram patches). Once registration of the models (TIP) has been computed, every user wearing a HoloLens with optical markers is able to visualise the dynamic MR. (It does therefore not need to be repeated for each user.)

Evaluation experiment through the Validation phantom. The left image shows the 3D printed frame on the HoloLens and how the quantitative experiments were performed. The images on the right show, in the upper, the holographic tip at a distance from NDI’s pointer, and in the lower, the position of the Polaris in HoloLens Camera coordinates

The same procedure can be applied on patients without the washers using anatomical landmarks: greater trochanter, the spina iliaca anterior superior on the pelvis or the medial and lateral epicondyle of the distal femur. These positions can easily be selected by the orthopaedic surgeon in the hologram (or pre-operatively on the CT scan or 3D model), and then can be sampled with any optically tracked instrument while the patient is on the operation table. Transformation TPI, in Eq. 1, is the result to image-to-patient registration.

To dynamically update the position of the MR while the HoloLens or the patient is moved during diagnostics, the following equation (according to [10]) was used:

In which, O is the coordinate system for the OTS, M for the HoloLens optical markers, I is the image coordinate space, C is the HoloLens camera, P is the origin of the patient space coordinates (in our study, the OTS origin was used as global reference) and C is the camera pose. Through image-to-patient registration and the reference frame on the patient, the holographic model can be aligned to the patient’s body and dynamically updated while moving the limb by changing TPO, as shown in Fig. 1. TCM is computed through hand-eye camera calibration with the additional transformation to left-handed coordinates required by Unity, whereas TPO and TOM are provided by tracking respectively the markerplate on the patient and on that on the HoloLens camera. Finally, TIP represents the image-to-patient transformation matrix computed by the methods previously described (either using a optical pointer to sample anatomical locations, or using an optical markerplate visible in the CT scan).