Step 1. Capturing image data from the surgical robot

Prior studies in the field of AI and RAS have used the dVLogger to capture image and system data from the dVSS or used datasets that were made in collaboration with Intuitive Surgical Systems Inc [5, 9, 13, 20]. The dVLogger is a software device that records video in endoscopic view with 30 frames per second (FPS), system data such as kinematic data (instrument movement, instrument travel time, velocity, path length) and event data (frequency of clutch use, third arm swap, camera movement, and energy use) in 50 Hz through an Ethernet connection [9].

Our method will seek to capture raw image data of the surgical system from the surgeon’s console (robot control platform).

Since the dVSS uses a stereoscopic camera with two oculars, raw video footage of the right and left endoscope ocular can be accessed through the back of the surgeon’s console using two HDMI to USB Video Capture Cards (VCCS), one for each ocular output. Many VCCS exists (Maxwell, Epiphan, etc.), and most are equally good, our previous choice of VCCS have been dependent on type of video signal, i.e., SDI or HDMI, or based on the operating system of the recording PCs. In this case we used HDMI to USB video capture card from Ozvavzk. Both capturing cards should be connected to a computer through two different USB slots, best not to use a USB hub. For physical connection between the capture cards and the robot, we used two HDMI cables connected to the robot through two HDMI-to-DVI Cables (0.15 m HDMI Male to DVI Female). The setup is illustrated in Fig. 1. The image output from the robot were recorded using open-source software (Open Broadcaster Software Studio, Wizards of OBS, OBS Studio v. 27.2.4, 64-bit), which allows for multiple inputs to be displayed and recorded at synchronously.

Data collection from da Vinci Surgical System and depth cameras. A Image outputs from the surgical robot B The surgical console used by the surgeon. C Depth camera capturing 3D footage of surgeon’s movements. D Capture devices used as a gateway to capture image output. The capture device is connected through a DVI to HDMI converter. E Local computer for recording and storage of footage. C* shows the cardboard camera holder

Resolution and FPS can be configured in OBS studio before recording. In this study, 2560 × 720 was used with 15 FPS. For the sake of simplicity, only one of the endoscopic ocular outputs was used in this study, using two however could enable 3D data analysis of the field, using stereogrammetry. The video was cropped in separate left and right views using Free Video Crop, RZSoft Technology Co. Ltd, v. 1.08, to 1280 × 720 format. Whenever sequences were recorded where surgical instructions were given during the RAS courses or changes of instruments took place where nothing of surgical relevance occurred in the video footage, these sequences were cut out using Windows Video Editor, Microsoft Corporation. Other free video editors and recorders are also available such as the command-line software called FFmpeg, which was also used in this study.

Connecting VCCS the computer was the same for both the da Vinci Si and the da Vinci Xi, only difference being in the video output, as seen in Fig. 2. There were mainly differences in the on-screen placement of bars and indicators and are described in further detail in the upcoming sections.

The lines of readiness and activation of the instruments. In picture 1 the green horizontal and vertical line at the top and left side indicates that the surgeon’s foot is hovering above the coagulation-pedal which is pressed in picture 2. In pictures 3 and 4 the same is seen for the left cut-pedal of the da Vinci Si system