Determining the action timing and variability

In order to infer more about the underlying mechanisms of improvement, we quantified the changes in subjects’ control policies. We therefore examined the applied forces as function of the system state, specifically as function of the pole angle (Fig. 7a). The rationale of our approach is to define events in the state space and analyse the applied forces relative to those events (event-triggered averaging, Fig. 7b,c). We focus our analysis on the situations when the pole is tilted by a certain angle and is rotating downwards. In these situations, which we describe as events, a counter-action is necessary. Using our method we determined when and how variable these counter-actions were performed. The following four steps describe the procedure in more detail.

We defined the integer valued pole angles in the range from −25 to 25 degrees as events (Fig. 7a). For each of these 51 events we determined the occurrences in every trial. Event occurrences between two time-discretization steps (frames) were estimated using linear interpolation.

We then excluded all event occurrences in which the pole is actually rotating upwards and therefore no counter-action is required. We further excluded extreme pole angle velocities. To this end, we excluded all event occurrences in a running window of two minutes across trials, for which the pole angle velocity did not lay between the 20%- and 80%-quantiles of all observed pole angle velocities.

Next, we extracted segments of one-second length from the force input trajectory, which are centred on the previously determined event occurrences, meaning that half of the segment happed before (negative time-lag) and the other half happened after (positive time-lag) the event occurred (Fig. 7b). These segments look roughly like sigmoidal functions going from negative to positive force values or vice versa, corresponding to a left-right or right-left movement of the device knob by the subject.

In the last step, we averaged all segments corresponding to one event (pole angle) in a running window of 2 minutes length across trials and determined the zero crossing of the average segment (Fig. 7c). Thereby we find the time of change from a negative (leftwards) to a positive (rightwards) force (or vice versa) relative to the event occurrence (zero time lag).

Schematic of the action timing and the change of measures as function of the gravity. (a) All pole angles investigated as events (integer valued pole angles from −25 to 25°). The arrows indicate the direction of the pole movement. (b) Pole angle (blue) and input force (orange) trajectories in a representative trial illustrating two event occurrences (black crosses) and corresponding two force segments (thick lines). Negative force values correspond to a leftwards force. Aligning all segments that correspond to one event and averaging the segments over different trials in a window of 2 minutes yields similar curves to those shown in panel c. (c) Average force segments of a representative subject for two periods during learning (first 15 minutes: purple, last 15 minutes: dark green) for illustrating the action timing and variability measures. The circles indicate the time (action timing) when the subjects changed the direction of the force relative to the occurrence of the event (zero time lag). Negative and positive time lags represent the time before and after the event. As expected, early (first 15 minutes, purple) during learning actions are performed rather in reaction to event occurrences (towards positive time lags) whereas learning leads to the ability to make actions predictively (more negative time lag, dark green). Coloured areas illustrates the variability in force segments. The dark coloured areas indicate the action variability. It is the average standard deviation of the force segments ± 60ms around the zero crossing (action timing). The variance in input force round the zero crossing (action variability) is lower for actions late during learning (last 15 minutes), suggesting more consistency. (d) Illustration of the procedure to examine changes in different measures relative to an increase in gravity. We here show exemplarily the trial length (red curve) in relation to the gravity (dashed line). The light grey areas illustrate the periods under investigation. Subtracting the average trial length in the highlighted periods yields the change within (P_2,g − P_1,g) and across (P_1,g+1 − P_2,g) the gravity step(s).

We refer to this time (when the actions change relative to the state of the system) as action timing. Notice, that we do not interpret this measure as the reaction time, even though it might be related. Furthermore, we estimated the variability of the actions by calculating the mean standard deviation of the applied forces in a centred window of 120ms length around the zero crossing (Fig. 7c).