Motor Control & Sensory Integration

298 N. O'Dwyer & P. Neilson vary significantly with frequency, so that the coupling between the hands was essentially a non-dynamic (zero order) one. 3.4.2 Left Hand and Right Hand Tracking Performance (L, R, IJL+R, R/L+R) Performance of both one-handed and two-handed tracking improved following practice of two-handed tracking. Tracking error decreased for all tracking conditions (L, R, IJL+R and R/L+R), though the changes were not significant (F[ 1,5]--4.58, p=0.09). However, the overall coherence (Figure 5) increased significantly (F[1,5]=8.72, p<0.05), so that on average about 80% of the subjects' response movements were linearly correlated with the target after practice. It can be seen that following practice of the two-handed condition, performance was similar for the left hand and fight hand in both one-handed and two-handed tracking. 100 ]~ 1 Hand ] g5 ..c>- 2 Hands \"-\" 9(3 L| 8 5 7s 70 o\" 0 65 After Before After Before Left Right Figure 5. Overall coherence of left and right hand responses with the target, before and after practice, during one-handed(L, R) and two-handed(L/L+R,R/L+R)tracking. The gain of the target to response relation quantifies the amplitude of the response with respect to the target. Following practice the gain (Figure 6A) did not differ significantly between the left and fight hand (F[1,5]=0.04, p----0.85) nor between one-handed and two- handed tracking (F[1,5]-2.98, p--O.15). The shape of the gain versus frequency curves is typical for tracking these kinds of target signals. We have described previously how this particular shape can be attributed in large measure to prediction of the target signals and so it will not be considered further here (Neilson, Neilson and O'Dwyer, 1993 and this volume). It can be seen that this shape was similar for all conditions (L, R, LtL+R and R/L+R), except at the highest frequency (1.1Hz), but the absence of any significant interactions with frequency showed that this difference was not significant.

Learning a Dynamic Limb Synergy 299 A.o 1 ,--o.- 1 Hand l A -,o.. 2 Hands 0~ .:1 .3 .~i .'7 .~) 1'1 Hz 9~1 .3 .5 .'7 .~) 111 Hz t,_ m 0.9 \"0 N. o.8 E ~ o.7 e- ~ (.9 0.6 0 \"\" A t/) (D -20 -40 O 9 (D 91 .3 .5 .7 .9 1.1 H z O \"0 .1 .3 .5 .7 .9 1.1 Hz Left -60 Right tl) ett1/-):l -80 ll. -1 O0 Figure6. Targetto responsegain (A) and phase (B) versusfrequency,afterpractice,during one-handed (L, R) and two-handed(L/L+R,R/L+R) tracking. The phase of the target to response relation quantifies the timing of the response with respect to the target. As with the gain, the phase following practice (Figure 6B) did not differ significantly between the left and right hands (F[1,5]--0.07, p--0.80). However, unlike the gain, the phase did differ, for both hands, between one-handed and two-handed tracking, as shown both by a main effect (F[1,5]=11.86, p<0.025) and in the interaction of phase with frequency (F[5,25]=10.18, p<0.001). The small difference (about 10~ at 1Hz) means that the response of either hand lagged on average about 28ms more behind the target during two- handed compared to one-handed tracking. 3.5 Discussion The main finding of this study was that, following just over two hours of distributed practice at two-handed tracking with a simple synergy between the hands (L+R), performance of one-handed and two-handed tracking was very similar for both hands. Initial small (non- significant) differences between the left and fight hand (in these fight-handed subjects) and between one- and two-handed tasks had largely disappeared after practice. The one important remaining difference was the small increase (about 28ms) in tracking time delay in two-handed compared with one-handed tracking. This difference may reflect an extra processing load on

300 N. O'Dwyer & P. NeUson the CNS when the two hands are required to perform together in a simple synergy compared with single-handed performance. It may reasonably be questioned whether in the two-handed task the left and right hands were not really working together in a synergy but rather were working independently to track a common target. The coherence between the hands might thus have been induced simply by their similar behaviour in following the common target. Against this view, however, is the finding that the coherence between the hands was significantly greater than the coherence of either hand with the target. This showed that the left and right hands were making movements that were correlated with each other but not correlated with the target. In other words, they were making correlated tracking errors. This was especially apparent at higher frequencies where tracking errors increase but the coupling between the hands was maintained. The nature of this coupling was shown (by the gain and phase curves) to be a simple, non-dynamic (zero order) relation. 4. EXPERIMENT 2 4.1 Subjects Six right-handed, male volunteers aged 22-37 years (mean=27.3) participated. As with Experiment 1, all were university students or graduates and the inclusion and exclusion criteria were as before. All were fully informed of the procedures before data collection was undertaken. 4.2 Setup The tracking setup was the same as before except that now we wished to induce a dynamic synergy between the hands. In order to achieve this we introduced a low-pass filter between the right joystick and the fight screen cursor. This meant that the deflection of the fight response cursor on the screen was now the filtered fight joystick rotation. The effect of the filter on the relation between the left and fight screen cursors can be seen in Figure 7, where the thin lines show the excursion of the response cursors on the tracking screen and the thick lines show the corresponding sequence of joystick rotations (and hence movements of the hands). Consider the situation where both hands move in perfect synchrony, as illustrated in the top part of the figure. On the left side, the hand and screen cursor move identically in a one-to-one relation. On the fight side, the hand moves in perfect

Learning a Dynamic Limb Synergy 301 synchrony with the left, but the cursor movement, due to the f'dter, is smaller and smoother and is also lagged in time. This smoothing is greater for faster (higher frequency) movements, such that the fastest movements are virtually eliminated. Thus, the hands move identically on both sides but the screen cursors do not. The relation between the two hands is a simple one- to-one synergy, while the relation between the cursors is the dynamic, linear relation of the falter.1 NOCOMPENSATIONFORFILTER Lcursor / \" filte; . . _ _ . . . / - - ~ ~ , Rcursor 1t~ /'t filterT Lhand /-' ~.,, , / .-..- v ~,,~ (\"~ 1-to-1 /'J ~ _ ,..-.,~ (\"~ Rhand Nf FULLCOMPENSATIONFORFILTER ,/t-~ Lcursor /-' \" ~ / \" - k ) - - \" ~ i/'~\\ ~-to-~ /-J 'k~,j,~_~/,....\".\"~'kJ\\(-'~ Rcursor \\ 1t~ T~ inverse filter T L hand of filter Rhand 1 sec tt Figure 7. Hypothetical left (L) and right (R) hand (thick lines) and screen cursor (thin lines) movements with no compensation and with full compensation for the filter. The nature of the left-right and hand-cursor relations are also indicated. The scenario in the top part of Figure 7 applies where the subject makes absolutely no compensation for the effect of the f'dter - the hands move identically on both sides but the screen cursors move differently. However, the tracking task requires the subject to make both screen cursors move identically in order to track their common target. To achieve this, the ~rhe equation for the filter (digital, first-order, exponentially-weighted, moving average) is as follows: y, = au, + (1-a)y,. I where u, = the input to the filter at time t, y, = the output of the filter at time t a n d Y,-I = t h e o u t p u t of t h e filter a t t i m e t - l ; a 6= 0 . 1 1 7 6 ) is t h e p a r a m e t e r t h a t sets the time constant of the filter.

302 N. O'Dwyer & P. NeUson hands must move differently. In order to overcome the filter effects and make the fight response cursor behave as though there was no filter present, the subject must compensate by making considerably larger and faster hand movements on the fight, as illustrated in the bottom part of the figure. This shows the situation if the subject compensates fully for the effect of the filter. If this can be achieved, then the left and fight cursors will have a simple one-to-one relation but the left and fight hands will have a dynamic linear relation. This dynamic relation will be the inverse of the low-pass filter. In other words, instead of moving in a simple synergy as in Experiment 1, the hands will be moving in a dynamic synergy. It is important to point out that the degree of coupling between the hands and the nature of the coupling are in principle quite independent and need to be considered separately. Thus, in both scenarios illustrated in the Figure 7, the left and fight hands are still perfectly coupled (ie, left-fight coherence = 1), regardless of the degree of compensation for the filter. In the case of no filter compensation, however, the left-fight synergy is a scalar, one-to-one relation, whereas in the case of full compensation, the synergy is a dynamic relation. Furthermore, this distinction between degree and nature of coupling still applies if partial rather than full compensation occurs, which as will be,seen, turns out to be the case in the present experiment. Figure 8 shows the characteristics of the filter in the frequency domain. The gain and phase of the filter are plotted above the spectrum of the target waveform, so that the effects of the filter over the target bandwidth may be appreciated. At 1Hz, the gain of the filter is about 0.6 and the phase lag is about 47~. In order to compensate for the filter at this frequency, therefore, the subject needs to produce hand movements at 1.7 (= 1+0.6) times the normal amplitude and with 130ms (= [47+360]x1000) lead. 4.3 Procedure The design and procedure were similar to Experiment 1 but now there were five measurements: (i) two-handed tracking (L+R) (ii) two-handed tracking with the joystick filter on the fight hand (L+RF) (iii) single-handed tracking with the fight hand (R) (iv) single-handed tracking with the fight hand and the joystick filter (RF) (v) single-handed tracking with the left hand (L).

Learning a Dynamic Limb Synergy 303 As before, for the two-handed tests, the response cursors were independent, so that rotation of the left joystick moved only the left cursor and rotation of the fight joystick moved only the fight cursor; for all single-handed tests, the response cursors were linked together so that rotation of one joystick moved both the left and fight cursors. 1 Filter: Gain vs Frequency Filter:. Phase vs Frequency 0 0.8 ~ -20 0.6 u~ .~ r Q. C3-40 ,~ 0.4 -60 0.2 o %i i Target: Power vs Frequency 1 Target: Power vs Frequency 0.8 ~ 0.8 ,, 0.6 0.4 0.6 1234 0.2 Frequency - Hz \"g~ 0 . 4 t't zO 0.2 ~0 1 2 3 '$ Frequency - Hz (3 0 Figure 8. Frequencycharacteristicsof first-order, low-pass digital filter (top) and average frequency spectrum of test targetsignals(bottom). Following an initial one-minute familiarisation test (L+R) using the 'practice' target, the subjects performed each of the five tests - (L), (R), (L+R), (RF) and (L+RF) - using the two 'test' targets, to give a total of 10 tests. The order of the tests was counterbalanced across the group in order to minimise possible sequence effects. On the subsequent 13 days (spread over two months), they practised only the two-handed test with the filter (L+RF) using the 'practice' target. On the final day, eachsubject repeated the test sequence that they had performed on the first day. 4.4 Analysis of Tracking Performance Nine sets of analyses were required, as illustrated in Table 2. The relation between the target and response signals was analysed for each of the single-handed tests. Similarly, the

304 N. O'Dwyer & P. NeUson relation between the target and the left response and the target and the fight response was analysed for the two-handed tests. In addition, the relation between the left and fight response signals was analysed for the two-handed tests. Thus, the two-handed tests required three sets of analysis each. Table 2. Tracking analyses Single-Handed Two-Handed (R) (L) (RF) (L+R) (L+RF) Target-Left Target-Left Target-Left (L/L+R) (IL+RF) (L) Target-Right Target-Right Target-Right Target-Right (R) (RF) (R/L+R) (R/L+RF) Left-Right Left-Right (L-R) (L-RF) 4.5 Results Traces of the target, left cursor and fight cursor for the two-handed tracking conditions are shown in Figure 9. Before practice, without the filter (L+R), both cursors show typical tracking responses to this type of target - they reproduce the target waveform with a lag due to reaction time delay (ie, phase lag > 0) and with added 'noise' or deviations from the target waveform (ie, coherence < 1). It can also be seen that, even on this first day of tracking, the left and fight cursors are well synchronised in time, as was found in Experiment 1, but less well matched in amplitude. The equivalent traces for tracking with the falter (L+RF) before practice show that the left cursor is similar in amplitude and timing to the case without the filter, but the deviations from the target waveform are different. With regard to the fight cursor, the major differences due to the filter are, not surprisingly, that the response lags further behind the target and is generally reduced in amplitude. Hence, the left and fight responses are quite clearly different. The equivalent traces after practice are also shown in Figure 9. Without the filter (L+R), it can be seen that, again as was found in Experiment 1, the left and fight cursors are very tightly synchronised and are now well matched in amplitude. With the filter in place (L+RF), the left and fight cursors are now also tightly synchronised and closely matched in amplitude. However, the fight cursor still lags the target more than the left, though the gap clearly has

Learning a Dynamic Limb Synergy 305 been narrowed as a result of practice. It is also evident that both without and with the falter, the tracking responses are superior after practice since they reproduce the target waveform more accurately (ie, with less noise). Before Practice After Practice t L+R - Target ~ Left Cursor Right Cursor , 1 sec , Figure 9. Leftand right responseto sametargetsegmentduring two-handedtracking without (L+R) and with (L+RF) the filter on the right hand, before and after practice. Signalsare 4.4 seconds in duration and were selectedfromsubjectwhoseperformancewas close to the groupmean. For ease of presentation, the quantitative results will be organised into four sections: (i) left-right coordination, (ii) left hand tracking performance, ('tii) right hand tracking performance and (iv) right hand compared with left hand performance. 4.5.1 Left-Right Coordination The overall coherence between the left and right hands, which quantifies the degree of coupling between them in two-handed tracking, is shown in Figure 10. It can be seen that, even before practice, the hands were highly coupled. The coupling increased significantly (F[ 1,5]=10.25, p<0.025) with practice for both conditions (L+R, L+RF), so that on average it accounted for 87% of the variance of the movements of the hands. The comparable figure for two-handed tracking (L+R) in Experiment 1 was 91%. Despite the fact that only the filter condition was practised, there was no significant difference in the coupling between the hands with and without the filter (F[1,5]=1.30, p=0.31). The difference before practice was not significant (F[1,5]--0.99, p=0.37) and the two conditions were virtually identical after practice.

306 N. O'Dwyer & P. Neilson The variation across frequency of 100 the coherence between the hands is -0- L+R ] shown after practice in Figure 11. The 95 9-o-. L+RF] coherence is lowest at the lowest 90 frequencies, rises to a plateau for intermediate frequencies and drops off ID O __o 85 slightly at the highest frequencies. The >~ coherence between the left and right O 80 hands can be compared in this figure with the coherence of either hand with o After Praclice the target. For the two hands in the two conditions (L+R, L+RF), the correlation 75 Before Praclice Figure 10. Overall coherence of left and right hand during two-handedtracking withoutand with the f'flter on the right hand. with the target accounted for 72-77% of the total variance of their movements and there was no significant difference between the left and right hands (F[1,5]=0.18, p=0.69) nor between tracking without and with the f'dter (F[1,5]=3.79, p=0.11). However, as in Experiment 1 (Figure 3), it can be seen that the coherence between the two hands is both greater overall than the coherence of either hand with the target (F[2,10]=23.60, p<0.001) and also varies differently with frequency (F[14,70]=10.87, p<0.001). This is especially apparent at the higher frequencies. 100 90 Oo.,..,O.-.-..-~,~ ,,o......,o.......o,, (D / 0\" . . . . . ''0 ....~ .... 0 .r . . . . . . . . . . ~. 0 f L~~D\" 8 0 f -o r t- Q 9t O --o.. T-R 9 121 9- o - . . L - R 70 91 . 3 . 5 . 7 . 9 1.1 1 . 3 1 . 5 M z .1 .3 .5 .7 .9 111 113 115 Hz L+RF 60 L+R Figure 11. Coherence versus frequencyof target and left hand (T-L), target and right hand (T-R) and left and right hand (L-R), after practice, during two-handedtracking without(L+R) and with (L+RF) the filter on the right hand.

Learning a Dynamic Limb Synergy 307 Of equal importance to the degree of coupling between the hands is the nature of their coupling, that is, their relative amplitude and timing as quantified by the gain and phase relations, respectively. Before practice the average gain of the right hand relative to the left hand during two- handed tracking without the f'flter (L+R) was about 0.8 for all frequencies, as shown in Figure 12. This means that, on average, the movements of the right hand were only 0.8 the amplitude of the movements of the left hand. With the f'flter in place (L+RF), however, the movements of the right hand increased so that they were larger in amplitude than the left hand for all but the lowest frequency. The relative amplitude increased with frequency up to 0.7Hz and thereafter declined slightly. This indicates partial compensation for the attenuation of the filter even on the first day of tracking. Actually, the f'dter has been slightly over-compensated at the lower frequencies and significantly under-compensated at the higher frequencies, as can be seen by comparison with the curve in Figure 12 representing the gain necessary for full compensation for the filter. 2.2 ........ 2.2 / 2 i 1.8 --o- Before o...-(rl, c- 1.6 Practice 1.8 -,9o\",.- \"-o..,,i 3 (.~ 1.4 1.6 \"'Q- After 1.4 \"\"o Practice 1.2 9.0... Full Compensation 1.2 .,.El 1 d I'I'P t, ~ OO 13 0.8 0.8 i b 1'3 's.z L+R L+RF Figure 12. Lefthand-right hand gain versus frequency,beforeand afterpractice,during two-handedtracking without (L+R)and with (L+RF)the filter. Alsoshownis the left-rightgain thatwouldhavebeen observedhad the filterbeen fullycompensated. After practice, further compensation f o r the filter has occurred, extending to higher frequencies, but it still falls short of the ideal beyond 1.1Hz. Furthermore, during tracking without the filter (L+R), it can be seen that there is an after-effect of practice with the f'dter (L+RF), whereby the gain of the right hand has increased relative to the left at the highest frequencies (Figure 12). This increase occurred at those frequencies where the greatest increase with practice occurred for tracking with the f'dter. The change in the gain versus

308 N. O'Dwyer & P. Neilson frequency curves with practice was significant (F[7,35]=10.40, p<0.001), as was the difference between tracking without and with the f'dter (F[7,35]=34.67, p<0.001). The phase of the fight hand relative to the left is shown in Figure 13. Before practice, in tracking without the f'dter (L+R) the two hands were virtually in phase on average for all frequencies. With the f'dter in place (L+RF) the fight hand led the left by up to 10~ at the lower frequencies but this lead gradually switched to a lag of about 10~ as the frequency increased. This phase relation fell far short of that required for full compensation for the fdter, also shown in Figure 13. 60 ..... . 60 50 ...^\"\".......o\" ......o 40 - o - - Before 30 Practice 50 Q.c.: 2O 10 9- o - - After p .......~\"\" Practice 40 9..o-.. Full ~e w~ Compensation 30 i .13,...13.- .13 20 \"El, ~ ,f 10 -10 -10 ...... ., , , , , , , .i .3 .5 .7 .9 1.1 1.3 1.5 I-Iz .1 .3 .5 .7 .9 1.1 1.3 1.5 Hz L+R L+RF Figure 13. Lefthand-righthand phaseversusfrequency,beforeand after practice, during two-handedtracking without (L+R) and with (L+RF) the f'flter. Alsoshown is the left-rightphase that wouldhave been observed had the filterbeen fully compensated. After practice, clear compensation for the falter had occurred, so that the fight hand led the left at all frequencies. However, the compensation fell short of the ideal at all but the lowest frequencies. Again, during tracking without the fdter (L+R), it can be seen that there was an after-effect of practice with the fdter (L+RF), so that the fight hand now led the left by about 15o, similar to the amount of actual compensation (10o-20~ for the fdter achieved with practice. As with the gain, the change in the phase versus frequency curves with practice was significant (F[7,35]=8.03, p<0.001), as was the difference between tracking without and with the f'dter (F[7,35]=151.40, p<0.001). In addition to these changes in left-fight coordination, it was necessary to evaluate the tracking performance of each hand and any changes with practice. For this, it was necessary to analyse the relation of the left and fight response cursors to the target.

Learning a Dynamic Limb Synergy 309 4.5.2 Left Hand Tracking Performance (L, L/L+R, L/L+RF) The mean tracking error before and 120 .-o- L ,_ 110 -'~\" L/L+R after practice for the left hand 9\"~\"\" L/L+RF conditions is shown in Figure 14. The error reduced significantly with practice t~ 8~1,Q o9 for all conditions (F[1,5]=18.33, ~ 100 p<0.01). Moreover, it is clear that performance was similar for the three 90 conditions after practice. Indeed, apart Before Practice After Practice from the improvement with practice, the most important finding for left hand was Figure 14. Rootmean square (RMS)error for the left hand during single-handedwaeking(L) and two-handedtracking without the filter (I.JL+R)and with the filter (L/L+RF) on the righthand. that, after practice, performance was apparently unaffected by what the right hand was doing. 4.5.3 Right Hand Tracking Performance (R, RF, R/L+R, R/L+RF) The mean tracking error before and after practice for the right hand conditions is shown in Figure 15. The error reduced significantly with practice for all conditions (F[1,5)=26.59, p<0.01). Although there was greater improvement for the two conditions with the filter present (F[ 1,5)=13.64, p<0.025), performance was still superior without the filter because the effect of the filter was not fully compensated. 120 1 Hand .-o-. 2 Hands 110 i1 L01.. W rr 1O0 90 Before Practice After Practice Filter (RF, R/L+RF) Before Practice After Practice No filter (R, R/L+R) Figure 15. Rootmean square(RMS)trackingerror for the right hand, beforeand after practice, during single- handed (R) and two-handed(R/L+R)tracking, withoutand with the filter on the right hand.

310 N. O'Dwyer & P. Neilson In light of the finding that left hand tracking performance was apparently unaffected by what the fight hand was doing, it was of interest here whether fight hand performance was similarly unaffected by what the left hand was doing (ie, whether the left hand was tracking or not). With no filter present (R, R/L+R) this was indeed true. This can be seen in the error scores in Figure 15 and it was also reflected in the gain and phase functions. With the filter present, however, a difference in performance between the one-handed and two-handed tasks (RF, R/L+RF) was apparent in the gain and phase functions (though not in the error scores in Figure 15). Specifically, as illustrated in Figures 16A and 16B, the gain was lower (F[7,35]=4.58, p<0.01) and the phase lag longer (F([7,35]=3.48, p<0.01) at the higher frequencies in the two-handed compared with the one-handed task when the filter was present. What this means is that there was greater compensation for the filter if only one hand was tracking rather than two, the difference being apparent at the higher frequencies only. --o-- 1 Hand I A --El-- 2 Hands 0.9 o 0.8 s 0.7 .~ .& .s .~, .b 1'1 ~ i a l : ~ z .1 .3 .; .7 .; 1\"1 1:3 115 Hz B 0.6 b 0.5 .3 .5 .7 .9 1.1 1.31.5 0.4 Filter (RF, R/L+RF) 0 -20 -40 -60 -80 /- a. -1 O 0 -120 -140 -160 .1 .; .; .7 .9 1 \"1 1 : 3 1:5 H Z Hz No Filter (R, R/L_+R) Figure 16. Target to response gain (A) and phase 03) versus frequency for the right hand, after practice, during one-handed and two-handed tracking, without and with the filter. 4.5.4 Right Hand vs Left Hand Tracking Performance (L/R, L+R, L+RF) The mean tracking error did not differ overall between the left and fight hands (F[1,5]=2.95, p=0.15) in this group of fight-handed subjects. For tracking without the filter, the initial difference in means for both one-handed and two-handed tracking (Figure 17) was

Learning a Dynamic Limb Synergy 311 not significant (F[1,5]=2.82, p---0.15) and their performance was virtually identical following practice. Whenever the filter was present on the right hand, of course, its performance was significantly poorer than on the left both before and after practice, as shown by a significant interaction (F[2,10]=53.04, p<0.001) between the hands and the three tracking conditions (L/R, L+R, L+RF). (The RF condition was excluded from this comparison since there was no corresponding condition for the left hand). 120 110 n --o-- Left Hand --~-- Right Hand 0 # # [] L.. # --o LU # .d # 0o # r:~r 1 0 0 o o o o o o El 90 i i i iii L, FI L+FI L+RF L, FI L+FI L+RF Before Practice After Practice Figure 17. Rootmean square (RMS) tracking error for the left and right hand, before and after practice, during single-handedtracking (L, R), two-handedtracking withoutthe filter (L+R) and two-handedtracking with the filter on the righthand (L+RF). 4.6 Discussion As in Experiment 1, the high coherence (87%) between the hands in the two-handed tasks and the fact that this was higher than the coherence of either hand with the target, provides strong evidence that the hands were working together in a synergy. Again, the coupling between the hands was seen to be maintained even for the highest frequencies of movement. Furthermore, this was true regardless of whether the filter was in place for the right hand. However, it is the nature of the synergy that is of most interest here. Before practice, in the task without the filter (L+R), it is clear from the gain and phase relations between the hands that the synergy was a simple, non-dynamic one. In contrast, in the task with the filter (L+RF), the synergy was a dynamic one. After practice, at low frequencies the amplitude of the movement was similar in the two hands, but as the frequency increased, the amplitude of movement of the fight hand increased to 1.6 times that of the left and remained at about this level for the highest frequencies. The fight hand also had a phase lead ahead of the left for all

312 N. O'Dwyer & P. Neilson frequencies, with a maximum lead of about 20~ at 0.5Hz. Furthermore, the after-effect of learning this dynamic synergy was apparent at the higher frequencies in the task without the filter. The subjects in this study exhibited a dynamic synergy between the hands, but the characteristics of this synergy deviated from what was required for full filter compensation. The amplitude characteristics of the fight hand movements showed good compensation up to about 1.1Hz, while the phase characteristics showed good compensation only up to 0.3Hz. Partial compensation was obtained beyond these frequencies. It is not surprising that compensation occurred in this low to high frequency pattern, since tracking in general becomes increasingly difficult with increasing frequency. Furthermore, the more complete amplitude (gain) compared to timing (phase) compensation is consistent with other studies of inter-limb coordination which show that it is easier to vary the amplitude of movement in two limbs than their timing (eg., Kelso et al., 1979; Schmidt et al., 1979; Swinnen et al., 1991). Indeed, in the present study, considerable amplitude differentiation (and thus, filter gain compensation) was evident even on the first day, whereas the liming differentiation (and thus, filter phase compensation) was small. It is possible that more complete compensation for the falter, and thus closer approximation to the expected inter-limb synergy, might be achieved with more extended practice, especially under less artificial conditions than those which necessarily obtained in the present laboratory study. Coordinated movements may require many thousands or even millions of repetitions in order to be perfected (Kottke, Halpem, Easton, Ozel and BurriU, 1978) and this level of repetition is difficult to accomplish in a laboratory. Thus, in subjects who have experienced extensive practice at a task, such as highly trained musicians and percussionists, it appears that a degree of temporal differentiation between limbs can be achieved (Shaffer, 1981). It is important to point out, however, that in the present study the amplitude and timing of the two hands were not required to be statistically independent, as is the case in producing different rhythms with each hand; rather, the hands were required to be statistically dependent, but with their amplitude and timing dynamically related. (The inter-limb coherence of 87% showed that the requirement of statistical dependence was largely achieved). This distinction between statistical independence between signals and a dynamic relation between them is an important one that may not be generally appreciated. In both cases the signal waveforms

Learning a Dynamic Limb Synergy 313 appear different on visual inspection, but in the former they are uncorrelated whereas in the latter they are correlated. An alternative view of the incomplete filter compensation is that it is due to an adaptive optimal control strategy that incorporates a compromise between tracking accuracy and the demand for muscular energy. A similar compromise has been incorporated into other optimal control models of movement (e.g., Agarwal, Logsdon, Corcos & Gottlieb, 1993; Hasan, 1986; Hogan, 1984; Meyer, Abrams, Komblum, Wright & Smith, 1988). In other words, the subject sacrifices filter compensation and thus, tracking accuracy, in order to reduce the muscular energy that would otherwise be necessary (the extent of this requirement can be appreciated from the illustration in Figure 7 of the magnitude and velocity of hand movements necessary for full filter compensation). We have elaborated this hypothesis in some detail and argued that the accuracy-energy compromise is equivalent to the well-known speed-accuracy trade- off underlying Fitts' law (Neilson, Neilson & O'Dwyer, this volume). In support of this view, in other experiments with various filters we have observed incomplete filter compensation even with single-handed tracking (Sriharan, Neilson & O'Dwyer, 1995). It should be noted also that this alternative view of the incomplete filter compensation is not incompatible with the earlier explanation in terms of insufficient practice. The optimal control strategy would be adapted for the existing level of skill in a task, so as to limit the expenditure of energy. However, as skill increases during the course of practice at a new task, the expenditure of energy would be expected to decrease (Sparrow, 1983). Therefore, the energy-accuracy trade-off could be adapted in line with this hypothesised reduced energy demand to enable greater accuracy at the task. 5. CONCLUSION In both experiments, tracking performance improved for all conditions following practice, despite the fact that only one tracking condition was practiced. This improvement can presumably be attributed to a general increase in skills that contribute to tracking performance, such as more proficient control of the joysticks and more accurate statistical prediction of the target signals (see Neilson et al., 1993). Both experiments demonstrated small but significant differences in performance between single-handed and two-handed tracking. In Experiment 1 the difference was a small increase in tracking time delay in two-handed tracking (L+R) compared with one-handed tracking (L,

314 N. O'Dwyer & P. Neilson R). Interestingly, this difference was not reproduced in the equivalent conditions in Experiment 2, perhaps due to confounding effects of the filter on the subjects' tracking phase lag. However, a difference was found in the extent of compensation for the filter in two- handed (L+RF) compared with single-handed (RF) tracking, the gain and phase functions (Figure 16) showing less high frequency filter compensation in the two-handed condition. As noted earlier, these differences likely reflect the cost in extra processing for a two-handed compared with a single-handed motor task. However, it must be emphasised that the differences observed were minor, such that they were not reflected in the tracking error scores and actually, the performance of each hand was surprisingly insensitive to activity in the other hand. Consequently, it would appear that the information processing demands of forming the inter-limb synergies, whether non-dynamic or dynamic, interfered only minimally with ongoing tracking performance. The degree of coupling between the hands in two-handed tracking was very high, even before practice. In Experiment 1, the average overall coherence between the hands was 88% before practice, rising to 91% after practice. In Experiment 2, the average coherences were 80% (L+R) and 76% (L+RF) before practice, both rising to 87% after practice. Nevertheless, since the coupling was not perfect, this meant that there was still some independent activity in each hand. Conceivably, this lack of perfect coupling was due simply to 'noise' in the motor output that was independent in each hand. Schmidt et al. (1979), for example, reported that spatial (though not timing) errors in a two-handed aiming task were largely independent. Alternatively, however, it is possible that the independence between the hands was real. In the tracking situation each hand-joystick system comprised one degree of freedom and the task required the subjects to reduce these two degrees of freedom to a single virtual degree of freedom system. It is not unlikely that in the course of the one-minute tracking tests, the hands may have intermittently reverted to their independent 'ground' state, briefly moving out of synergy; or alternatively, they may have moved briefly between the dynamic and the non- dynamic synergy. With improved skill at the task, however, such time-varying behaviour would be expected to diminish, consistent with the observed increase in coherence between the hands after practice. This suggestion of time-varying behaviour has an important bearing on the techniques employed to analyse the tracking performance. The cross-correlational and spectral analysis employed is often described as the 'quasi-linear approach'. This is because it involves the application of highly developed linear theory to study behaviour that is not strictly linear. To

Learning a DynamicLimb Synergy 315 use this approach is not to deny nonlinear behaviour (such as time variation) but to characterise the subject's linear behaviour and thereby separate it from the remainder of the subject's output. This remainder that is not accountable by the linear analysis is often known as the 'remnant'. Since it is well known that there is 'noise' in human motor output (e.g., Hatze, 1979; Kim, Carlton & Newell, 1990; Meyer, Smith & Wright, 1982; Schmidt et al., 1979), the remnant may consist entirely of noise that is unrelated (either linearly or nonlinearly) to the (target) input. However, it may also contain a component that has a nonlinear relation with the input. In the present studies, the subjects' response had a high average coherence with the target (about 80% after practice in Experiment 1 and about 75% in Experiment 2), so that a linear analysis seems entirely justified and any nonlinear component in the subjects' response was necessarily minor (it will be recalled that coherence measures the proportion of variance accounted for by the linear relation). In the case of the relation between the hands, of course, the coherences were even higher and the remnant component was almost negligible. Finally, it may be noted that the requirement of the task in these studies was for a strictly linear response - the two hands were required to be linearly coupled and each hand was required to linearly follow the target. REFERENCES Agarwal, G.C., Logsdon, J.B., Corcos, D.M., & Gottlieb, G.L. (1993). Speed-accuracy trade-off in human movements: an optimal control viewpoint. In K.M. Newell & D.M. Corcos (Eds.), Variability and Motor Control (pp. 117-155). Champaign, II: Human Kinetics Publishers. Bendat, J.S., & Piersol, A.G. (1966). Measurement and Analysis of Random Data. New York: Wiley. Hasan, Z. (1986). Optimized movement trajectories and joint stiffness in unperturbed, inertially loaded movements. Biological Cybernetics, 53, 373-382. Hatze, H. (1979). A teleological explanation of Weber's law and the motor unit size law. Bulletin of Mathematical Biology, 41,407-425. Hogan, N. (1984). An organizing principle for a class of voluntary movements. Journal of Neuroscience, 11, 2745-2754. Kelso, J.A.S. (1984). Phase transitions and critical behavior in human bimanual coordination. American Journal of Physiology, 246 (Regulatory, Integrative and Comparative Physiology 15), R1000-R1004.

316 N. O'Dwyer & P. Neilson Kelso, J.A.S., & DeGuzman, G.C. (1992). The intermittent dynamics of coordination. In G.E. Stelmach & J. Requin (Eds.), Tutorials in Motor Behavior H (pp. 549-561). Amsterdam: Elsevier. Kelso, J.A.S., Southard, D.L., & Goodman, D. (1979). On the coordination of two-handed movements. Journal of Experimental Psychology: Human Perception and Performance, 2, 229-238. Kim, S., Carlton, L.G., & Newell, K.M. (1990). Preload and isometric force variability. Journal of Motor Behavior, 22, 177-190. Klapp, S.T. (1979). Doing two things at once: the role of temporal compatibility. Memory and Cognition, 7, 375-381. Kottke, F.J., Halpem, D., Easton, J.K.M., Ozel, A.T., & Burrill, C.A. (1978) The training of coordination. Archives of Physical Medicine and Rehabilitation, 59, 567-572. Marteniuk, R.G., MacKenzie, C.L., & Baba, D.M. (1984). Bimanual movement control: information processing and interaction effects. The Quarterly Journal of Experimental Psychology, 36A, 335-365. McRuer, D.T., & Krendel, E.S. (1959). The human operator as a servo element. Journal of the Franklin Institute, 267, 381-403 and 511-536. Meyer, D.E., Abrams, R.A., Komblum, S., Wright, C.E., & Smith, J.E.K. (1988). Optimality in human motor performance: ideal control of rapid ann movements. Psychological Review, 95, 340-370. Meyer, D.E., Smith, J.E.K., & Wright, C.E. (1982). Models for the speed and accuracy of aimed movements. Psychological Review, 89, 449-482. Neilson, P.D. (1972). Speed of response or bandwidth of voluntary system controlling elbow position in intact man. Medical and Biological Engineering, 10, 450-459. Nelson, P.D., Nelson, M.D., & O'Dwyer, N.J. (1993). What limits high speed tracking performance? Human Movement Science: 12, 85-109. Partridge, L.D. (1965). Modifications of neural output signals by muscles: a frequency response study. Journal of Applied Physiology, 20, 150-156. Peters, M. (1985). Constraints in the performance of bimanual tasks and their expression in unskilled and skilled subjects. The Quarterly Journal of Experimental Psychology, 37A, 171-196.

Learning a Dynamic Limb Synergy 317 Schmidt, R.A., Zelaznik, H., Hawkins, B., Frank, J.S., & Quinn, Jr. J.T. (1979). Motor- output variability: a theory for the accuracy of rapid motor acts. Psychological Review, 86, 415-451. Shaffer, L.H. (1981). Performances of Chopin, Bach, and Bartok: Studies in motor programming. Cognitive Psychology, 13, 326-376. Sherwood, D.E. (1989). The coordination of simultaneous actions. In S.A. Wallace (Ed.), Perspectives on the Coordination of Movement ( pp. 303-327). Amsterdam: Elsevier. Sparrow, W.A. (1983). The efficiency of skilled performance. Journal of Motor Behavior, 15, 237-261. Sriharan, A., Neilson, P.D., & O'Dwyer, N.J. Adaptation of the human operator in compensatory tracking tasks with first order linear filters. Manuscript submitted for publication. Swinnen, S.P., Walter, C.B., Beirinckx, M.B., & Meugens, P.F. (1991). Dissociating the structural and metrical specifications of bimanual movement. Journal of Motor Behavior, 23, 263-279. Turvey, M.T., Schmidt, R.C., & Beek, P.J., (1993). Fluctuations in interlimb rhythmic coordination. In K.M. Newell & D.M. Corcos (Eds.), Variability and Motor Control (pp. 381-411). Champaign, II: Human Kinetics Publishers.

Motor Control and Sensory Motor Integration: Issues and Directions 321 D.J. Glencross and J.P. Piek (Editors) 9 1995 Elsevier Science B.V. All rights reserved. Chapter 12 ADAPTATION OF ARM MOVEMENTS TO ALTERED LOADS: IMPLICATIONS FOR SENSORIMOTOR TRANSFORMATIONS. G.K. Kerr School of Human Movement Studies, Queeensland University of Technology Department of Human Movement, The University of Western Australia University Laboratory of Physiology, Parks Road, Oxford, OX1 3PT, UK R.N. Marshall Department of Human Movement, The University of Western Australia In th& chapter we consider how arm movements may be adapted in response to alterations in proprioceptive information. The first part of the chapter introduces issues related to the planning and control of arm movements and the coordinate space that these processes may be represented in. The second part describes how arm movements are performed when propriocepfive information is altered by the addition of loads to the hand. In the final part we describe how methods of computer simulation may be applied to the examination of these questions. 1. COORDINATE SPACE When we move through our environment we continuously interact with objects within it. The location of these objects, whether they be located in peripersonal or extrapersonal space, is determined by the central nervous system (CNS) from many sources of sensory information. Although visual information is predominantly used to derive the location of objects, auditory and to some extent somaesthetic and kinaesthetic information may be utilised. Our movements may also be visually guided within peripersonal space. Thus they can be considered to be controlled within a coordinate system that is relative to the external world; that is in extrinsic coordinates. In contrast, in the absence of vision, the position and orientation of our limb, both statically and during movement, are predominantly derived from somaesthetic and

322 G.K. Kerr & R.N. Marshall kinaesthetic information. This information is expressed in terms of joint angles and muscle lengths (Stein, 1989). Because such a coordinate system is relative to the body it is referred to as an intrinsic coordinate system. There is, however, some controversy as to how sensory and motor information are represented within the central nervous system. For each of the above coordinate systems there are a number of possible representations. For example, an extrinsic coordinate system might use Cartesian or polar coordinates. An intrinsic coordinate system might use limb angles or muscle lengths (Lacquaniti, 1989). Knowledge of the shape, size and mass of our limbs is also necessary if we are to produce coordinated movement. To some extent this is a dynamic process as we acquire a knowledge of our \"body image\" during our development. The ability to modify this \"body image\" is also important if we are to use tools, manipulate objects, adapt to different external environments, and to rehabilitate from traumatic body injury. In this sense a coordinate system that represents the positions of our limb in space may be considered to be enmeshed with the mechanical properties of the limb (Kugler & Turvey, 1987). Ascertaining what type of representation or coordinate system is utilised by the CNS for movement control is difficult because the numerous muscular and mechanical degrees of freedom involved allow movements to be performed in a number of different ways. However, ann movements display a number of regularities in their kinematic prof'des and from these have been inferred variables essential to representation and control of movement. Descriptions of both the path and time prof'de of a movement may therefore provide insight as to how movements are planned, programmed and performed (Bernstein, 1967; Lacquaniti, 1989). 2. EXTRINSIC COORDINATE CONTROL If planning of an arm movement is performed at the level of the hand, that is in extrinsic coordinates, then essentially straight movement paths are predicted. Such a strategy, in which a coordinate system that is relative to the external world is used, is teleologicaUy appealing because it allows for greater ease of interaction between the biological system and the external environment (Atkeson & Hollerbach, 1985; Hildreth & HoUerbach, 1986). This assumption has implications in terms of the computations that must be performed. In order that the hand may be controlled along a particular path an inverse dynamics computation is required so that the correct magnitude and sequencing of joint torques may be produced. This imposes a considerable computational burden on a biological (or robotic) control system but has the advantage in that it allows the trajectory of the hand to be planned independently of the specific joint and muscle patterns (Morasso, 1983).

Adaptation of Arm Movements to Altered Loads 323 The use of an extrinsic coordinate system implies that some form of positional control may be exerted during movement. This has most convincingly been demonstrated by applying perturbations to single-joint arm movements performed by deafferented monkeys. Their movement trajectories were able to be reassumed despite large perturbations to their limb (Bizzi, Polit & Morasso, 1976; Bizzi et al., 1981, 1982; Polit & BiTzi, 1978, 1979). Results of perturbations to forearm movements performed by humans (Cooke, 1979, 1980a, 1980b) are also consistent with this hypothesis. The trajectories of two degree of freedom arm movements in the horizontal plane of both monkeys (Georgopoulos et al., 1981) and humans (Morasso, 1981) are also consistent with some form of positional control. Hand paths of these movements are straight or slightly curved and the tangential hand velocities of these movements generally show a single peaked bell-shaped curve. In contrast, joint angular velocities exhibit variable patterns which are direction dependent. Curved hand movements also tend to be controlled in short segments of shallow curvature with velocity maxima that coincide with local curvature minima. These relationships remain unaltered despite the use of different movement spaces, the tracing of curved outlines, and additional movement of the wrist. Such observations indicate that control is manifest via the trajectory of the hand and involves the precise coordination of joint torques and compensation for interactional forces (Abend, Bizzi & Morasso, 1982; Atkeson & Hollerbach, 1985; Flash, 1990). A feature of observed movement paths, for both single (Hogan, 1984; Nelson, 1983; Stein, Oguztoreli & Capaday, 1985) and multi-joint movements (Abend, Bizzi & Morasso, 1982; Atkeson & Hollerbach, 1985; Morasso, 1981) has been that, providing spatial and temporal accuracy constraints of the movement are not high, the tangential velocity profile of the hand is unimodal. Moreover, alterations in the speed of unconstrained movements are able to be accomplished by appropriate scaling of the tangential velocity profile, irrespective of the path of the hand (Atkeson & Hollerbach, 1985). This observation provides support for planning in terms of hand coordinates, but it has further implications as to additional constraints applied to the planning and control of movements. These velocity profdes have been found to adhere to a minimum-jerk cost function which is related to minimum energy expenditure. Such movement optimisation criteria appear to apply to both single (Hogan, 1984; Nelson, 1983; Stein et al., 1985) and multi-joint movements (Flash, 1983). Optimisation of movement control by the minimisation of jerk cost (Hogan, 1984; Hogan & Flash, 1987) or minimum torque-change (Uno, Kawato & Suzuki, 1989) also appears to enable a movement to be performed more smoothly and efficiently. However, other variables such as time, force, impulse or energy (Nelson, 1983), and joint stiffness (Hasan, 1986) may also provide an appropriate minimisation term, depending on the task and environmental context of the movement (Nelson, 1983; Stein et al., 1985).

324 G.K. Kerr & R.N. Marshall 2.1 Problems For Extrinsic Coordinate Control However, trajectory may not necessarily be the only controlled feature of the movement. As Atkeson and Hollerbach (1985) have noted, arm movements in the horizontal plane, from which much of the evidence for trajectory control has been obtained, are not typical of naturally occurring movements. In addition, limiting movements to one plane of movement in contact with a surface restricts them within a class of compliant motions and implies that atypical control strategies may be used in their performance (Atkeson & Hollerbach, 1985; Brady, 1982; Saltzman, 1979). Movement trajectories also differ depending on the context within which a movement is performed (Marteniuk et al., 1987). Even when unconstrained three dimensional movements are examined it is still unclear as to which coordinate system movements are planned and controlled within. Hand paths may be straight or curved depending on the workspace within which movements are performed (Atkeson & Hollerbach, 1985; HoUerbach & Atkeson, 1986, 1987a,b; Hollerbach, Moore & Atkeson, 1987). At the extreme ranges of movement, and when movements are performed in a straight line through the shoulder, both straight-line hand trajectories and constant joint- angle ratios are obtained 0_,acquaniti & Soechting, 1982; Soechting & Lacquaniti, 1981). Under these conditions it is possible to attribute control to both endpoint Cartesian interpolation (extrinsic coordinates) and joint interpolation (intrinsic coordinates) (Hollerbach & Atkeson, 1987a). Experimental attempts to distinguish between the different coordinate systems that might be employed in controlling movements have therefore proven difficult to achieve. 2.2 Adaptation To Altered Visual Extrinsic Coordinates One method that has been used to study the coordinate system in which we control our movements, at least in the visual world, has been to systematically alter or bias the available visual information. If visual information is delayed (Smith & Bowen, 1980) or prismatically altered (e.g. Held, 1965; Held & Freedman, 1963) there are considerable deviations from normal trajectories. In these circumstances we are faced with the challenge of modifying our internal representation of the externally perceived world such that it matches with the known intrinsic coordinates of our body; that is, those aspects of our movement that are not visually derived. Experiments in which the visually perceived world is altered therefore provide a means by which we can examine the coordinate systems in which our movements are planned and controlled. In the situation of prism adaptation the visual representation of an object is displaced to one side. Consequently there is a misreaching by the arm as it strives to accomplish its initial visually def'med path. Over a number of attempts at this task, however,

Adaptation of Arm Movements to Altered Loads 325 there is adaptation to this altered visual field such that the object to which the action is directed is reached with a movement path that is similar to that of movements in the normal visual field (Jakobson and Goodale, 1989). When the prisms are removed the process is reversed with the initial misreaching occurring in the opposite direction. Overall these experiments are highly supportive of movement planning and control in terms of visually defined coordinates. 3. INTRINSIC COORDINATE CONTROL Movement planning performed in terms of joint variables, that is in intrinsic coordinates, implies the existence of curved hand paths. In computational terms, the control of movement in terms of joint coordinates requires an integral dynamics calculation. This is used to produce a trajectory based on a time sequence of torque inputs to the joints. However, this method of control does not allow the trajectory of the hand to be explicitly determined between each new joint position. Rather, hand position is an a posteriori consequence of a change in joint angle. This level of planning and control has been equated with a muscle oriented mechanism of trajectory formation where a cognitive level, responsible for planning the movement, continuously and directly controls the action of the muscles (Morasso & Ivaldi, 1982). Controlling movement in terms of intrinsic coordinates may confer certain advantages to a biological system. Because such a system exerts control 'closer' to the actual implementation of a movement it is potentially more responsive to the different forces generated throughout the course of a movement. These include joint and limb inertia interactions, centripetal and centrifugal forces, all of which must be compensated for if the desired movement trajectory is to be maintained (Hollerbach & Flash, 1982). Different movement speeds require different dynamics and result in different magnitudes of interactional forces (Atkeson & Hollerbach, 1985) which, given that the motor control system is trying to maintain a particular movement path, further complicate the control of multi-joint ann movements. Such a control system is potentially more responsive to external perturbations. Thus, if obstacles are encountered during the course of a movement or, for example, when an object is lifted that is heavier or lighter than is anticipated, the nervous system is able to rapidly adapt to these localised disturbances. Such rapid alterations have been well documented in 'reflexive' patterns of EMG activity, although these mechanisms may not be sufficient to maintain the limb on its intended course (Soechting, 1988). If the perturbation is predictable, however, then over a number of repetitions the movement is able to be 'adapted' to these constraints such that the same end point trajectory is maintained.

326 G.K. Kerr & R.N. Marshall 3.1 Adaptation To Altered Intrinsic Coordinates Thus another way to examine how the nervous system plans and controls movements is to alter the kinaesthetic and somaesthetic sources of information that contribute to intrinsic coordinates. One method that we have used is to alter the inertial characteristics of the limb and thereby cause a modification of the limb dynamics required to perform a movement. Such a method should produce a mismatch in the relationship between the intended and actual movement and result in an alteration of the movement trajectory. Insight as to the controlled features of ann movements may therefore be obtained from observed changes in the kinetic and kinematic prof'des of a movement. In a recent experiment (Kerr & Marshall, 1990, submitted) we examined the effects of adding a load to the hand during a pointing movement. We were interested in how the nervous system used proprioceptive information to control these movements and whether this information was sufficient to maintain a particular movement trajectory. For this experiment we required subjects to make movements from a set starting position to a 2.5 cm. sphere situated at the subject's eye level and at a horizontal distance from the subject equal to their shoulder to wrist distance; this ensured a standard reaching distance for each subject relative to their ann length (Figure 1). A screen prevented vision of the limb except when the finger appeared in front of the target. The positioning of subjects was such that arm movements were predominandy located in a vertical plane. Subjects were allowed to practice without any load attached to their limb until they were comfortable in performing the movement. Data were then collected on eight consecutive trials. A one kilogram object F Figure 1. Experimental arrangement for load adaptation

Adaptation of Arm Movements to Altered Loads 327 with uniformly distributed mass was then attached to the subject~ hand. This was done in such a manner that the subject was prevented from ascertaining the magnitude of the mass prior to the next sequence of trials. Data were then collected on eight consecutive hand- weighted trials. Examples of movements performed in the unweighted condition are contrasted with those of weighted movements in Figure 2. This figure shows the f'mger paths for one subject in 8 consecutive trials. Of note in the unweighted condition (Figure 2A) is the consistency of the f'mger path across trials. In contrast the first weighted trial shows a large deviation from the original trajectory (Figure 2B). After this initial \"perturbation\" however, the path assumed by the finger was adapted over subsequent trials until it matched that of the finger path in the unweighted condition. 0.8 A E o 0.6 ._~ .~_ 0.4 > 0.2 o'.2 o'.6 o'.8 Horizontal Distance (rn) 0.8 E v 0.6 Q ._~ 0.4 0.2 o o.2 o.6 o.8 Horizontal Distance (m) Figure 2. Movement paths of the finger for unweighted (a) and hand-weighted (b) movements for one subject. Arrow indicates first loaded trial.

328 G.K. Kerr & R.N. Marshall Similar trends can be observed for the finger tangential velocity profiles illustrated in Figure 3. The shape of these profdes was quite different for the first one or two trials of the hand weighted condition. The profdes exhibited either an increase or decrease in peak tangential velocity and these adjustments covaried with respective decreases and increases in movement duration. 2 1.8 1.6 \"~ 1.4 v -~ 1.2 O O A~ 1 ._m 0.8 E 0.6 I- 0.4 0.2 0.2 0.4 0.6 0.8 1 1.2 Time (s) 2 1.8 1.6 ~ 1.4 ~\" 1.2 N 0.8 ~ o.6 ~ 0.4 0.2 0 0 0.2 0.4 0.6 0.8 1 1.2 Time (s) Hgure 3. Tangential velocity profiles of the fmger for unweighted(a) and hand-weighted (b) movements for one subject. Arrowindicates fwstloaded Irial. These results, in which differences due to the addition of a load occur in the first trial of a loaded movement, are similar to those of other studies (Atkeson & Hollerbach, 1985; Bock, 1990). After this initial deviation, hand paths of pointing movements with external loads remain similar to those of unloaded movements (Atkeson & HoUerbach, 1985; Bock, 1990; Lacquaniti et al. 1982). However, some slight biases in the hand paths of loaded

Adaptation of Arm Movements to Altered Loads 329 movements have been reported (Uno et al., 1989). Thus the nervous system appears to have compensated for the increased load by maintaining a similar endpoint trajectory. This consistency of endpoint trajectory independent of load offers further support for planning and control of movements in terms of extrinsic coordinates. 4. SIMULATION The stereotyped hand trajectories observed during ann movements suggest that the motor system is able to simplify control by scaling some parameters of the movement while leaving others unchanged. Hollerbach and Flash (1982) have suggested that the inertial and gravitational resulting joint moments (RJM) are separately controlled. Inertial RJMs are the forces due to acceleration and velocity of body segments involved in the movement. Gravitational RJMs are the forces due to the action of gravity on the masses of the body segments involved. For example, an increase in speed could be achieved by scaling the inertial RJMs while the gravitationalRJMs remained constant (Atkeson & Hollerbach, 1985; Hollerbach & Flash, 1982). Explanations of mechanisms involved in compensating for external loads have also distinguished between adjustments to the torques necessary to move the limb or the load. Atkeson and HoUerbach (1985) proposed that adjustments in some components of these torques were required in proportion to the increased mass in order that similar movement trajectories could be maintained. Thus increasing mass by a factor 'r' resulted in an increase in the inertial ('drive') and gravity torques for the load component by a similar proportion, that is, Tload unweighted= Tloaddrive + Tloadgravity Tloadweighted= rTload drive + rTloaagravity These modified torques were then used as drive parameters to their \"phantom ann\". Thus a simple scaling of some of the dynamic components of the torques would produce the required trajectory. In contrast, translating the trajectory of the hand into torques via an inverse dynamics relationship would be computationally more intensive. Bock (1990) also proposed that load adaptation was achieved by scaling inertial and gravitational terms. The scaling factor derived by Bock was obtained from a ratio of the apparent mass (MA) of the limb to the actual load (ML), at the instant of peak hand velocity. Apparent mass was defined as the mass associated with a unit vector force producing a specified acceleration. It should be noted that the ratio as calculated changes continuously through the movement.

330 G.K. Kerr & R.N. Marshall This was then used to scale the torques by k and k2 for the inertial and gravitational terms respectively. Thus Bock's gravitational scaling equates to Atkeson and Hollerbach's (increase the torque by the proportion (MA+ML)/MA), while his inertial term scales the torques by the square root of this ratio. Bock demonstrated a linear increase in the scaling factor with the applied load within the constraints of moving \"as quickly and accurately as possible\". However, the universality of this scaling factor is questionable, as it was found to change with different instructions: Bock states a scaling factor 3r of approximately 1.4 for the mass of 4.5 lb and the original instruction set, and a 'k' of 1.03 for a mass of 4.5 lb with the new instructions (move at the same speed as in the unloaded condition, regardless of a possible reduction in pointing accuracy). Although these studies have suggested that inertial and gravitational torques may be treated separately with respect to the overall control of the movement, neither group have directly addressed how control is implemented at the individual joints during unweighted and weighted movements. Their models have used a \"nett\" force rather than the resultant joint moments (RJM) required about individual joints. An aim of this study was to ascertain whether previously described models are implemented by the motor control system at the joint level. We therefore examined how the inertial and gravitational components of the RJMs were scaled in the unweighted and hand- weighted movements. To achieve this the motion of the upper limb from one trial was simulated using an inverse dynamics approach. The x,y marker coordinate data from one trial were used as the basis for each simulation, and modifications were made to the limb segments and hand weight masses. This technique has the advantage of allowing examination of trials with identical kinematic characteristics while modifying load variables. The inverse dynamics simulation provided RJM time histories for the wrist, elbow and shoulder joints during the motion. These RJMs consisted of the combined armgravity and armdrive RJMs. Initially data from an unweighted trial were used and then the simulation was run twice more, with additional loads of 1 and 2 kg in the hand. These runs produced the nett RJMs at the wrist, elbow and shoulder joints. Finally, two more simulations were performed with the 1 and 2 kg weights where the masses of the upper aim, forearm and hand segments were set to zero. These trials gave the 'phantom' RJMs for the two load conditions (loadgravity and loaddrive RJMs).

Adaptation of Arm Movements to Altered Loads 331 Shoulder RJM --~--- arm ------- normal vs phantom arms ---~--- phantom 2kg ph 2kg § arm 22- RJM 2kg 2O 18 16 14 A =. 12 z 10 -n3- 8 6 4 2- 0 10 20 30 4'0 50 frame Figure 4. Resultantjoint moments at the shoulderfrom the simulationfor the ann and phantom load componentsof the RJM, the sum of the arm and phantom load, and the totalRJM. Our inverse dynamics simulation results agreed with Atkeson and HoUerbach (1985), that the nett RJM at a joint equals the sum of the arm and load components (Figure 4 shows example results for the shoulder joint). This work also supports their suggestion that a linear relationship exists to scale RJMs for load increases in a movement with identical kinematic characteristics. Figure 5 shows the combined loadgravity and loaddrive shoulder RJMs for the two weight trials, and it can be clearly seen that the RJM for the 2 kg trial is double that for the I kg trial. Further, as seen in Figure 6 the scaling factor for the load only, in kinematically similar movements, also holds for the elbow and wrist joints. This result agrees with both Atkeson and Hollerbach (1985) and Bock (1990). One implication from these results is that the control of variables associated with intrinsic joint coordinates is subservient to mechanisms that derive control information from extrinsic coordinates. 5. ELBOW-SHOULDER RELATIONSHIPS However, further examination of our data show that some changes appear to be required in order that the same movement path can be achieved. Figure 7 shows the elbow-shoulder angle relationships for the unweighted and hand-weighted conditions from one subject.

332 G.K. Kerr & R.N. Marshall Shoulder RJM ....o--- phantomlkg phantom arms 1 vs 2 kg --4--- phantom 2kg 16 14 /r / 12 / A 10 E z \"n3- _j~ ~9~~176 0 , _| , I 0 1 0 20 30 40 50 frame Figure 5. Phantom arm load shoulder RJM components from the simulation for loads of 1 and 2 kg. RJMs for 1 & 2 kg loads 14 Fa= ---=--- elbow 2kg 1=, ,, ---~--- elbow lkg 12 : 99 ,=e 99 1 4 9 - wrist 2kg 10 ~ wrist lkg 8 / [] [] 9 A (' \" \" E =~ z6 0 ~-- , : , i , l , I ,J 0 10 20 30 4o 50 Frame Figure 6. Phantom arm load elbow and wrist RJM components from the simulation for loads of 1 and 2 kg.

Adaptation of Arm Movements to Altered Loads 333 This clearly shows that the elbow-shoulder angle relationship did not remain invariant under different hand load conditions. This alteration was compensated for at the wrist joint and allowed an invariant finger trajectory to be maintained. 1.8 , , , ,, , 1.6 A ro~ 1.4 ~ 1.2 0 [] 0.8 0.6 0.4 |I III I 1.4 0 0.2 0.4 0.6 0.8 1 1.2 1.8 Shoulder Angle (radians) 1.6 S A to-'-~ 1.4 ~ 1.2 B~ ~ 0 i'~ 0.8 0.6 0.4 ,,, I I IIi I 1.4 0 0.2 0.4 0.6 0.8 1 1.2 Shoulder Angle (radians) Figure7. Relationshipbetweenelbowand shoulderangles for unweighted(a) and hand-weighted (b) movementsfor one subject. These alterations in the elbow-shoulder relationship between the unweighted and weighted condition must be associated with altered shoulder and elbow RJMs. Note that the above simulation and the associated 'scaling' models of control all required identical kinematic characteristics to be maintained if such an approach was to be successful. The fact

334 G. K. Kerr & R. N. Marshall that there was an alteration in joint angular relationships raises concerns about the applicability of a theory based upon end point kinematics, when the kinematic redundancy in the system permits a variety of joint RJM combinations to produce the same kinematics. One implication from this result is that scaling of joint RJMs in real-life movements is non-linear. Because the gravitational component of the RJMs are subject to simple linear scaling, we suggest that the scaling for inertial components must therefore be non-linear. This non-linear scaling for inertial components of the RJM assumes even more importance when movements are performed at different speeds. It is worth noting that previous models such as Atkeson and Hollerbach's (1985) necessitate an exponential scaling for inertial effects, even if the same movement trajectories are maintained. These results suggest that control based on an intrinsic coordinate system is crucial for maintaining coordinated movement. Thus changes in joint kinetics and kinematics are able to effected in order that the same movement path can be achieved. 6. INTEGRATING EXTRINSIC AND INTRINSIC COORDINATES It is apparent that the planning and control of movement in either a purely extrinsic or intrinsic coordinate system can not fully account for all of the previous observations. However, because the spatial and temporal movement properties are able to be partially explained in terms of both coordinate systems, it is plausible that some combination of coordinate systems may be effectively used. An overall knowledge of the trajectory of the hand must be available if the appropriate spatiotemporal sequence of proximal joint rotations is to be implemented. In addition, for the final goal to be achieved, an adequate correction for errors in the initial movement is required. For unconstrained arm movements these corrections appear to be related to movements of the distal joints of the limb. Thus control of distal joints is not exerted independently of proximal joints (Lacquaniti, 1989; Soechting & Flanders, 1989a,b). These perspectives indicate that there is an intermeshing of both extrinsic and intrinsic coordinate systems within some mutually accessible spatial representation. In this manner the different sources of sensory information can be thought to have been transformed and represented within a common co-ordinate system (Knudsen, DuLac & Esterly, 1987; Stein, 1989). In fact there is physiological and psychological evidence that the posterior parietal cortex is an area where such transformations occur (Stein, 1989). These notions also speak to the idea of multiple levels of sensorimotor representation (Saltzman, 1979) that may interact in a heterarchical manner (Turvey, Shaw & Mace, 1978). Thus although each sensory modality is represented within its own spatial map these may overlap and become enmeshed with each other.

Adaptation of Arm Movements to Altered Loads 335 Attempting to integrate our results into a unifying model of control is quite a challenge. At present there is no one model that can fully integrate or reconcile positional control with joint torque control. The Fel'dman (lambda) model of equilibrium point control (Ferdman, 1966a,b; 1986) and developments of this model by Latash and Gottlieb (1991a,b), all support a positional control mechanism in terms of the interaction of centrally derived commands with the current state of the peripheral apparatus as signalled by afferent feedback. These approaches do, to some extent, allow for compensation of altered inertial forces during movement. In fact, Fel'dman (1966b) has posited that there are two independent systems of control; one for equilibrium positions of the joint and the other for dynamic parameters which determine a movements acceleration and form. However, a problem for these models is that activity level in participating muscles is not positionaUy unique and is highly dependent on moment arms of the musculo-skeletal system and on the external forces applied to it (Hasan & Enoka, 1985; van Ingen Schenau et al., 1992). The kinematic redundancy observed in multi-joint movements, as described in our experiments, further complicates matters for proposed models because the centrally defined trajectory of the finger must be independent of the muscles and joints involved in the task. In addition, recent work by Shadmehr and Mussa-Ivaldi (1993) indicates that the human nervous system has a strong joint torque dependent mechanism of control and that scaling and adaptation to novel forces occurs at the joint level rather than at the level of the hand. Thus proposed models must integrate mechanisms of planning and control in terms of extrinsic coordinates of the hand and the intrinsic coordinates of the joints and muscles. One of the goals in our research is to quantify the magnitude of the initial alteration in movement trajectory and the time course of the adaptation. In this way we hope to understand more fully how real-life movements are planned and controlled. We also hope to shed some light on the central representations employed by the central nervous system in these processes. Such knowledge is important in understanding how the motor apparatus, which is constrained within a body-centredcoordinate system, is able to produce a movement towards an object localised in a coordinate system with respect to the external world. ACKNOWLEDGEMENTS This research was supported by grants from the Wellcome Trust (U.K.), British Council (Australia), McDonnel-Pew Center for Cognitive Neuroscience (Oxford, U.K.) and the Department of Human Movement, The University of Western Australia. This work was conducted while G. Kerr was a Medical Research Council (U.K.) postdoctoral researcher. We would like to thank John Stein (University Laboratory of Physiology, Oxford, U.K.) for his support and encouragement.

336 G.K. Kerr & R.N. Marshall REFERENCES Abend, W., Bizzi, E., & Morasso, P. (1982). Human arm trajectory formation. Brain, 105, 331-348. Atkeson, C.G., & Hollerbach, J.M. (1985). Kinematic features of unrestrained vertical ann movements. The Journal of Neuroscience, 59, 2318-2330. Bemstein, N. (1967). The co-ordination and regulation of movements. Oxford: Pergamon Press. Biz~.i, E., Accomero, N., Chapple, W. & Hogan, N. (1981). Processes underlying ann trajectory formation. In O. Pompeiano & C.A. Marsan (Eds.), Brain mechanisms and perceptual awareness (pp. 311-318). New York: Raven Press. Bizzi, E., Chapple, W., & Hogan, N. (1982). Mechanical properties of muscles. Implications for motor control. Trends in Neurosciences, 395-398. Bizzi, E., Polit, A., & Morasso, P. (1976). Mechanisms underlying achievement of final head position. Journal of Neurophysiology, 39(2), 435-AAA.. Bock, O. (1990). Load compensation in human goal-directed ann movements. Behavioural Brain Research, 41,167-177. Brady, J.M. (1982). Trajectory planning. In J.M. Brady, J.M. Hollerbach, T.L. Johnson, T. Lozano-Perez and M.T. Mason (Ecls.), Robot motion: planning and control. Cambridge: MIT Press. Cooke, J.D. (1979). Dependence of human arm movements on limb mechanical properties. Brain Research, 181(2), 493-497. Cooke, J.D. (1980a). The organization of simple skilled movements. In G.E. Stelmach & J. Requin (Eds.), Tutorials in motor behavior (pp. 199-212). Amsterdam: Elsevier-North Holland. Cooke, J.D. (1980b). The role of stretch reflexes during active movements. Brain Research, 181(2), 493-497. Fel'dman, A.G. (1966a). Functional tuning of the nervous system with control of movement or maintenance of a steady posture-II. Controllable parameters of the muscles. Biophysics, 11(3), 498-508. Fel'dman, A.G. (1966b). Functional tuning of the nervous system with control of movement or maintenance of a steady posture-III. Mechanographic analysis of the execution by man of the simplest motor task. Biophysics, 11(4), 667-675. Fel'dman, A.G. (1986). Once more on the equilibrium-point hypothesis (lambda model) for motor control. Journal of Motor Behavior, 18(1), 17-54. Flash, T. (1990). The organisation of human ann trajectory control. In J.M. Winters & S.L- Woo (Eds.), Multiple muscle systems: Biomechanics and movement organisation. New York: Springer-Verlag.

Adaptation of Arm Movementsto AlteredLoads 337 Flash, T. (1983). Organising principles underlying the formation of hand trajectories. Ph.D. dissertation, Harvard/MIT Division of Health Sciences and Technology, Cambridge, MA. Georgopoulos, A.P., Kalaska, J.F., & Massey, J.T. (1981). Spatial trajectories and reaction times of aimed movements: Effects of practice, uncertainty, and change in target location. Journal of Neurophysiology, 48(4), 725-743. Gilbert, F.C., & Thach, W.T. (1977). Purkinje cell activity during motor learning. Brain Research, 128, 309-328. Hasan, Z., & Enoka, R. (1985). Isometric torque-angle relationship and movement-related activity of human elbow flexors: implications for the equilibrium point hypothesis. Experimental Brain Research, 59, 441-450. Held, R. (1965). Plasticity in sensorimotor systems. Scientific American, 213, 84-94. Held, R., & Freedman, S. (1963). Plasticity in human sensorimotor control. Science, 142, 455-462. Hildreth, E.C., & Hollerbach, J.M. (1986). The computational approach to vision and motor control. In F. Plum (Ed.), Handbook of Physiology: The Nervous System V (pp. 605- 642). American Physiological Society. Hogan, N. (1984). An organizing principle for a class of voluntary movements. The Journal of Neuroscience, 4(11), 2745-2754. Hogan, N., & Flash, T. (1987). Moving gracefully: quantitative theories of motor coordination. Trends in Neurosciences, 10, 170-174. Hollerbach, J.M., & Atkeson, C.G. (1986). Characterization of joint-interpolated arm movements. In H. Heuer & C. Fromm (Eds.), Generation and modulation of action patterns (pp. 41-54). Berlin: Springer-Verlag. Hollerbach, J.M., & Atkeson, C.G. (1987a). Deducing planning variables from experimental ann trajectories: pitfalls and possibilities. Biological Cybernetics, 56, 279-292. Hollerbach, J.M., & Atkeson, C.G. (1987b). Inferring limb coordination strategies from trajectory kinematics. Journal of Neuroscience Methods, 21, 181-194. Hollerbach, J.M., & Flash,T. (1982). Dynamic interactions between limb segments during planar ann movement. Biological Cybernetics, 44, 67-77. Hollerbach, J.M., Moore, S.P., & Atkeson, C.G. (1987). Workspace effect in arm movement kinematics derived by joint interpolation. In G.N. Gantchev, B. Dimitrov & P. Gatev (Eds.), Motor control (pp. 197-208). Plenum Publishing Corporation. Jakobson, L.S. & Goodale, M.A. (1989). Trajectories of reaches to prismatically-displaced targets: evidence for \"automatic\" visuomotor recalibration. Experimental Brain Research, 78, 575-587. Kerr, G.K., & Marshall, R.N. (1990). Adaptability of multi-joint arm movements to alterations in segmental inertia. Societyfor Neuroscience Abstracts, 16, 891.

338 G.K. Kerr & R.N. Marshall Kerr, G.K., & Marshall, R.N. Compensating for external loads in a pointing movement. Manuscript submitted for publication. Knudsen, E.I., DuLac, S., & Estedy, S. (1987). Computational maps in the brain. Annual Review of Neuroscience, 10, 41-65. Kugler, P.N., & Turvey, M.T. (1987). Information, natural law, and the self-assembly of rhythmic movement. Hillsdale, New Jersey: Lawrence Erlbaum. Lacquaniti, F. (1989). Central representations of human limb movement as revealed by studies of drawing and handwriting. Trends in Neurosciences, 12, 287-291. Lacquaniti, F., & Soechting, J.F. (1982). Coordination of arm and wrist motion during a reaching task. Journal of Neuroscience, 2, 399-408. Lacquaniti, F., Soechting, J.F. & Terzuolo, C.A. (1982). Some factors pertinent to the organisation and control of arm movements. Brain Research, 252, 394-397. Latash, M.L., & Gottlieb, G.L. (1991a). An equilibrium-point model for fast, single-joint movement: I. Emergence of strategy-dependent EMG patterns. Journal of Motor Behavior, 23(3), 163-178. Latash, M.L., & Gottlieb, G.L. (1991b). An equilibrium-point model for fast, single-joint movement: II. Similarity of single-joint isometric and isotonic descending commands. Journal of Motor Behavior, 23(3), 179-191. Marteniuk, R.G., MacKenzie, C.L., Jeannerod, M. Athenes, S., & Dugas, C. (1987). Constraints on human ann movement trajectories. CanadianJournal of Psychology, 41, 365-378. Morasso, P. (1983). Coordination aspects of arm trajectory formation. Human Movement Science, 2, 197-210. Morasso, P. (1984). Spatial control of arm movements. Experimental Brain Research, 42, 223-227. Morasso, P., & Ivaldi, F.A.M. (1982). Trajectory formation and handwriting: A computational model. Biological Cybernetics, 45, 131-142. Nelson, W.L. (1983). Physical principles for economies of skilled movements. Biological Cybernetics, 46, 135-147. Polit, A., & BiTJi, E. (1978). Processes controlling arm movements in monkeys. Science, 201, 1235-1237. Polit, A., & BiTJ.i, E. (1979). Characteristics of motor programs underlying ann movements in monkeys. Journal of Neurophysiology, 42(1), 183-194. Saltzman, E. (1979). Levels of sensorimotor representation. Journal of Mathematical Psychology, 20, 91-163. Shadmehr, R., & Mussa-Ivaldi, F.A. (1993). Geometric structure of the adaptive controller of the human arm. Massachusetts Institute of Technology Artificial Intelligence Laboratory Memo No. 1437.

Adaptation of Arm Movements to AlteredLoads 339 Smith, W.M., & Bowen, K.F. (1980). The effects of delayed and displaced visual feedback on motor control. Journal of Motor Behavior, 12, 91-101. Soechting, J.F. (1988). Effect of load perturbations on EMG activity and trajectories of 'pointing' movements. Brain Research, 451,390-396. Soechting, J.F., & Lacquaniti, F. (1981). Invariant characteristics of a pointing movement in man. Journal of Neuroscience, 1, 710-720. Soechting, J.F., & Flanders, M. (1989a). Sensorimotor representations for pointing to targets in three dimensional space. Journal of Neurophysiology, 62(2), 582-594. Soechting, J.F., & Flanders, M. (1989b). Errors in pointing are due to approximations in sensorimotor transformations. Journal of Neurophysiology, 62(2), 595-608. Stein, J.F. (1989). Representation of egocentric space in the posterior parietal cortex. Quarterly Journal of Experimental Physiology, 74, 583-606. Stein, R.B., Oguztoreli, M.N., & Capaday, C. (1985). What is optimized in muscular movements? In N.L. Jones, N. McCartney & A.J. McComas (Eds.), Human muscle power. Champaign: Human Kinetics. Turvey, M.T., Shaw, R.E., & Mace,W. (1978). Issues in the theory of action: Degrees of freedom, coordinative structures and coalitions. In J. Requin (Ed.), Attention and Performance VII (pp. 557-595). Hillsdale, N.J.: North-Holland. Uno, Y., Kawato, M., & Suzuki, R. (1989). Formation and control of optimal trajectory in human multi-joint arm movement. Biological Cybernetics, 61, 89-101. van Ingen Schenau, G.J., Beek, P.J., & Bootsma, R.J. (1992). Is position information alone sufficient for the control of external force? Behavioral and Brain Sciences, 15(4), 804- 805.

Motor Control and Sensory Motor Integration: Issues and Directions 341 D.J. Glencross and J.P. Piek (Editors) 9 1995 Elsevier Science B.V. All rights reserved. Chapter 13 THE USE OF VIRTUAL ENVIRONMENTS IN PERCEPTION ACTION RESEARCH: GRASPING THE IMPOSSIBLE AND CONTROLLING THE IMPROBABLE John P. Wann 1 Department of Psychology, University of Edinburgh, 7 George Sq., Edinburgh, Scotland Department of Human Movement Studies, University of Queensland, St. Lucia, Australia Simon K. Rushton Department of Psychology, University of Edinburgh, 7 George Sq., Edinburgh, Scotland Two of the most fundamental animal behaviours are controlling locomotion to arrive at an intended destination, and capturing objects upon arrival Research into the control of naturalistic behaviours such as these has been hampered by the difficulty in manipulating environmental information, without impinging upon the naturalness of the task. We outline the potential of computationally specified 3-D optic arrays (virtual environments) to allow control over the visual information provided to the performer. We then provide examples of re-examining the control of locomotion and interceptive timing using virtual environment displays. 1. I N T R O D U C T I O N A major constraint upon research into perception and action has been the ability to precisely control the environment in which movement takes place. Research that observes an actor's response to a natural environment can only establish that the resultant movement is consistent with a particular model of control. In a natural setting the actor often has access to multiple sources of information, and this redundancy clouds the issue of whether control is effected through the use of a primary information source, a limited sub-set or integration of multiple sources of information (e.g. Bruno & Cutting, 1988). To test a putative neural model or control axiom it is invariably necessary to perturb the environment in which the action unfolds. Typical examples have been the use of prisms 1 The research reported in this chapter was supportedby JCI/MRC GrantG911369: Principlesfor PerceptionandActionin VirtualReality(JPW) and JCI/MRC Grant G9212693: VisualControlof Steering(D.N. Lee & JPW).

342 J.P. Wann& S.K. Rushton (Held & Freedman, 1963) or occlusion (Held & Gottlieb, 1958; Prablanc, Echallier Komilis & Jeannerod, 1979) to control the visual information available to the performer. Other manipulations have altered the movement dynamics during motion (Denier van der Gon & Thuring, 1965) or changed the size of objects that are the goal of a planned interaction (Jeannerod, 1981). Some of these physical manipulations have been remarkably ingenious, but all are ultimately limited in the environmental control that is possible. Hence prismatic shifts cannot be continuously varied within a test session, so adaptation invariably takes place. Similarly, the manipulation of apparent object size (Jeannerod, 1981) can only be made within a limited set of discrete categories, or with limited precision (see later analyses). The essence of the problem is that we have very little control over the environmental niche in which we act out the majority of our behaviour. Environments where the experimenter can exert control, such as keyboard tasks or constrained laboratory tasks place restrictions upon the generalizability of the results to natural behaviour. The escalation in the processing power of digital computers, particularly in the domain of computer graphics has now provided the facility for the precise manipulation of an observer's visual environment, Virtual reality or virtual environment systems (VE) can create the illusion of a 3-D environment for a wide range of visualisation tasks including perceptual-motor experimentation. 1.1 Recreation of the optic array A V E system in its simplest form recreates a 2-D pictorial image, with appropriate linear perspective, texture gradients and motion parallax in response to the motion of the observer. We will call this a single-channel VE system. If the image from such a system is presented to a large portion of the observer's field of view, and natural environmental information (e.g. the screen surround) is diminished, then a suitable VE illusion can be produced. This is particularly the case when the pertinent information is in far space (e.g. more than 5-6m from the observer). An important distinction we would draw between pure simulation and a VE system is the degree to which the observer is able to interact with the display in real-time. We agree with the def'mition provided by Ellis et al (1992) at the NSF workshop: \"By virtual environments, we mean real-time interactive graphics with three-dimensional models, when combined with display technology that gives the user immersion in the model worm and direct manipulation.\"

Virtual Environments in Perception-Action Research 343 A good example of an effective single-channel system is the bobsled simulator used for pre-season training by the US Olympic team (Hubbard, 1994). This VE system used a graphics workstation to calculate the perspective from a single, movable viewpoint of a computational def'med bobsled track, and presented this on a large screen CRT monitor. This monitor was mounted within a swivelling sled, where the degree of sled tilt was servo-controlled by the workstation. The sled driver then controlled the virtual motion with conventional bobsled controls and hence was presented with an interactive illusion of steering down the computer generated course. Because a bobsled travels at considerable speed, the driver is looking several metres ahead to glean steering information. This is ideal for single channel simulation where proximal, stereoscopic information may have little if any role. A single channel system can also present the illusion of objects moving into more proximal zones through the use of optic looming. It is not unusual for observers in a cinema or simulator to flinch if an object rapidly looms toward them on a large screen. In this case, however, expansion information is placed in conflict with information from binocular disparity, which specifies a single depth image plane. The empirical evidence seems to suggest that when looming is presented on a flat screen display, temporal judgements of arrival tend to be imprecise and also prone to bias by known size effects (e.g. DeLucia & Warren, 1994; Kaiser & Mowafy 1993; McLeod & Ross, 1983; Schiff & Detwiler, 1979; Schiff & Oldak, 1990). An alternative is to use a dual-channel system to present the observer with two overlapped images that contain appropriate disparity for near and distant visual features. A common means of presenting such images is a head-mounted display (HMD: Figure 1) although an identical stereoscopic display can be presented on a back-projection screen placed in front of the observer. An extension of the back-projection approach has been the CAVE (Cruz-Neira, Sandin & Defanti, 1993) where stereoscopic images are presented to a frontal screen, two side screens a ceiling and floor. This allows the observer to move through a full- volume stereoscopic environment and also interact with full view of their own limbs. The primary shortcoming of the CAVE is the massive computational power required to generate the synchronous 6-10 channels of real-time graphics that it requires. It is possible to produce effective single channel presentations using appropriate software on common hardware platforms such as those based upon Inte1486 processors. As such, the technology is accessible to many research groups. For most systems a compromise has to

344 J.P. Warm & S.K. Rushton be arrived at between level of detail and update rate; the larger the geometric database the longer the system will require to update the geometrical perspective from a specified viewpoint and render the surfaces for viewing. A common benchmark quoted for systems is the number of polygons/sec that can be achieved. In practice, however, high polygon rates may be less important than the ability of the system to support texture mapping. !! Inter Pupillary Distance I, , Virtual Image Plane Accommodative ar Proximal Vergence LCD Screens Magnifying Optics Nodal Point Center of Rotation O~ Figure 1. Designof a typical stereoscopichead-mounted display. Imagespresented on LCD screens are viewed through magnifying optics, which project two half images at a visual angle (a) and a fixed focal depth (virtual image plane). Fusion of these images will result in the percept of an object in front of (crossed disparity), or behind the virtual image plane (uncrosseddisparity). Bringing the half images onto the fovea will also require a degree of vergence that may be in conflict with that promoted by proximal and accommodativecues (b). The distancebetween the projected half-imagesneeds to be calculated on the basis of the viewers inter-pupillarydistance to provide specific depth intervals. A small number of polygons overlaid with photo-generated textures can produce a strong percept of surfaces in depth and texture expansion and flow support an illusion of relative motion. Where the requirement is for a dual channel (e.g. stereoscopic) system, or there are additional computational overheads due to peripheral input devices, then a more advanced hardware platform may be required that includes graphics processors for geometry and rendering and/or links a series of workstations through an operating system that allows them to act as parallel processors. The technology available for the generation of interactive computer graphics is undergoing rapid development. It is impossible to

Virtual Environments in Perception-Action Research 345 recommend specific solutions. Researchers wishing to recreate optic arrays for experimental settings need to consider which aspects of the natural environment appear to be most salient to the performer. The following observations are made on the current state of VE technology in 1994: i) Depth cues such as linear perspective, occlusion and motion parallax can be supported through relatively simple geometry. More subtle cues such as lighting, shadow and haze can have disproportionate overheads. ii) Texture gradients and photo-graphic overlays can greatly enhance realism at a relatively low computational cost. iii) Head-motion parallax can be supported but the technical problems are in the real time sensing and data transmission of head movements, rather than in image generation. iv) Stereoscopic systems are readily available, but there is a computational cost in rendering two viewpoints from the same database. v) The display system will introduce limitations on the information, that can be provided. The current resolution of HMD and projection systems limit the level of detail that can be presented, and it is not possible to present a stimulus for ocular accommodation that matches the binocular cues for vergence/stereopsis (see section 3.1). 2. PERCEPTUAL EXPERIMENTATION WITH VIRTUAL ENVIRONMENTS 2.1 The perception of interceptive timing: grasping the impossible What information is necessary to accurately catch a ball in flight? Studies that have addressed this question provide prime examples of attempts to physically manipulate the vision information available to the performer: Temporal Occlusion: Whiting (1968) and Sharp & Whiting (1974) introduced controlled periods of occlusion during the flight of projected balls. Temporal occlusion was used by several research groups in subsequent studies with a range of methodological refinements. A technological advancement has been the use of liquid crystal goggles to provide precise stroboscopic periods of occlusion (Elliott, Zuberec & Milgram, 1994). Spatial Scaling: An apparent spatial manipulation was introduced by Judge and Bradford (1988) who presented subjects with the task of catching tennis balls while wearing telestereoscopic glasses. A telestereoscope displaces the line of sight for each eye, so that the effective inter-ocular separation is increased. As a result the binocular disparity

346 J. P. W a n n & $. K. Rushton presented by an object in depth varies in proportion to the increased ocular separation and appears to be closer than its true distance. In this context Judge and Bradford observed that subjects grasped for the ball well in advance of its arrival, testifying to the salience of binocular cues, which were in conflict with veridical information from optical looming (which is not affected by inter-ocular separation). The manipulation of optical looming was attempted by Savelsbergh, Whiting and Bootsma (1991; see also Savelsbergh et al, 1993), through the use of a deflating balloon to change the diameter of the optical contour of a ball during flight. The results were interpreted as supporting the primacy of optical looming over other arrival cues such as binocular disparity. We argue that the manipulations cited above can be presented with greater precision through computational scaling of virtual objects. For temporal occlusion there is an obvious advantage, despite the limitations of frame rate, in being able to remove sight of the ball for a precise period or at a specific depth, without removing vision of the ambient environment. The spatial scaling introduced by using a telestereoscope provides an interesting observation on catching skill, but because the whole of the visual array is rescaled when viewed through a telestereoscope, it is impossible to determine if the subjects responded to the (absolute) stereoscopic depth of the ball, or its relative depth in a stereoscopically rescaled environment. It is also the case that binocular rescaling of the full visual array promotes rapid adaptation and within 20 catches subjects had modulated their catching action to allow for the new viewing condition (Judge & Bradford, 1988). Attempts to manipulate optical looming provide a good illustration of the problems of trying to control the information in the visual array through physical devices. Although Savelsbergh et al (1991, 1993) changed the size of the optical contour during the trajectory of the ball, they did not elaborate on how this change affected the time-to- contact ('Iq'C) information present in the inverse of the relative rate of dilation (tau: Lee, 1976). For a ball travelling at a constant velocity, a precise offset in TIC, as specified by tau, can be introduced through scaling the ball diameter by: - Z(t) Rnew(t) = R [Z(t)_C] (1)

Virtual Environments in Perception-Action Research 347 where R is the original diameter of the ball; Z(t) is its instantaneous distance from the observer; C is the distance of the lag/lead required (equivalent to the required time-lag * ball-velocity ) and is positive for a shift towards and negative for a shift away from the observer; Rnew(t) is the resulting, time dependent ball diameter. The problem that occurs with a physical manipulation of diameter such as a balloon, is illustrated in Figure 2, where measurement of the rate of deflation demonstrates that the contour diameter changes in a quasi-linear fashion. A steady rate of deflation will often over-scale Rnew(t) during the early stages of the trajectory, where Z(t) is large and C is small by comparison. It will then under-scale the diameter during the latter stages, where the converse is true (Equation 1). 11,0 I~ 1 0 . 0 9.0 q ~ ..... - 8.o ~-~~_% ..,.,, 2: 0 7.0 0 6.0 I Horizontal Deflation = 9.98 - 0.72t - 0.22t 2 ', Vertical Deflation = 10.4 - 1.03t - 0.19t 2 \" J 5.0 . . . . I . . . . i . . . . i . . . . , . . . . i 0.0 0.50 1,0 1.5 2.0 2.5 TIME s Figure 2. Deflation of a balloon inflated to 10cm diameter, with deflation controlled so that the diameter reduced at an approximate rate of 1.25cm/s (c.f. Savelsbergh et al, 1991). Equations specify the least squares polynomial approximations plotted as continuous lines. The broken line indicates the rate of deflation that would be required to produce a stable 0.1s delay in the time specified by tau, for a ball travelling at 2m/s from a distance of 5m, and arriving at the observer at 2.5s. If the intention is to present a constant delay in time-to-contact (TI'C) specified through looming, then physical deflation will often reduce the diameter too rapidly during the early stages of flight, resulting in a much longer T r c than intended. Then in the final approach phase (e.g. 2.2 - 2.5s) it may chronically under-scale optical size resulting in rapid looming, equivalent to a sudden decrease in apparent \"FrC. The perceptual effect is that looming information will specify a T r c estimate that initially lags well behind the actual T r c and then in the later stages converges rapidly on the actual TrC, somewhat like a ball accelerating at the observer (Figure 2). A re-analysis of data presented in Savelsbergh et al (1991, 1993) suggests that these effects were present

348 J.P. Wann & S.K. Rushton within these studies hence it is unclear what information subjects were using, because perturbations of the visual array were not controlled with sufficient precision. With a stereoscopic (dual channel) VE the binocular disparity of every surface in the visual array is computed to provide the illusion of objects lying in depth. Within such an array the apparent depth, size and visibility of virtual objects to be changed at will independent of other features of the ambient environment. Hence a VE enables optical looming to be computationally scaled in line with Equation 1, to provide precise control over the information presented through changing optical size (tau) and changing disparity (depth). To explore the respective roles of such information in judging time-to-contact, we used a head mounted display (Figure 1) to present subjects with the task of catching virtual tennis balls that appeared to travel towards them at a speed of 2m/s from a distance of 4m. A purpose built strain-gauge instrumented glove was used to monitor the subjects grasping action in response to the illusory projectiles. Figure 3 presents a typical response, illustrating that subjects were able to time their catching action appropriately and catch an illusory ball. Figure 3. Two typical grasp responses in catching a virtual ball. Both the index finger and thumb were monitored to measure the grasp response (presented in electronic units). Because the distance and speed of the ball was varying by 10% across trials the flight time was different for each trial. The split rectangle indicates the mean arrival time for the virtual ball in the control trials, bracketed by 1 standard deviation either side. The accuracy of the grasp responses was appraised using the precise time of arrival for each trial.

Virtual Environments in Perception-Action Research 349 The participants first practised catching virtual balls without any temporal/spatial scaling. They were then presented with 20 ball flights where optical size and binocular disparity were in concord, interleaved with 10 trials where optical size was scaled using Equation 1, so that it specified a TI'C lOOms earlier, and 10 trials where it specified a T/'C lOOms later than the actual, disparity cued TIC. Hence, the information specified by optical looming (optical tau), was differentiated from the information contained in changing disparity. The grasping responses for the concordant interleaved trials were used as the standard against which to compared the non-concordant trials, and this demonstrated a different effect for perturbations toward as opposed to away from the observer. If optical looming specified that the virtual ball would arrive earlier than was suggested by binocular depth cues, then subjects did indeed respond predominantly to looming (tau) rather than disparity and grasp at the ball earlier (mean = 82ms; s.c. = 7ms). In the converse, however, where binocular depth provided the most immediate information and tau specified a later arrival, then subjects did not delay their grasp by an equivalent period and biased their response more toward arrival specified by binocular disparity (Wann& Rushton, in press). Perceptual selectivity of this type is what one might predict from any animal that has developed robust strategies for collision control, but these findings also contrast with the some of dogmatic statements made by researchers subscribing to a neo-Gibsonian theory of visual perception. The findings qualify the results of Savelsbergh et al (1991, 1993), by providing some explanation of the relatively minor shifts that they observed in grasp timing for a deflating ball. This study also serves to illustrate the theme of this chapter, namely the power and utility of VE technology for the exploration of human visual perception and perception-action coupling. There are further manipulations of the virtual optic array that might be presented to observers during a ball catching task to address the following questions: i) If there is selective weighting of changing disparity and looming information, does this hold over a wide speed range, or are particular stimuli relied upon for different speeds of object/observer travel (e.g. Regan & Beverley, 1979) ? Control: Non-congruence of disparity vs expansion over a range of object speeds. ii) Are observers sensitive to the relative rate of expansion rather just the expansion velocity (Regan & Hamstra, 1993)? How does object size affect the judgement of time to arrival when depth is also binocularly specified?

350 J.P. Warm & S.K. Rushton Control: Between or within trial manipulation of object size, balanced for expansion velocity or relative rate of expansion. iii) How do other relative depth/size cues affect the judgement of time to arrival? Control: Between or within trial manipulation of size or detail of background environment. iv) Are there critical or different time windows for the perception of changing size and changing disparity, or are both continually monitored? Control: Disappearance of the ball in flight for specified periods without total occlusion of the visual environment. v) How do observers judge the time to arrival of non-spherical objects which rotate, thereby distorting their optical contour (e.g. Tresilian, 1991) ? Control: Object shape and rotation around major/minor axes. 2.2 The perception of heading: controlling the improbable The information necessary to discern the direction of heading during egomotion has posed an interesting question as to the sufficiency of the optic and retinal flow fields. Linear translation across a ground plane produces a radial expansion of elements within the visual field that in principle provides information about the observers speed and direction of heading. The specificity of visual information in discerning heading has been addressed by presenting passive observers with displays that simulate the optical transformations that would occur during natural locomotion. Previous studies have considered a wide range of information sources and how they might be used to perceive heading and/or steer towards a target location. A simple steering strategy is to try and maintain the target location in the centre of the cyclopean visual field which does not, in principle, require the perception of absolute heading (LleweUyn, 1971). Perception of heading can be gleaned from the differential motion parallax of objects at different distances from the observer (Cutting, 1986) or extracted from the global velocity flow field (Warren, Morris & Kalish, 1988). The latter proposal has stimulated considerable debate on the issue of whether observers can determine the focus of expansion or focus of radial outflow (FRO) from a first-order velocity flow field. Warren et al (1988) used a computer screen to present radial expansion of an array of surface dots to observers. The display simulated the transformations that would arise from the observer locomoting at

Virtual Environments in Perception-Action Research 351 speeds between 1 - 3.8m/s in a direction 0.2 - 4~ to the left or right of a target line. The subjects' task was to make a 2 choice judgement of whether they were heading to the left or fight of the target. By calculating the threshold for judgements that were 75% correct, Warren et al estimated that observers could detect the FRO to within 1.2o. The use of dot displays allows the manipulation of the quality of information available to the observer and Warren et al were able to demonstrate extuivalent accuracy when the display only used 10 dots (threshold = 1.26o), which then declined for very sparse displays of 2 dots (threshold = 2.51o). Warren, Blackwell, Kurtz, Hatsopoulos and Kalish (1991) controlled temporal and spatial features of the velocity field to argue that a first-order velocity field was sufficient to support judgements of the order of 0.7~. This was demonstrated by manipulating the permanence of the surface dots so that each element provided a 2-frame velocity vector, but then disappeared. Additional manipulations were also introduced that selectively confounded the use of the velocity vector components of direction and speed. Although these were elegant experimental manipulations of simulated motion, such displays would not be considered as VEs under the terms of the definition adopted in section 1.1. The major limitation was that observers did not interact with the display, but made judgements of passive heading, which breaks the perception-action cycle, by removing the judgement from the activity to which it is germane. This point arises from Warren's (1988) discussion of \"laws of control\" but has also been aptly made by Nakayama (1994) who observed \"As yet, however, researchers have not gone beyond these psychophysical observations to show that animals actually use this information to perform real locomotor tasks\" (p.333). It seems clear that an interactive display or VE simulation of the type presented by Warren et al (1988, 1991) could shed light on the use of the velocity flow field in controlling heading in a setting more akin to natural locomotion. To provide an interactive steering task we presented observers with simulated locomotion across a ground plane where they could actively control directional heading. The ground surface presented was black, textured with 800 randomly placed white dots extending to 300m in front of the starting point. Participants (n=5) were presented with simulated motion at 5m/s, equivalent to a fast run or steady cycle, in an initial direction 10~ to the left or fight of a target tower that appeared on the horizon in the centre of the display. To test the sufficiency of the optic flow field, the direction of the computer