PHYSIOTHERAPY CANADA

Emerson Kavchak et al. Learning Curves Observed in Establishing Targeted Rate of Force Application in Pressure Pain Algometry 135 https://www.utpjournals.press${contentReq.requestUri} - Horizon College Physiotherapy <[email protected]> - Monday, February 22, 2021 11:13:27 PM - IP Address:43.246.243.82 METHODS ensured that no one volunteer was involved in testing more than twice in 1 week or on consecutive days. Clinician participants In this prospective, cross-sectional study, participants Feedback paradigms Clinician participants were instructed to apply the were practising clinicians working in an outpatient reha- bilitation department. Before the study, all clinicians pressure algometer at a rate of force of 5 N/second until participated in a departmental in-service aimed at im- the 40 N/cm2 target was reached. In the first two para- proving the management of patients with chronic pain, digms, they received concurrent feedback during each which included a 1-hour tutorial on QST that allowed of the 12 trials; in the third paradigm, which also con- the opportunity for practice time. The clinician partici- sisted of 12 trials, they received terminal feedback if their pants were instructed to use PPT at their discretion performance missed the target. The feedback paradigms during their clinical practice. Approximately 12 months were consistently completed in the same order: audio- later, after reviewing and signing written informed con- visual (AV) feedback, then visual feedback, then no sent approved by the institutional review board (2012- audiovisual (no AV) feedback. 0780), all clinician participants completed a demo- graphic data sheet that included questions on their use In the AV paradigm, clinician participants listened to of PPT in clinical practice. Clinicians were included in a metronome set at 60 Hz (i.e., chiming once per second) the study if they were interested in participating, had and could observe the visual display on the pressure worked in the clinic for at least the past year, and had algometer. We hypothesized that the ability to hear the attended the in-service described earlier. target rate while observing the actual force applied would facilitate the motor programme development phase2 Healthy volunteers as learners received instruction (pacing) and feedback Healthy volunteers were recruited from among the (visual display). Audio feedback had previously been used to help participants maintain a standard pace12 entry-level Doctor of Physical Therapy student interns and would be analogous to the concurrent audio feed- assigned to the clinic as part of their clinical education back discussed in other psychomotor studies.3 requirement and were included if they reported no musculoskeletal disorders within the previous 6 months. In the visual paradigm, clinician participants could The volunteers reviewed and signed the written informed observe the visual display on the pressure algometer, as consent. described in previous studies.23 We hypothesized that at this stage the learner might be able to correct any errors Testing procedures if the 40 N/cm2 target was reached too quickly or too To assess the learning curve for applying rate of force slowly, through the learner’s own interpretation. at 5 N/second, the authors predetermined a cut-point In the no-AV-feedback paradigm, clinician participants of 40 N/cm2, which allowed for 8 seconds of testing per could no longer see the visual display, as in a previous trial. A handheld 500 N capacity pressure algometer with study in which participants were blinded to the visual a 1 cm2 rubber tip (ForceTen FDX; Wagner Instruments, display.23 They were instructed to apply the algometer Greenwich, CT) is commonly used in the clinic and until they thought the maximum of 40 N/cm2 had been was therefore used for this study. We chose a target of reached; in an effort to protect the volunteers, the inves- 40 N/cm2, which is below lumbar PPT thresholds previ- tigator stopped the trial (essentially providing negative ously reported in people without back pain (45.5 [SD feedback) if force application went on for too long or 25.82] N/cm2 on the right side and 44.7 [SD 22.93] N/cm2 with too much intensity. on the left)28 and within the range of anterior tibialis (AT) PPT thresholds in a healthy female population Force-sensing resistor (33.41 [SD 15.7] N/cm2).14 Three sites were tested bilat- The force-sensing resistor (FSR) was constructed and erally, in random order chosen by rolling a die: the quad- ratus lumborum, the lumbar paraspinals, and the AT. calibrated following the instructions of Tuttle29 (Figure 2), The quadratus lumborum sites were measured 5 cm who developed this device as a way to provide accurate lateral to the fourth lumbar spinous process; the paraspi- real-time or archived feedback of force applied between nal sites were measured 3 cm lateral to the first lumbar 5 and 45 N (e20%).29 Our version of the device used a spinous process; and the AT sites were measured 5 cm Flexiforce A201 1 lb sensor (Tekscan, Inc., South Boston, inferior to the tibial plateau and 2.5 cm lateral to the MA) attached to an 11 mm diameter washer and soft- tibial crest. Three feedback paradigms were used, during ware to quantify force intensity to the thousandth of a which two different sites were assessed on the volun- second. Pilot study findings indicated that the pressure teers. At each site, the clinician participant applied the algometer’s visual display and the FSR findings were algometer six times, with each trial starting 30 seconds comparable. To optimize accuracy, the FSR was cali- after the previous trial ended. The testing schedule brated before each clinician participant was tested and after approximately every three trials during the testing sessions.29

136 Physiotherapy Canada, Volume 68, Number 2 https://www.utpjournals.press${contentReq.requestUri} - Horizon College Physiotherapy <[email protected]> - Monday, February 22, 2021 11:13:27 PM - IP Address:43.246.243.82 of trials on the x axis. These graphs were then qualita- tively examined to determine whether participants’ per- formances demonstrated an initial flat line, an upslope, and a final flat line, indicating a learning curve.5 Figure 2 Force-sensing resistor used with pressure algometer. RESULTS We recruited 22 clinician participants (17 women, 5 men), of whom 17 were included in the final analysis (see Table 1); the remaining 5 were removed from the final data analysis because of technical difficulties with retrieving the archived data. Of the 17 clinician partici- pants with complete data, 5 reported using PPT testing 3–4 times per month; the remaining 12 reported using it rarely or never. We also recruited six volunteers (all female) to act as patients for the testing process. The testing sites and number of trials for each paradigm are described in Table 2. Rate of force application AV paradigm The mean maximum force applied, according to the pressure algometer’s visual display, was 40.57 (SD 2.31) N/cm2 for testing site 1 and 40.53 (SD 0.75) N/cm2 for testing site 2. None of the clinician participants was able to achieve 5 N/second in at least 7 of 8 seconds. Although four clinician participants were able to apply a consistent force during the majority of the testing trials, the rates applied did not meet the criterion standard. Data analysis Visual paradigm We graphed the data generated by each of the 36 The mean maximum force applied, according to the trials, with intensity of force generated on the y axis and pressure algometer’s visual display, was 40.83 (SD 1.23) time on the x axis. The graphs were qualitatively exam- N/cm2 for testing site 1 and 40.62 (SD 0.70) N/cm2 for ined via visual analysis.5 For each trial, we monitored a testing site 2. Although no clinician participant achieved graph of the observed increase in force intensity on a 5 N/second during at least 7 of 8 seconds, five were able second-by-second basis. If the intensity of force in a trial to apply a consistent force, though not at the criterion increased by a total of 5 N/cm2 by the beginning of the standard. subsequent second during 7 of 8 seconds, that trial was dichotomized as ‘‘accurate,’’ meaning that the target No-AV-feedback paradigm force application rate had been met (see Figures 3a and The mean maximum force applied, according to the 3b for examples). pressure algometer’s visual display, was 41.59 (SD 2.72) To determine whether a learning curve was demon- N/cm2 for testing site 1 and 39.54 (SD 3.96) N/cm2 for strated over the course of 36 trials, we would have testing site 2. A single clinician participant was able to needed to analyze the exact rate of force application achieve 5 N/second during at least 7 of 8 seconds; five over each second for each individual trial. However, were able to apply a consistent force, but not at the because the FSR records force to the thousandth of a criterion standard. second, the actual force recorded varied greatly, with large positive-to-negative swings within each second, Learning curves which would have influenced the final calculated rate. Of 17 clinician participants, 1 demonstrated no learn- We therefore chose an alternative method, the time to maximal force. Measuring the time to maximal force ing curve because a steady state of the performance was mimics a clinical situation, in which PPT testing would observed over all three testing paradigms; the other 16 be halted at the maximum amount of force the patient showed no overall learning curve across trials because could tolerate. Using this method, we calculated the rate their performance was inconsistent from one feedback of force application for each trial and plotted it on a paradigm to another. graph, with force per second on the y axis and number In the AV paradigm, four clinician participants showed the initial components of the curve (the initial flatline), indicating that a stable rate of force application was

Emerson Kavchak et al. Learning Curves Observed in Establishing Targeted Rate of Force Application in Pressure Pain Algometry 137 https://www.utpjournals.press${contentReq.requestUri} - Horizon College Physiotherapy <[email protected]> - Monday, February 22, 2021 11:13:27 PM - IP Address:43.246.243.82 Figure 3 Examples of consistent (a) and inconsistent (b) rate of force. being reached, although it was either faster or slower in Figures 3 and 4 are also novel because they give us than the target 5 N/second. It is interesting that perfor- insight into how the performance of force application mance began to worsen in the visual paradigm for those changed in each trial. Because the actual rate of force same four participants. Because performance worsened application during PPT testing can now be quantified, after trial 24 (at the initiation of the no-AV-feedback our results suggest that practising clinicians—even those paradigm) for six participants, there was no final upslope who report more frequent use of PPT—do not con- of the curve (see Figure 4). sistently achieve a rate of 5 N/second. The clinical impact of this finding is that the reliability of PPT, which DISCUSSION is increasingly popular, may need to be improved through To our knowledge, this is the first study to examine more systematic and nuanced educational strategies. The consistent terminal PPT observed in all three paradigms the learning curves associated with deliberate practice suggests that terminal PPT and rate of force application of the psychomotor skill of applying a consistent 5 N/ second of force during PPT testing. The graphs shown

138 Physiotherapy Canada, Volume 68, Number 2 https://www.utpjournals.press${contentReq.requestUri} - Horizon College Physiotherapy <[email protected]> - Monday, February 22, 2021 11:13:27 PM - IP Address:43.246.243.82 Table 1 Participant Demographics Mean (SD)* Clinician Participants 13 4 Physical therapist, no. 82.35 Occupational therapist, no. 36.86 (10.55) % female 12.65 (10.30) Age, y 10.85 (9.19) Time in practice, y Practice in an orthopaedic setting, y *Unless otherwise indicated. Table 2 Number of Trials at each Testing Site per Feedback Paradigm Figure 4 Learning curve with change in performance in the no-AV- feedback trials. No. of trials maximum of four sessions per month for 1 year). How- AV Visual No AV feedback ever, the other clinicians who reported the same level of PPT use in the previous year did not demonstrate a Site 12 1 2 1 2 learning curve. It would appear, therefore, that the actual number of trials is not the only contributor to learning. AT 2 3 5 4 Further research is needed to identify factors other than Right 4 1 4 3 1 3 number of trials that may be required for optimal psy- Left 0 6 chomotor skill learning. 3 3 1 4 QL 2 2 2 4 Our study may not have identified the optimal feed- Right 3 4 back style for learning this skill. We expected that the Left 3 4 4 5 6 2 transition to the visual paradigm would result in an 3 2 3 2 upslope of the curve, suggesting an associative phase of PS learning. Although some clinician participants were able Right 1 1 to demonstrate a more consistent ramp rate, the upslope Left 7 2 of the learning curve was not identified in this paradigm. In the no-AV-feedback paradigm, six clinician participants AT ¼ anterior tibialis; QL ¼ quadratus lumborum; PS ¼ paraspinals. demonstrated worse performance. Although the no-AV- feedback paradigm could be considered a test of perfor- are two distinct aspects of PPT that should both be mance rather than deliberate practice, participants still assessed when examining reliability. received terminal feedback if intensity was too high or duration of testing was too long. Interruption of the trial Learning curves can be used to identify not only the indicated non-optimal performance (i.e., negative feed- optimal number of trials required for deliberate practice5 back). Therefore, although the style of feedback changed, but also, perhaps, the style of feedback that is most effec- clinician participants were still completing deliberate tive. In our study, learning curves reflected poor perfor- practice. Nonetheless, the final acquisition phase of true mance. Participants’ poor performance may have been psychomotor skill learning was not reached; this phase due to the moderate number of trials in our study (36), requires executing the skill without feedback, in varying which may have been insufficient. Even in the AV para- paradigms and over varying durations of practice.3 digm, only four clinician participants were able to demonstrate the initial flat line of cognitive performance The value of demonstrating accurate ramp rate in during the first 12 trials. The institutional review board the no-AV-feedback paradigm can be relevant in clinical acceptance was dependent on demonstrating the welfare practice as well as in research. If terminal PPT is known, of the volunteers; the number of trials we chose was de- the clinician may be biased and may adjust the ramp signed to minimize their spatial or temporal summation. rate as the known terminal threshold is reached. An interesting future study could take the number of trials It was interesting that of the five clinician participants required to demonstrate a learning curve with a visual who reported relatively frequent use of PPT testing (3–4 display and examine performance with the same number times per month), only one showed a steady state of of trials without a visual display. performance over the 36 trials; that is, this clinician had already learned this psychomotor skill and was able to Emerging understandings of the neurophysiological establish a consistent rate of force in varying feedback changes that occur during nociceptive processing and paradigms. This finding may suggest that a higher technical advances in methods of examining in vivo number of trials is needed than was used in this study (i.e., this participant completed a minimum of 144 trials, assuming three applications per testing session and a

Emerson Kavchak et al. Learning Curves Observed in Establishing Targeted Rate of Force Application in Pressure Pain Algometry 139 https://www.utpjournals.press${contentReq.requestUri} - Horizon College Physiotherapy <[email protected]> - Monday, February 22, 2021 11:13:27 PM - IP Address:43.246.243.82 ramp rate make it possible to reconsider the influence of practising clinicians over the course of three different noxious stimulus parameters (intensity of force and feedback paradigms. Therefore, we recommend using speed). In our study, terminal PPT was consistent, but increased deliberate practice with varying feedback para- the rate of force application was not. Contrary to previ- digms to ensure that the optimal rate of force application ously published findings,21,23 our results indicate that is achieved in PPT. achieving a consistent rate of force application requires more deliberate practice, and they do not support the KEY MESSAGES currently reported protocols. The inconsistent rate of force application has practical clinical relevance in that What is already known on this topic PPT is considered a tool to assist clinical diagnostic and Reliability of assessment of terminal pressure pain prognostic decision making. As the use of PPT becomes more commonplace, the current educational guidelines threshold has been established. should be adjusted on the basis of a scientific method for establishing a consistent rate of force application. What this study adds This study quantifies the force per second of the ramp LIMITATIONS Our study has several limitations. The first relates to rate and highlights the difficulty clinicians had in consis- tently applying the recommended rate of force application the equipment used in the trials. The FSR, designed at 5 N/second, which suggests that current educational for easy and inexpensive use in the clinic, has a reported strategies may be insufficient. error rate of e20%.29 On one hand, a more sensitive FSR might help to minimize the variability observed within REFERENCES each second of testing, perhaps allowing for actual calcu- lation of the rate of force application for every second 1. Oermann MH, Kardong-Edgren S, Odom-Maryon T, et al. Deliberate of the trial. On the other hand, the device was able to practice of motor skills in nursing education: CPR as exemplar. Nurs minimize artefacts between the pressure algometer and Educ Perspect. 2011;32(5):311–5. http://dx.doi.org/10.5480/1536- the volunteer, replicating the clinical setting. Further- 5026-32.5.311. Medline:22029243 more, the testing equipment used a convex washer glued to the strip, which is practically similar to assessing PPT 2. Boe SG, Cassidy RJ, McIlroy WE, et al. Single session motor learning on a bony surface. Although the perpendicular algometer demonstrated using a visuomotor task: evidence from fMRI and was aligned to the testing surface,20 it was difficult to behavioural analysis. J Neurosci Methods. 2012;209(2):308–19. maintain the incongruent surfaces. The unequal pressure http://dx.doi.org/10.1016/j.jneumeth.2012.06.016. Medline:22743802 required frequent readjustment of the convex surface; the FSR was recalibrated after the adjustments and between 3. Sigrist R, Rauter G, Riener R, et al. Augmented visual, auditory, clinician participants to minimize error. haptic, and multimodal feedback in motor learning: a review. Psychon Bull Rev. 2013;20(1):21–53. http://dx.doi.org/10.3758/ Second, because clinician participants demonstrated s13423-012-0333-8. Medline:23132605 poor overall learning, it may be that the testing scenarios were not ideal. We attempted to replicate the conditions 4. Ericsson KA. Deliberate practice and the acquisition and mainte- of clinical practice with algometer choice and volunteers nance of expert performance in medicine and related domains. Acad (i.e., in vivo simulation change in normal tissue resis- Med. 2004;79(10 Suppl):S70–81. http://dx.doi.org/10.1097/ tance as the algometer was applied at greater depth). 00001888-200410001-00022. Medline:15383395 This approach limited the number of trials we were able to include. 5. Pusic M, Pecaric M, Boutis K. How much practice is enough? Using learning curves to assess the deliberate practice of radiograph inter- Third, all feedback paradigms were completed in one pretation. Acad Med. 2011;86(6):731–6. http://dx.doi.org/10.1097/ testing session. Although this was a practical choice, ACM.0b013e3182178c3c. Medline:21512374 designed to limit the study’s impact on productivity during clinical hours, a future study could examine 6. Keele KD. Pain-sensitivity tests; the pressure algometer. Lancet. deliberate practice with the different feedback paradigms 1954;266(6813):636–9. http://dx.doi.org/10.1016/S0140- over varying days or weeks. A similarly designed study 6736(54)92347-8. Medline:13143740 that included both short-term (i.e., 1 day) and long-term (i.e., 1 month) follow-up could investigate whether any 7. Giesbrecht RJ, Battie´ MC. A comparison of pressure pain detection of the participants’ learning curves could be demon- thresholds in people with chronic low back pain and volunteers strated or maintained over time. without pain. Phys Ther. 2005;85(10):1085–92. Medline:16180957 CONCLUSION 8. Neziri AY, Curatolo M, Limacher A, et al. Ranking of parameters of Our results indicate that applying a consistent rate of pain hypersensitivity according to their discriminative ability in chronic low back pain. Pain. 2012;153(10):2083–91. http:// force at 5 N/second during PPT testing is difficult for dx.doi.org/10.1016/j.pain.2012.06.025. Medline:22846347 9. Paungmali A, Sitilertpisan P, Taneyhill K, et al. Intrarater reliability of pain intensity, tissue blood flow, thermal pain threshold, pressure pain threshold and lumbo-pelvic stability tests in subjects with low back pain. Asian J Sports Med. 2012;3(1):8–14. http://dx.doi.org/ 10.5812/asjsm.34718. Medline:22461960 10. Gomes MB, Guimara˜es JP, Guimara˜es FC, et al. Palpation and pressure pain threshold: reliability and validity in patients with temporomandibular disorders. Cranio. 2008;26(3):202–10. Medline:18686497 11. van Wilgen P, van der Noord R, Zwerver J. Feasibility and reliability of pain pressure threshold measurements in patellar tendinopathy. J Sci Med Sport. 2011;14(6):477–81. http://dx.doi.org/10.1016/ j.jsams.2011.05.004. Medline:21669552 12. Kavchak AJ, Ferna´ndez-de-Las-Pen˜ as C, Rubin LH, et al. Association between altered somatosensation, pain, and knee stability in

140 Physiotherapy Canada, Volume 68, Number 2 https://www.utpjournals.press${contentReq.requestUri} - Horizon College Physiotherapy <[email protected]> - Monday, February 22, 2021 11:13:27 PM - IP Address:43.246.243.82 patients with severe knee osteoarthrosis. Clin J Pain. 2012;28(7):589– threshold testing in people with and without acute neck pain. J 94. http://dx.doi.org/10.1097/AJP.0b013e31823ae18f. Orthop Sports Phys Ther. 2011;41(9):644–50. http://dx.doi.org/ Medline:22146110 10.2519/jospt.2011.3666. Medline:21885906 13. Walton DM, Macdermid JC, Nielson W, et al. Pressure pain threshold 22. Wylde V, Palmer S, Learmonth ID, et al. Test-retest reliability of testing demonstrates predictive ability in people with acute quantitative sensory testing in knee osteoarthritis and healthy whiplash. J Orthop Sports Phys Ther. 2011;41(9):658–65. http:// participants. Osteoarthritis Cartilage. 2011;19(6):655–8. http:// dx.doi.org/10.2519/jospt.2011.3668. Medline:21885908 dx.doi.org/10.1016/j.joca.2011.02.009. Medline:21329759 14. Walton DM, Macdermid JC, Nielson W, et al. A descriptive study of 23. Chesterton LS, Sim J, Wright CC, et al. Interrater reliability of pressure pain threshold at 2 standardized sites in people with acute algometry in measuring pressure pain thresholds in healthy or subacute neck pain. J Orthop Sports Phys Ther. 2011;41(9):651–7. humans, using multiple raters. Clin J Pain. 2007;23(9):760–6. http:// http://dx.doi.org/10.2519/jospt.2011.3667. Medline:21885907 dx.doi.org/10.1097/AJP.0b013e318154b6ae. Medline:18075402 15. Yarnitsky D. Quantitative sensory testing. Muscle Nerve. 24. Jones DH, Kilgour RD, Comtois AS. Test-retest reliability of pressure 1997;20(2):198–204. http://dx.doi.org/10.1002/(SICI)1097- pain threshold measurements of the upper limb and torso in young 4598(199702)20:2<198::AID-MUS10>3.0.CO;2-#. Medline:9040659 healthy women. J Pain. 2007;8(8):650–6. http://dx.doi.org/10.1016/ 16. Curatolo M, Arendt-Nielsen L. Central hypersensitivity in chronic j.jpain.2007.04.003. Medline:17553750 musculoskeletal pain. Phys Med Rehabil Clin N Am. 2015;26(2):175– 25. Portney LG, Watkins MP. Statistical measures of reliability. In: 84. http://dx.doi.org/10.1016/j.pmr.2014.12.002. Medline:25952059 Portney LG, Watkins MP, editors. Foundations of clinical research: 17. Nie H, Graven-Nielsen T, Arendt-Nielsen L. Spatial and temporal applications to practice. 3rd ed. Upper Saddle River, NJ: Pearson summation of pain evoked by mechanical pressure stimulation. Prentice Hall; 2009. p.585–618. Eur J Pain. 2009;13(6):592–9. http://dx.doi.org/10.1016/ 26. Rolke R, Andrews Campbell K, Magerl W, et al. Deep pain thresholds j.ejpain.2008.07.013. Medline:18926745 in the distal limbs of healthy human subjects. Eur J Pain. 18. List T, Helkimo M, Karlsson R. Influence of pressure rates on the 2005;9(1):39–48. http://dx.doi.org/10.1016/j.ejpain.2004.04.001. reliability of a pressure threshold meter. J Craniomandib Disord. Medline:15629873 1991;5(3):173–8. Medline:1812145 27. Vaughan B, McLaughlin P, Gosling C. Validity of an electronic 19. Jensen K, Andersen HO, Olesen J, et al. Pressure-pain threshold in pressure algometer. Int J Osteopath Med. 2007;10(1):24–8. http:// human temporal region. Evaluation of a new pressure algometer. dx.doi.org/10.1016/j.ijosm.2006.12.003. Pain. 1986;25(3):313–23. http://dx.doi.org/10.1016/0304- 28. Binderup AT, Arendt-Nielsen L, Madeleine P. Pressure pain sensitivity 3959(86)90235-6. Medline:3748589 maps of the neck-shoulder and the low back regions in men and 20. Rolke R, Baron R, Maier C, et al. Quantitative sensory testing in the women. BMC Musculoskelet Disord. 2010;11(1):234–40. http:// German Research Network on Neuropathic Pain (DFNS): standar- dx.doi.org/10.1186/1471-2474-11-234. Medline:20939890 dized protocol and reference values. Pain. 2006;123(3):231–43. 29. Tuttle N, Jacuinde G. Design and construction of a novel low-cost http://dx.doi.org/10.1016/j.pain.2006.01.041. Medline:16697110 device to provide feedback on manually applied forces. J Orthop 21. Walton DM, Macdermid JC, Nielson W, et al. Reliability, standard Sports Phys Ther. 2011;41(3):174–9. http://dx.doi.org/10.2519/ error, and minimum detectable change of clinical pressure pain jospt.2011.3461. Medline:21212499

https://www.utpjournals.press${contentReq.requestUri} - Horizon College Physiotherapy <[email protected]> - Monday, February 22, 2021 11:13:27 PM - IP Address:43.246.243.82 EDUCATION An Exploration of Canadian Physiotherapists’ Decisions about Whether to Supervise Physiotherapy Students: Results from a National Survey Mark Hall, PhD, PT;* Cheryl Poth, PhD;† Patricia Manns, PhD, PT;* Lauren Beaupre, PhD, PT *‡ ABSTRACT Purpose: To explore Canadian physiotherapists’ perceptions of the factors that influence their decisions whether to supervise students in clinical place- ments. Methods: Using accepted survey development methodology, a survey was developed and administered to 18,110 physiotherapists to identify which factors contribute to the decision to supervise students. The survey also gave respondents opportunities to provide comments; these were analyzed via directed content analysis, using the factors identified in an exploratory factor analysis as an organizing structure. Results: A representative sample of 3,148 physiotherapists responded to the survey. Qualitative analysis of respondent comments provided a rich understanding of the factors contributing to the decision on whether to supervise students, which centred on themes related to stress, workplace productivity, the evaluation instrument, student preparation, and physiotherapists’ professional roles and responsibilities. Challenges specific to loss of income and the ethics of charging for student services in private practice were also identified. Conclusions: Supervising students can be stressful, and stress is perceived by respondents to be most influential in deciding whether to supervise students. Effective supervisor training may mitigate some of the stresses related to supervising students. A collaborative approach involving all stakeholders is needed to resolve the issues of student placement capacity. Key Words: factor analysis; internship and residency; preceptorship; students; survey. RE´ SUME´ Objectif : Explorer les perceptions que les physiothe´ rapeutes du Canada ont des facteurs qui jouent sur leur de´ cision d’accepter ou non de surveiller des e´ tudiants en stage clinique. Me´ thodes : On a utilise´ une me´ thodologie reconnue de cre´ ation de sondage pour cre´ er un sondage et l’administrer aupre` s de 18 110 physiothe´ rapeutes afin de de´ gager les facteurs qui contribuent a` la de´ cision d’accepter ou non de superviser des e´ tudiants. Le sondage a aussi permis aux re´ pondants de formuler des commentaires dont on a analyse´ directement le contenu. On a utilise´ comme structure les facteurs de´ gage´ s a` la suite de l’analyse exploratoire des facteurs. Re´ sultats : Un e´ chantillon repre´ sentatif de 3 148 physiothe´ rapeutes a re´ pondu au sondage. L’analyse qualitative des commentaires des re´ pondants a enrichi la compre´ hension des facteurs qui contribuent a` la de´ cision d’accepter ou non de superviser des e´ tudiants, qui pivotait sur des the` mes lie´ s au stress, a` la productivite´ du lieu de travail, a` l’instrument d’e´ valuation, a` la pre´ paration des e´ tudiants et aux roˆ les et responsabilite´ s professionnels des physiothe´ rapeutes. Les re´ pondants ont aussi mentionne´ des de´ fis particuliers au manque a` gagner et a` l’e´ thique de la facturation de ce service d’e´ tudiants dans la pratique prive´ e. Conclusions : La supervision d’e´ tudiants peut eˆ tre stressante et les re´ pondants conside` rent le stress comme le facteur qui a le plus d’influence sur leur de´ cision d’accepter ou non de superviser des e´ tudiants. Une formation en supervision efficace peut atte´ nuer une partie du stress lie´ a` la supervision d’e´ tudiants. Tous les intervenants doivent se concerter pour re´ gler les enjeux lie´ s a` la capacite´ de placement d’e´ tudiants. Clinical education (CE) is critical to physiotherapy force.6 Integral to CE is the willingness of supervising education; it allows students to develop and apply class- physiotherapists, or clinical instructors (CIs), to provide room knowledge, skills, and behaviours in a clinical quality placement experiences. A meaningful and posi- setting under the supervision of a registered physiothera- tive learning environment is created when CIs have time pist.1–7 By developing students’ professional socializa- available for discussion and provision of feedback, form tion,8 CE helps to prepare them for entry into the work- From the: *Department of Physical Therapy, †Centre for Research in Applied Measurement and Evaluation, Department of Educational Psychology, and ‡Department of Surgery (Division of Orthopaedic Surgery), University of Alberta, Edmonton. Correspondence to: Mark Hall, 2–50 Corbett Hall, University of Alberta, Edmonton, AB T6G 2G4; [email protected]. Contributors: All authors designed the study; collected, analyzed, and interpreted the data; drafted or critically revised the article; and approved the final draft. Competing interests: None declared. Acknowledgements: The authors thank Dr. Maxi Miciak for her assistance with the external audit and peer debrief. This study was supported by a grant from the Physiotherapy Foundation of Canada. Physiotherapy Canada 2016; 68(2);141–148; doi:10.3138/ptc.2014-88E 141

142 Physiotherapy Canada, Volume 68, Number 2 https://www.utpjournals.press${contentReq.requestUri} - Horizon College Physiotherapy <[email protected]> - Monday, February 22, 2021 11:13:27 PM - IP Address:43.246.243.82 good interpersonal relationships with their students, and of survey data are presented in detail elsewhere;17 this are enthusiastic about teaching.7 article focuses on the qualitative analysis of respondent comments, using the identified factors as an organizing In physiotherapy, as in many other health science structure. programmes in Canada and elsewhere, quality clinical placements are in short supply.1,7,9,10 Staff shortages, in- METHODS creased patient acuity and health care complexity, re- organizations of health care delivery, and lack of funding Survey instrument have reduced clinical placement opportunities.10–12 In A review of the literature on physiotherapy CE high- addition, anecdotal reports have indicated that the Clini- cal Performance Instrument (CPI),13 currently used to lighted several themes related to clinical placements. evaluate students on clinical placements in Canada, has Augmented by anecdotal reports from Canadian physio- a negative impact on physiotherapists’ willingness to therapists, these themes became constructs to be mea- supervise students. sured. Using standard survey and item development methodology,18–20 we developed a 53-item survey (see The shortage of clinical placement opportunities is Appendix 1 online) to measure four main constructs: significant for physiotherapy university programmes in personal and professional, context, evaluation, and stu- Canada and abroad, which have steadily increased stu- dent. The survey used a 5-point Likert-type scale with dent enrolment to meet health workforce needs.10,14,15 only the extreme anchors (strongly agree and strongly As physiotherapy programmes grapple with placement disagree) labeled. Survey validation procedures involved shortages, understanding the factors that influence phys- review by an expert panel consisting of the academic iotherapists’ supervisory decisions is a critical first step coordinators of clinical education (ACCEs) and assistant in addressing them. ACCEs from 14 Canadian physiotherapy programmes, as well as survey pretesting with a small sample of clini- Current literature specifically addressing Canadian cians from diverse clinical areas.19,21 To ensure that the physiotherapists’ attitudes toward and opinions on CE is survey fully captured the contributors to the decision to limited to small-scale qualitative studies not necessarily supervise a student, respondents were also given the generalizable to the physiotherapy population or the opportunity to provide additional comments. Canadian context.16 A recent Canadian study highlighted stress related to time and space limitations, apprehension Participant recruitment about challenging students, and the decreased flexibility Each of the 11 regulatory colleges in Canada sent a associated with having a student as significant barriers to supervision. Employer support and workplace culture link inviting participation in the online survey via email seemed to affect physiotherapists’ perceptions of the to all actively practising Canadian physiotherapists regis- barriers to and benefits of supervising students and often tered with the college (n ¼ 18,110). The survey was com- contributed to their decisions.16 pleted anonymously; completion of the survey implied consent. The survey remained open for 3 weeks. The Similar themes of increased stress, the need for em- number of reminder emails sent by the regulatory colleges ployer support, and lack of recognition and direct benefit varied, but each college circulated at least one reminder. to CIs recur in previous international work.1,8 All, how- Ethics approval was granted by the University of Alberta ever, take the perspective of public-sector physiotherapists Health Research Ethics Board. actively supervising students. We have found no Canadian studies in which the views of private practitioners or those Data analysis of physiotherapists who do not participate in CE are repre- Respondent comments were analyzed using directed sented, even though a significant number of practising physiotherapists in Canada fall into these two groups.15 content analysis (DCA) to generate validation evidence.22 Lack of empirical evidence on physiotherapy CE in In this study, the six factors identified in the exploratory Canada makes it difficult for educators and policymakers factor analysis represented the theoretical framework to to reach informed decisions about CE, building place- be validated.17 One author with experience in CE (MH) ment capacity, or student evaluation. Comprehensive, coded respondents’ comments deductively, using the six generalizable empirical evidence on the factors that may identified factors as the organizing structure; the factor contribute to physiotherapists’ decisions about becom- items served as operational descriptors for the code. ing involved in CE is needed; therefore, the overall goal Items not explicitly represented by the six factors were of our study was to provide a national, cross-sectional flagged and analyzed after initial coding. The flagged perspective on CE-related issues affecting Canadian phys- comments were examined to determine whether they fit iotherapists in public and private practice. within the six-factor coding structure or whether they represented a new category; when possible, new cate- This article presents results on the qualitative com- gories were aligned with the existing factors. In an effort ponent of a national survey study that identified the to enhance procedural rigour and address trustworthi- factors contributing to physiotherapists’ decisions on ness, we then conducted a peer debrief and external whether to supervise physiotherapy students. The devel- audit; as part of this process, an external consultant, a opment of the survey instrument and statistical analysis

Hall et al. An Exploration of Canadian Physiotherapists’ Decisions about Whether to Supervise Physiotherapy Students 143 Table 1 Survey Response Rates by Province and Territory, Number of Supervisors Active in the Previous 3 Years, and Percentage of Surveys Completed in French* https://www.utpjournals.press${contentReq.requestUri} - Horizon College Physiotherapy <[email protected]> - Monday, February 22, 2021 11:13:27 PM - IP Address:43.246.243.82 Province/territory No. Responses % No. (%) % Email addresses response active in completed past 3 in French rate years British Columbia 3,022 804 27 406 (53) 0.4 Alberta 2,172 314 15 192 (64) 0.3 Saskatchewan 84 13 61 (76) Manitoba 646 166 20 122 (77) 0 Ontario 825 921 13 595 (67) 1 Quebec 7,180 479 17 311 (67) 2 New Brunswick 2,839 178 36 89 (52) 88 Nova Scotia 502 153 24 89 (63) 29 Prince Edward Island 636 —‡ 3 (100) 0 Newfoundland and Labrador —† 3 11 15 (58) 0 Yukon 254 27 12 4 (100) 0 Northwest Territories 34 4 —‡ 3 (75) 0 Nunavut —§ 4 —‡ 1 (100) 0 Missing —§ 1 0 Total 10 124 (4) 18,110 3,148 17 1,895 (63) 16 *Not all respondents completed all survey items; percentages given in this table have been calculated based on the number of responses to each item. † Data not provided. ‡Unable to calculate. §No regulatory college; numbers not known. physiotherapist with qualitative research training, reviewed Clinical instructor feelings of stress all comments and examined coding and categorization to Factor 1 captured physiotherapists’ feelings of stress determine whether the data supported our findings.23,24 and anxiety related to student supervision, including the For the purposes of this study, physiotherapists who added workload of having to supervise students while had supervised at least one student in the previous 3 also completing their own job-related tasks, fear of being years were considered active supervisors; those who had judged by students because their knowledge or skill set not supervised a student in the previous 3 years were might be limited, and worrying that supervising students considered inactive or non-supervisors. might be a burden. In total, 124 comments were classi- fied into this category; a further 259 comments referred RESULTS to aspects of supervision and the work environment that A total of 3,148 Canadian physiotherapists completed ultimately increased CIs’ stress. These comments were placed in nine sub-categories: timing–vacation, staffing, the survey; of these, 1,895 (63% of those who responded lack of time, employer/team support, space, loss of to this item) reported supervising at least one student in income, private practice ethics, and personal. Stress the previous 3 years. Table 1 presents survey response seemed to arise as a result of the perceived extra work rates by province and territory; Table 2 presents respon- involved in supervising a student, including juggling dent demographics. existing caseloads, feelings of being overwhelmed, and the stress caused by a challenging student. These feelings Description of factors were exacerbated by a lack of physical space to accom- The six factors generated by the exploratory factor modate students, lack of employer and team support, and staffing shortages. In addition, fear of supervising analysis represent the multiple and complex issues that a struggling student and the extra time and emotional ultimately contribute to Canadian physiotherapists’ de- energy required during such a placement—though they cisions whether to supervise a student: clinical instructor happen infrequently—left a lasting impression on the CI: feelings of stress, student contribution to workplace effi- ciency, dislike of the assessment instrument, student prep- Having had a weak student this adds to the stress level aration and attitude, clinical instructor preparation to and your ability to ensure all the patients are seen while evaluate, and professional role and responsibility. We still supervising the student and providing feedback along analyzed the 1,792 respondent comments using DCA, as the way to keep them progressing. It is a lot of work and described earlier, and organized them into 18 categories, personally I have put in overtime during those 5 weeks to 16 of which aligned with the six-factor structure. ensure that patient care gets done. It is exhausting. (active supervisor, Manitoba)

144 Physiotherapy Canada, Volume 68, Number 2 https://www.utpjournals.press${contentReq.requestUri} - Horizon College Physiotherapy <[email protected]> - Monday, February 22, 2021 11:13:27 PM - IP Address:43.246.243.82 Table 2 Respondent Demographic Statistics, Area of Practice, Practice Setting, and Workplace Geographic Distribution* No. (%)† Variable Sample Active Inactive or Population of actively supervisors non-supervisors practising physiotherapists Sex 2,607 (83) Female 520 (17) 1,553 (82) 959 (85) in Canada‡ Male 328 (17) 163 (14) 42.41 (10.62) 43.09 (11.35) 14,204 (77) Mean (SD) age, y 17.87 (11.36) 41.95 (10.12) 18.71 (12.27) 4,265 (23) Mean (SD) experience, y 1,983 (63) 17.32 (10.76) 609 (54) Work full time 1,294 (68) 42.0 Practice area§ 1,940 (62) 746 (66) — 1,000 (32) 1,115 (59) 374 (33) Musculoskeletal 587 (31) 251 (22) 8,810 (65) General practice 918 (29) 636 (34) 149 (13) Neurology 528 (17) 358 (19) 5,606 (41) Cardiorespiratory 263 (23) 4,419 (32) Practice setting§ 1,045 (37) 752 (40) 92 (8) General hospital 497 (18) 382 (20) 433 (38) 949 (7) Rehabilitation facility 885 (32) 395 (21) 137 (12) 508 (3) Private practice 276 (10) 130 (7) Community settings 852 (75) 6,288 (34) Practice location 2,513 (80) 1,570 (83) 237 (21) 1,437 (8) Urban 566 (18) 296 (16) 27 (2) 6,308 (34) Rural 49 (2) 22 (1) 1,202 (7) Remote 16,626 (90) 745 (4) 713 (4) *Not all respondents completed all survey items; percentages given in this table have been calculated based on number of responses. † Unless otherwise indicated. ‡Values based on Canadian Institute for Heath Information (CIHI) physiotherapy statistics for 2012.15 CIHI reports primary practice area; however, respondents were permitted to select all that applied. For the sample, rural includes semi-rural; however, CIHI data report rural only. Mean years of experience and standard deviation for the Canadian population are not available. §Respondents could choose more than one practice area and more than one practice setting. One category unique to private practitioners was the Although I have had negative experiences, and despite loss of income associated with supervising a student the fact that there are times I am concerned/stressed and the ensuing stress—particularly with junior students, that a student will challenge my knowledge etc. I feel it who are not able to see as many clients in a day and who is very important to provide students with clinical place- require more of the supervising physiotherapist’s time. ments—and so I ignore the stress and occasional discom- In each case, respondents reported a substantial loss fort. It is important for our profession. (active supervisor, of earnings that made supervising students difficult to Nova Scotia) justify: Student contribution to workplace efficiency It is difficult in a private practice setting to have the time The second factor related to physiotherapists’ effi- to teach clinical skills, especially to someone at a junior student level. Patients paying privately don’t always ciency and productivity while a student is with them. consent to paying full price to have a student learning This factor included items targeting efficiency related to new skills on their bodies. I find I ended up running way students at specific levels (junior vs. senior). Six respond- behind, as I would be teaching as I was going along, ents commented directly on workplace efficiency; addi- which is why I found supervising a student stressful. If I tional comments coded under the student preparation was compensated for my time, I could see less patients and attitude factor captured elements of student perfor- in an hour, to allow for a good learning experience for mance and efficiency related to preparation and training the student. (inactive or non-supervisor, British Columbia) level. Four comments mentioned the CI being slowed down by the student, and two mentioned clients receiv- Not all comments were negative; several respondents ing less effective care—for example, ‘‘I feel students slow acknowledged that although stress might be part of the me down and decrease the quality of care provided to experience, particularly at the beginning, placements were my patients’’ (active supervisor, Ontario). Alternatively, ultimately rewarding, and the initial discomfort was out- one respondent mentioned a collective increase in de- weighed by the overall positive outcome: partment productivity as a result of students’ contribu- tions: ‘‘It doesn’t make me more productive but it might

Hall et al. An Exploration of Canadian Physiotherapists’ Decisions about Whether to Supervise Physiotherapy Students 145 https://www.utpjournals.press${contentReq.requestUri} - Horizon College Physiotherapy <[email protected]> - Monday, February 22, 2021 11:13:27 PM - IP Address:43.246.243.82 make our whole department more productive (especially [I] want to avoid a situation where negative comments a more senior student)’’ (active supervisor, Alberta). have to be given to a student (i.e. professionalism, appro- priateness, dress code or attendance). Also, [I] feel anxious Dislike of the assessment instrument about having to fail a student due to lack of skill. [I] lack Factor 3 captured physiotherapists’ dissatisfaction confidence in my skill and don’t want this to be evident to a student. (inactive or non-supervisor, New Brunswick) with the CPI and the amount of time required to com- plete it. A total of 80 comments coded for this factor Professional role and responsibility validated a general dislike for the CPI. Some respondents Factor 6 focused on physiotherapists’ responsibility to described the CPI as a barrier to supervising students and stated that if a user-friendly instrument were avail- supervise students as part of their health professional able, they would do so more often: role. Items captured elements of professional practice, including reflection, continual professional learning, and I have long considered the evaluation tool to be vague, formal recognition of physiotherapists’ efforts. A total of lengthy, and awfully time-consuming. Another tool that 125 comments, the majority extremely positive, described has opposite characteristics would be a pleasure rather the personal rewards associated with supervising students than an absolute chore to complete and could attract and passing along one’s own positive experiences—for other physiotherapists in becoming supervisors. (active example, ‘‘Having a very strong and excellent mentor has supervisor, Ontario) influenced and shaped who I am as a physiotherapist; I want to be able to provide that same mentorship to other Student preparation and attitude new therapists’’ (active supervisor, British Columbia); ‘‘a Factor 4 pertained to student-related factors that may great way to recharge my physio batteries’’ (inactive or non-supervisor, Alberta). contribute to the outcome of the placement, including academic preparation, adequacy of the student’s edu- We coded 53 comments pertaining to recognition and cation, and the student’s attitude and professional be- compensation for the work done by supervisors in both haviour. A substantial number of comments (225) were public and private practice as recognition, which aligns coded for this factor; they related in particular to stu- with Factor 6. There was a sense that because extra work dents’ preparation for complex caseloads, the student’s goes into supervising students, clinicians should receive level (senior vs. junior), and students’ attitudes. In many some form of compensation—monetary or otherwise— cases, respondents felt that students did not have the for their expertise, making the supervisory experience academic training for complex (often neurological or more meaningful. The need for recognition from univer- pediatric) caseloads. sities, employers, or regulatory colleges was apparent in the number of respondents who suggested that CIs Private practitioners’ perspective was again apparent receive some form of thanks for providing a service, but in this factor, particularly with respect to students’ profi- not all suggested monetary compensation; others high- ciency in an environment in which clients are paying lighted access to the university’s library and e-journal directly for physiotherapy services. Two other codes, collections, clinical educator awards, and recognition loss of income and private practice ethics (57 com- that supervision may be a form of continuing competence ments), also appeared to fit here. Students often needed as compensation options. extra time to treat clients, and many respondents re- ported feeling unethical about charging clients the same Two categories—part-time work and work type— amount as they would for service provided directly by a clarified two reasons for not supervising students but registered physiotherapist; the unbilled extra time- did not align with any of the six factors. The 145 com- decreased clinic productivity and revenue. ments coded as part-time work came from physiothera- pists who work part time and therefore have not offered Clinical instructor preparation to evaluate to supervise a student; the 129 comments coded as work Factor 5 related to physiotherapists’ opinions of their type highlighted physiotherapists working in unusual or specialized roles that they assumed would be unsuitable own preparation to evaluate a student, including both for students. preparation to use the CPI and readiness to evaluate the student through the programme’s provision of clear DISCUSSION guidelines and instructor training. We coded 44 respon- Our goal was to identify Canadian physiotherapists’ dent comments under this factor, many of which related to the physiotherapy programme’s support for CIs deal- perceptions of the factors that influence their decisions ing with struggling students and having the skills to whether to supervise students. An exploratory factor manage that situation. Struggling students increased analysis of survey responses from a diverse and repre- stress and anxiety for some CIs, particularly those who sentative sample of practising Canadian physiotherapists lacked appropriate supervisory training to deal with identified six contributing factors: clinical instructor feel- challenging situations:

146 Physiotherapy Canada, Volume 68, Number 2 https://www.utpjournals.press${contentReq.requestUri} - Horizon College Physiotherapy <[email protected]> - Monday, February 22, 2021 11:13:27 PM - IP Address:43.246.243.82 ings of stress, student contribution to workplace effi- employer support and institutional culture as factors ciency, dislike of the assessment instrument, student in physiotherapists’ decisions to supervise students. The preparation and attitude, clinical instructor preparation stress and barriers associated with student supervision to evaluate, and professional role and responsibility. appeared smaller in supportive environments but were This article describes respondents’ comments related to magnified in institutions that did not support student these factors, which enhance our understanding of the teaching, which ultimately affected physiotherapists’ de- challenges and benefits associated with physiotherapy cision to participate in CE experiences.16 student supervision in Canada. Anecdotal reports have indicated that the CPI also Stress experienced by CIs is perceived to be the most influences physiotherapists’ decisions to supervise stu- influential contributor to the decision to supervise a dents. Recently, these anecdotal reports have become physiotherapy student—not unexpected, given that pre- more frequent, and some CIs have refused to supervise vious research has highlighted stress in CE.8,16,25 Two students if doing so requires using the CPI. Research components of CI stress emerged from our analysis: first, findings to support these reports have been limited, but the feelings of stress associated with student supervi- our study provides empirical evidence that the CPI is a sion—fear of being judged, being unprepared, or getting factor in decisions regarding whether to supervise stu- a challenging student—and, second, those associated with dents and validates the general dislike of the instrument a demanding work environment and with completing in Canada. In their comments, our respondents criti- caseload assignments in addition to providing mentor- cized the CPI’s repetitive nature and lengthy completion ship and supervision. Both component or elements of time. Our findings corroborate those of Creaser33 and stress may be reduced by adequate and appropriate highlight a need for change in student evaluation. supervisor preparation and training26,27 and a workplace that supports student teaching.16 Professional role and responsibility also influenced the decision to supervise students. According to Baldry Although Canadian physiotherapy programmes do Currens and Bithell1 and Sevenhuysen and Haines,34 provide supervisor training, and online supervisor train- most CIs consider CE a core professional role and have ing resources are freely available (e.g., http://www. a strong sense of duty to the profession; however, this preceptor.ca), some physiotherapists evidently do not responsibility is sometimes eroded by workplace de- feel adequately prepared to supervise students. More mands,1 and the drawbacks associated with supervising effective supervisor training is one mechanism for address- students represent a barrier to increased student place- ing some of the stresses associated with student supervi- ments.34 Although the Canadian Physiotherapy Associa- sion and could lead to improved CE experiences.26 The tion Rules of Conduct state that student supervision is importance of CI training for effective and positive CE the professional responsibility of Canadian physiothera- experiences, and of lack of training as a barrier to partic- pists,35 it appears that many physiotherapists are not ipation in CE, has been reported elsewhere,25,26,28,29 but involved in CE; however, this has not been investigated. the literature on physiotherapy CE is sparse. In a nursing study, attendees at supervisor training workshops re- Our study is unique in giving a voice to the roughly ported a better appreciation for the role of preceptor 40% of Canadian physiotherapists who work in private and a greater understanding of the placement expecta- practice.15 A substantial proportion of our sample (28%; tions, which gave them confidence for the supervisory n ¼ 885) reported working in private practice settings, process.27 Failing or struggling students, as reported here and this group articulated challenges with student super- and elsewhere, contribute substantially to supervisor vision that have not previously been reported. Two stress;8,16,30 Ilott31 found that specific training to deal themes related to private practice emerged that pertain with the struggling student was the most valuable com- to the unique stresses faced in these settings: (1) ethical ponent of supervisor training, and it remained valuable considerations concerning charging fees for services pro- even 1 year after the workshop vided by students and (2) loss of income as a result of perceived reductions in productivity. Private physiother- Support from employers is often necessary for em- apy services are often not funded by provincial health ployees to attend supervisor training sessions, and work- insurance plans, and clients (or their private insurance) place culture and employer support are reported to must pay for the services they receive. Some respondents contribute to participation in physiotherapy CE.16 Allen felt uneasy charging clients for services delivered by a and Simpson32 found that attendance at workshops was student rather than a registered physiotherapist; they sometimes poor because of a lack of employer support. felt an obligation to provide proficient, appropriate care, Sometimes information about workshops did not reach particularly for clients paying out of pocket, because clinicians because their managers did not want to arrange funds for physiotherapy services may be limited. coverage for staff attending the workshops.32 Another study found that supervisors who lacked employer support Other respondents reported stress about the loss of perceived supervising students as more stressful25 and income associated with supervising a student. In con- were less likely to use personal time to attend supervisor trast to previously published accounts of increased over- training sessions. Davies and colleagues16 highlighted all productivity of inpatient physiotherapists,36,37 some physiotherapists in our sample reported a reduction in

Hall et al. An Exploration of Canadian Physiotherapists’ Decisions about Whether to Supervise Physiotherapy Students 147 https://www.utpjournals.press${contentReq.requestUri} - Horizon College Physiotherapy <[email protected]> - Monday, February 22, 2021 11:13:27 PM - IP Address:43.246.243.82 productivity and throughput of clients that they attrib- tice community, it also included an over-representation uted to the extra time allocated to mentoring and teach- of supervising physiotherapists. Survey participants self- ing a student. The difference, we suspect, is that a junior selected, and although we specifically encouraged non- student may require more intensive supervision and may supervising physiotherapists to participate, it is possible take longer with clients, whereas a proficient senior that they chose not to complete the survey because they student may in fact increase productivity and the CI’s are disengaged from physiotherapy CE. Finally, our re- overall income. This may counterbalance the effect of sults may be affected by social desirability bias; respond- supervising a junior student at a different time, but the ents may have placed greater emphasis on factors external question warrants further investigation. to student supervision and may have underreported those internal to each respondent. Both of these issues pose a challenge for physiother- apy programmes, and neither is easily resolved. Although CONCLUSIONS students provide care under the supervision of registered Our goal in this study was to discover factors that physiotherapists, the care they provide is unlikely to be of the same standard and proficiency as that of an contribute to Canadian physiotherapists’ decisions about experienced practitioner. This discrepancy has been whether to supervise physiotherapy students. Although acknowledged elsewhere by students.38 At the same time, no single factor alone influences that decision, CI stress however, students acquire and refine clinical skills as they appears to be most influential. Our findings confirm progress through their CE and, later, their own clinical several previously identified contributors (e.g., stress) practice;38 in addition, because students may spend and highlight new ones (e.g., private practice concerns). longer with each client, clients may in fact feel they are A substantial component of stress seems to be related to receiving extra attention and getting better value for supervisors’ feeling unprepared to supervise a student, their money. In general, clients are often satisfied with despite the availability of training workshops and re- the care provided by students,39 but the impact of stu- sources to prepare and support CIs. dents on productivity and physiotherapist income in a private practice setting should be explored. Our findings also highlight positive aspects of supervi- sion, which can be overshadowed by the many challenges Clearly, the decision to supervise a student is multi- associated with CE. When motivated and well-prepared factorial and often individual, which poses a challenge students are supervised by dedicated CIs working in a to stakeholders in physiotherapist education (e.g., em- supportive environment, the result is successful clinical ployers, health care agencies, physiotherapy programmes), placements benefiting all parties. A multipronged approach who face considerable pressures associated with shortages that includes consultation with all stakeholders is needed in clinical placement capacity. Although supervisor train- to resolve the issues of student placement capacity. ing and preparation for the role of CI is within the control of physiotherapy programmes, factors external to phys- KEY MESSAGES iotherapy programmes, such as employer support, also play a role in the challenges related to CE. For these What is already known on this topic reasons, a shared approach between educators and em- A shortage of clinical placements poses a challenge ployers is needed if meaningful resolution to the issues surrounding clinical placement capacity is to be achieved. for many physiotherapy programmes. Current under- It is unlikely that a single intervention will resolve all the standing of the factors affecting clinicians’ participation issues. Moving forward, all stakeholders in CE will need in clinical education is anecdotal or not generalizable to to collaborate in multi-factorial solutions to make super- the Canadian physiotherapy community as a whole. vising students a more rewarding undertaking. What this study adds Our study has several limitations. First, the overall This study’s findings confirm previously reported factors response rate was 17%, which is low but comparable to those of other recent surveys of physiotherapists con- and identify new ones, based on data from a representative ducted in Canada (3% for education-related surveys;40 sample of the Canadian physiotherapy community, includ- 36% for practice-specific surveys41). The variation in re- ing private practice and non-supervising physiotherapists. minder emails sent out by regulatory colleges may have Supervisor training may be a key element in the decision affected response rates. Despite the low response rate, to supervise a student. however, our sample was large, diverse, and, on the basis of recent population statistics, generally representative REFERENCES of the Canadian physiotherapist population. 1. Baldry Currens JA, Bithell CP. Clinical education: listening to Second, although our sample—in contrast to other different perspectives. Physiotherapy. 2000;86(12):645–53. http:// studies—included physiotherapists from throughout dx.doi.org/10.1016/S0031-9406(05)61302-8. Canada, both supervising and non-supervising physio- therapists, and representation from the large private prac- 2. Canadian Physiotherapy Association. Position statement: clinical education of physiotherapy students [Internet]. Toronto: The Association; 2008 [cited 2015 Mar 20]. Available from: http:// www.physiotherapy.ca/getmedia/3b256d44-e16f-4350-b74f- 49e4721b7dec/Clinical-Education-of-Physiotherapy-Students_ en.pdf.aspx.

148 Physiotherapy Canada, Volume 68, Number 2 https://www.utpjournals.press${contentReq.requestUri} - Horizon College Physiotherapy <[email protected]> - Monday, February 22, 2021 11:13:27 PM - IP Address:43.246.243.82 3. Lekkas P, Larsen T, Kumar S, et al. No model of clinical education 23. Creswell JW. Qualitative inquiry and research design: choosing for physiotherapy students is superior to another: a systematic among five approaches. 2nd ed. Thousand Oaks (CA): Sage; 2007. review. Aust J Physiother. 2007;53(1):19–28. http://dx.doi.org/ 10.1016/S0004-9514(07)70058-2. Medline:17326735 24. Lincoln YS, Guba EG. Naturalistic inquiry. Newbury Park (CA): Sage; 1985. 4. Miller A, Pace T, Brooks D, et al. Physiotherapy internship: an alternative collaborative learning model. Physiother Can. 25. Maloney P, Stagnitti K, Schoo A. Barriers and enablers to clinical 2006;58(2):157–66. http://dx.doi.org/10.3138/ptc.58.2.157. fieldwork education in rural public and private allied health practice. High Educ Res Dev. 2013;32(3):420–35. http://dx.doi.org/10.1080/ 5. Murphy S, Dalton M, Dawes D. Assessing physical therapy students’ 07294360.2012.682255. performance during clinical practice. Physiother Can. 2014;66(2):169–76. http://dx.doi.org/10.3138/ptc.2013-26. 26. Hanson DJ. The perspectives of fieldwork educators regarding level Medline:24799754 II fieldwork students. Occup Ther Health Care. 2011;25(2-3):164–77. http://dx.doi.org/10.3109/07380577.2011.561420. Medline:23899033 6. Rogers JL, Lautar CJ, Dunn LR. Allied health students’ perceptions of effective clinical instruction. Health Care Manag (Frederick). 27. Myrick F, Luhanga F, Billay D, et al. Putting the evidence into pre- 2010;29(1):63–7. http://dx.doi.org/10.1097/HCM.0b013e3181cca311. ceptor preparation. Nurs Res Prac; 2012: 948593. http://dx.doi.org/ Medline:20145469 10.1155/2012/948593 7. Recker-Hughes C, Wetherbee E, Buccieri KM, et al. Essential 28. Higgs J, McAllister L. The lived experiences of clinical educators with characteristics of quality clinical education experiences: standards to implications for their preparation, support, and professional devel- facilitate student learning. J Phys Ther Educ. 2014;28(Suppl. 1):48– opment. Learn Health Soc Care. 2005;4(3):156–71. http://dx.doi.org/ 55. 10.1111/j.1473-6861.2005.00097.x. 8. O¨ hman A, Hagg K, Dahlgren L. A stimulating, practice-based 29. Yonge O, Ferguson L, Myrick F, et al. Faculty preparation for the job facing increased stress—clinical supervisors’ perceptions of preceptorship experience: the forgotten link. Nurse Educ. professional role, physiotherapy education and the status of the 2003;28(5):210–1. http://dx.doi.org/10.1097/00006223-200309000- profession. Adv Physiother. 2005;7(3):114–22. http://dx.doi.org/ 00006. Medline:14506351 10.1080/14038190510010359. 30. Luhanga F, Yonge OJ, Myrick F. ‘‘Failure to assign failing grades’’: 9. Mooney S, Smythe L, Jones M. The tensions of the modern-day issues with grading the unsafe student. Int J Nurs Educ Scholarsh. clinical educator in physiotherapy: A scholarly review through a 2008;5(1):1–14. http://dx.doi.org/10.2202/1548-923X.1366. critical theory lens. NZ J Physiother. 2008;36(2):59–66. Medline:18384275 10. Rodger S, Webb G, Devitt L, et al. A clinical education and practice 31. Ilott I. To fail or not to fail? A course for fieldwork educators. Am J placements in the allied health professions: an international Occup Ther. 1995;49(3):250–5. http://dx.doi.org/10.5014/ perspective. J Allied Health. 2008;37(1):53–62. Medline:18444440 ajot.49.3.250. Medline:7741159 11. Health Council of Canada. An environmental scan of current views 32. Allen C, Simpson A. Peers and partners: working together to on health human resources in Canada: identified problems, pro- strengthen preceptorship in mental health nursing. J Psychiatr Ment posed solutions and gap analysis. Available from: Toronto: Health Health Nurs. 2000;7(6):505–14. http://dx.doi.org/10.1046/j.1365- Council of Canada; 2005 [cited 2015 Mar 20]. http://healthcouncil- 2850.2000.00342.x. Medline:11933508 canada.ca/tree/2.13-EnvironScanENG.pdf. 33. Creaser GA. An exploration of clinical instructors’ experiences and 12. British Columbia Academic Health Council. Practice education perceptions of the physical therapy clinical performance instrument survey final report: planning for sufficient and appropriate student [dissertation]. Halifax (NS): Mount Saint Vincent University; 2006. placements for health professional students in BC. Vancouver: BC Academic Health Council; 2005. 34. Sevenhuysen SL, Haines T. The slave of duty: why clinical educators across the continuum of care provide clinical education in physio- 13. Roach K, Gandy J, Deusinger SS, et al.; Task Force for the Develop- therapy. Hong Kong Physiother J. 2011;29(2):64–70. http:// ment of Student Clinical Performance Instruments, American dx.doi.org/10.1016/j.hkpj.2011.06.002. Physical Therapy Association. The development and testing of APTA clinical performance instruments. Phys Ther. 2002;82(4):329–53. 35. Canadian Physiotherapy Association. Rules and regulations [Inter- Medline:11922850 net]. Ottawa: The Association; c2011 [cited 2015 Mar 16]. Available from: http://www.physiotherapy.ca/getmedia/162eaa85-5a9d-44c7- 14. Canadian Institute for Health Information. Physiotherapists in 97e6-019f5523a592/CPA-Rules-Regulations.pdf.aspx. Canada, 2009. Ottawa: The Institute; 2010. 36. Bristow D, Hagler P. Comparison of individual physical therapists’ 15. Canadian Institute for Health Information. Physiotherapist work- productivity to that of combined student-therapist pairs. Physiother force, 2012. Ottawa: The Institute; 2013. Can. 1997;Winter:16–24. 16. Davies R, Hanna E, Cott C. ‘‘They put you on your toes’’: physical 37. Dillon LS, Tomaka JW, Chriss CE, et al. The effect of student clinical therapists’ perceived benefits from and barriers to supervising experiences on clinician productivity. J Allied Health. students in the clinical setting. Physiother Can. 2011;63(2):224–33. 2003;32(4):261–5. Medline:14714600 http://dx.doi.org/10.3138/ptc.2010-07. Medline:22379263 38. Tryssenaar J, Perkins J. From student to therapist: exploring the first 17. Hall M, Poth C, Manns P, et al. To supervise or not to supervise a year of practice. Am J Occup Ther. 2001;55(1):19–27. http:// physical therapy student: a national survey of Canadian physical dx.doi.org/10.5014/ajot.55.1.19. Medline:11216362 therapists. J Phys Ther Educ. 2015;29(3):56–65. 39. Stiller K, Sorich M, Roberts K. Evaluating patients’ attitudes towards 18. Czaja R, Blair J. Designing surveys: a guide to decisions and proce- being assessed and treated by undergraduate physiotherapy dures. 2nd ed. Thousand Oaks (CA): Pine Forge Press; 2005. students in a rehabilitation centre. Internet J Allied Health Sci Prac. 2013;11(1): Article 3. 19. Bradburn NM, Sudman S, Wansink B. Asking questions: the defini- tive guide to questionnaire design: for market research, political 40. National Physiotherapy Advisory Group. Essential competency polls, and social and health questionnaires. Rev. ed. San Francisco: profile for physiotherapists in Canada [Internet]. The Group; c2009 Jossey-Bass; 2004. [cited 2015 Mar 16]. Available from: http://www.physiotherapyedu- cation.ca/Resources/Essential%20Comp%20PT%20Profi- 20. Fowler FJ. Survey research methods. 4th ed. Thousand Oaks (CA): le%202009.pdf. Sage; 2009. 41. Doyle L, Mackay-Lyons M. Utilization of aerobic exercise in adult 21. DeVellis RF. Scale development: theory and applications. 2nd ed. neurological rehabilitation by physical therapists in Canada. J Thousand Oaks (CA): Sage; 2003. Neurol Phys Ther. 2013;37(1):20–6. http://dx.doi.org/10.1097/ NPT.0b013e318282975c. Medline:23389387 22. Hsieh HF, Shannon SE. Three approaches to qualitative content analysis. Qual Health Res. 2005;15(9):1277–88. http://dx.doi.org/ 10.1177/1049732305276687. Medline:16204405

https://www.utpjournals.press${contentReq.requestUri} - Horizon College Physiotherapy <[email protected]> - Monday, February 22, 2021 11:13:27 PM - IP Address:43.246.243.82 Clinician’s Commentary on Hall et al.1 It is well established that clinical education (CE) is a vital be a deterrent to therapists who might otherwise offer to super- part of entry into practice training for physiotherapy students. vise a student.4 Although there is no denying that a struggling This form of education depends on the willingness of clinicians student can demand more of a therapist’s time and skills, can to take on a supervisory or instructor role. Educational pro- we be proactive to relieve this burden and perhaps lessen the grammes are increasingly being challenged to find quality fear? Are there ways to communicate this support to a potential placements, and larger class sizes are putting further demands clinical instructor that would help abate or lessen fears (and on these already scarce resources. The article by Hall and col- stress) before he or she makes the decision to offer an intern- leagues1 provides a compelling argument for understanding the ship? How can a university programme be there for a clinician factors that influence therapists’ decisions to supervise or not before he or she makes this decision? supervise students. Hall and colleagues1 also give a voice to the private sector, The article identifies six factors that influence this decision: which has been absent from the CE literature until now. Forty- stress, workplace productivity, the evaluation instrument, stu- eight percent of Canadian physiotherapy practice is in the dent preparation, instructor preparation (related to evaluation), private setting,5 and many new graduates will also work in and physiotherapists’ professional roles and responsibilities. these environments. Having private-practice CE opportunities The article also identifies challenges specific to loss of income is essential for preparing students to work in these settings. It is and the ethics of charging for student services in private prac- therefore important to understand the factors that influence a tice and clusters these challenges under the student preparation private practitioner in the decision to supervise a student. Loss factor. of income and difficult decisions related to charging patients appear to be at the forefront of the private sector’s concerns. The strongest influencer identified is therapist stress. Clini- University programmes need to find ways to work with this cians are experiencing increasing pressure in the health care group to seek solutions and overcome these barriers. environment, with leaner staffing models in patient care envi- ronments (including outright short staffing), increased acuity Two additional factors arose from the data that did not and complexity of patients, and quicker turnover of patients fit into the original six identified factors, and both had a nega- through the health care system. Practices are also changing. tive impact on a person’s decision on whether to supervise a For example, acute care environments are seeing a shift in the student. Part-time work (145 comments) and work type (129 cardiorespiratory world from manual chest care (percussions, comments) were seen as barriers to supervising students. The vibrations, etc.) to chest care interventions that are more focused part-time workforce makes up 35% of practising physiothera- on mobility. In some environments, therapists no longer believe pists,5 and omitting these potential supervisors results in lost that they are able to offer students a cardiorespiratory learning opportunities. Are there models of supervision that would be experience, voicing concerns about a disconnect between stu- more conducive for therapists in part-time work? How do we dents’ expectations and the type of learning that therapists are encourage this vital part of the profession to become engaged able to provide. The reluctance of these clinicians to supervise in clinical education, and how do we keep them engaged? places an additional barrier on a system that is already under stress. As health care evolves and roles change and develop, physio- therapists are finding themselves in different types of work. It has been suggested that the stress associated with CE can For example, some physiotherapists are working in advanced- be decreased by a work environment that supports student CE practice, consultative, or specialized physiotherapy roles. These and by training courses aimed at preparing the clinician for the are roles and environments that students arguably need to learn role of supervisor and evaluator. Hall and colleagues1 point out about and from. How do we stretch the historical view of CE that these types of courses are available—for example, through and create opportunities in these environments? free web-based programmes such as the Preceptor Education Program for Health Professionals and Students2—and are also As Canadian university programmes strive to maintain and offered by university physiotherapy programmes. But are these grow the level of CE opportunities in quality learning environ- programmes having the desired outcome of producing better ments, it is imperative to understand how to engage this pre- prepared (and therefore less stressed) clinical educators? Does dominantly voluntary workforce. Hall and colleagues1 have laid awareness of these programmes mean that people will use the groundwork for a better national understanding of the them? Are we preaching to the choir in that attending clinicians factors influencing CE supervision. It is incumbent on the mem- are the individuals who are already supervising students? How bers of the profession to work collaboratively with the university do we engage those who are not currently engaged? Can we start programmes to address the issue of student placement capacity. engaging therapists in student supervision early in people’s careers so that clinical instructor is one of the roles that thera- Robyn Davies, BHScPT, MAppSc pists naturally take on? Lecturer, Department of Physical Therapy, Faculty of Medicine, University of Toronto; Education Leader, Specific training in dealing with a struggling student has Physiotherapy Services, Sunnybrook Health Science been identified as the most valuable component of supervisory training.3 The fear of having a difficult or struggling student can Centre, Toronto 149

150 Physiotherapy Canada, Volume 68, Number 2 https://www.utpjournals.press${contentReq.requestUri} - Horizon College Physiotherapy <[email protected]> - Monday, February 22, 2021 11:13:27 PM - IP Address:43.246.243.82 REFERENCES 4. Davies R, Hanna E, Cott C. ‘‘They put you on your toes’’: physical therapists’ perceived benefits from and barriers to supervising 1. Hall M, Poth C, Manns, P, Beaupre, L. An exploration of Canadian students in the clinical setting. Physiother Can. 2011;63(2):224–33. physiotherapists’ decisions whether to supervise physiotherapy http://dx.doi.org/10.3138/ptc.2010-07. Medline:22379263 students: Results from a national survey. Physiother Can. 2016;68(1):141–48. 5. Canadian Institute for Health Information. Physiotherapist work- force, 2012. Ottawa, ON: Canadian Institute for Health Information; 2. Kinsella EA, Bossers A, Ferguson K, et al. Preceptor education 2013. program for health professionals and students [Internet]. 2nd ed. London: University of Western Ontario; 2007 [cited 2016 Jan 12]. DOI:10.3138/ptc.2014-88-CC Available from: http://www.preceptor.ca 3. Ilott I. To fail or not to fail? A course for fieldwork educators. Am J Occup Ther. 1995;49(3):250–5. http://dx.doi.org/10.5014/ ajot.49.3.250. Medline:7741159

https://www.utpjournals.press${contentReq.requestUri} - Horizon College Physiotherapy <[email protected]> - Monday, February 22, 2021 11:13:27 PM - IP Address:43.246.243.82 EDUCATION Examining the Need for a New Instrument to Evaluate Canadian Physiotherapy Students during Clinical Education Experiences Mark Hall, PhD, PT;* Patricia Manns, PhD, PT;* Cheryl Poth, PhD;† Lauren Beaupre, PhD, PT *‡ ABSTRACT Purpose: To gauge the need for a new assessment instrument for Canadian physiotherapy students on clinical placements. Methods: A national survey was developed and distributed to 18,110 Canadian physiotherapists. Results: A total of 3,148 physiotherapists from diverse practice settings responded to the survey. Of those who indicated that student evaluation was applicable to them (n ¼ 2,393), 70% stated that a new instrument was needed; of these, 78% felt that the new instrument should be based on Canadian practice standards and rated with an anchored visual analogue scale, and 73% said they would be comfortable completing the instrument online. Conclusion: The majority of physiotherapists surveyed perceive a need for a new clinical evalua- tion instrument based on Canadian practice standards. A shorter, Canadian-based instrument may help recruit more clinical instructors and build capacity for clinical placements. Key Words: educational measurement; students; survey. RE´ SUME´ Objectif : E´ valuer la ne´ cessite´ d’un nouvel outil d’e´ valuation pour les e´ tudiants en physiothe´ rapie du Canada en stage clinique. Me´ thodes : Un sondage national a e´ te´ mis au point et distribue´ a` 18 110 physiothe´ rapeutes canadiens. Re´ sultats : Un total de 3 148 physiothe´ rapeutes provenant de divers contextes de pratique ont re´ pondu au sondage. Des physiothe´ rapeutes qui ont indique´ que l’e´ valuation des e´ tudiants s’applique a` eux (n ¼ 2 393), 70 % ont affirme´ qu’un nouvel outil est ne´ cessaire; de ce groupe, 78 % sont d’avis que le nouvel outil devrait eˆ tre fonde´ sur les normes de pratique canadiennes et qu’il devrait utiliser une e´ chelle analogique visuelle pre´ sentant deux valeurs extreˆ mes, et 73 % ont affirme´ qu’ils n’auraient pas de difficulte´ a` remplir l’outil d’e´ valuation en ligne. Conclusion : La majorite´ des physiothe´ rapeutes qui ont re´ pondu au sondage jugent qu’un nouvel outil d’e´ valuation fonde´ sur les normes de pratique canadiennes est ne´ cessaire. Un outil plus concis et axe´ sur le contexte canadien pourrait contribuer au recrutement d’un plus grand nombre d’enseignants cliniques et accroıˆtre la capacite´ destine´ e aux stages cliniques. Clinical education (CE) is an important and sub- mining their competence,2 and the assessment tools stantial component of physiotherapy students’ educa- used to make decisions about competence must be tion that permits the application of classroom knowledge ‘‘robust, feasible and of educational value’’ and provide and professional socialisation in a real clinical environ- a valid measure of performance.2(p.488) However, the ment under the supervision of a clinical instructor (CI).1 instrument currently used to assess Canadian physio- The role of the CI is to mentor, observe, and assess stu- therapy students in CE has been identified as contribu- dents during their CE experiences and to provide feed- ting to a shortage of clinical placement offers.3 back to guide and improve performance. During clinical placements, students’ knowledge, skills, and behaviours At present, most Canadian university programmes in are assessed and compared with a reference standard; physiotherapy evaluate students using the 1997 Physical students must meet the expectations for each placement Therapist Clinical Performance Instrument (CPI), a 24- to progress to the next level. Assessing students’ perfor- item instrument developed and validated by the Ameri- mance in the clinical environment is important in deter- can Physical Therapy Association.4 The CPI is based on US practice standards and education guidelines5,6 and From the: *Department of Physical Therapy, †Centre for Research in Applied Measurement and Evaluation, Department of Educational Psychology, and ‡Department of Surgery (Division of Orthopaedic Surgery), University of Alberta, Edmonton. Correspondence to: Mark Hall, 2–50 Corbett Hall, University of Alberta, Edmonton, AB T6G 2G4; [email protected]. Contributors: All authors designed the study; collected, analyzed, and interpreted the data; drafted or critically revised the article; and approved the final draft. Competing interests: None declared. Acknowledgements: Funding for this project was provided by the Physiotherapy Foundation of Canada. Physiotherapy Canada 2016; 68(2);151–155; doi:10.3138/ptc.2014-89E 151

152 Physiotherapy Canada, Volume 68, Number 2 https://www.utpjournals.press${contentReq.requestUri} - Horizon College Physiotherapy <[email protected]> - Monday, February 22, 2021 11:13:27 PM - IP Address:43.246.243.82 includes a visual analogue scale (VAS) anchored by actively practising Canadian physiotherapists registered the performance indicators novice clinical performance with a provincial regulatory college (n ¼ 18,110) were and entry-level performance, with comment boxes for sent an email by their college inviting their participa- each criterion as well as summative comment boxes.4 tion in an anonymous online survey; completion of the Together, the 24 criteria (including safety, professional survey implied consent to participate in the study. The behaviour, assessment, intervention, and discharge plan- survey remained open for 3 weeks, and although the ning) are reported to represent a physiotherapist’s com- number of reminder emails varied, each college sent at petence.4 The skills required of a competent Canadian least one. The development and validation of the online physiotherapist are described under the seven roles of survey, as well as primary results from the survey, have the Essential Competency Profile for Physiotherapists in been reported elsewhere.3 Canada (ECP);7 although these skills are very similar to those described in the US competency standards, the Survey items were intended to measure physiothera- practice context, language, and legal requirements differ. pists’ opinions on whether a new instrument was needed and whether it should be based on the ECP. Questions Recently, academic coordinators of clinical education also addressed respondents’ perspectives on the desired have reported CI dissatisfaction with the time required format, the assessment rating scale, and training required to complete the CPI and its lack of applicability to the to use a new instrument. Respondents also had the Canadian context,8 but little research has corroborated opportunity to provide comments about a new instru- these reports.9 In a previous study, we identified the CPI ment. The survey used skip logic,11 such that only those as a factor in physiotherapists’ decision about whether to respondents who indicated that a new instrument was supervise a student and confirmed physiotherapists’ dis- needed were asked further questions about the develop- like of the instrument.3 This study highlighted the length ment of such an instrument. We analyzed the survey of time to complete the CPI and its lack of applicability data using descriptive statistics (mean, median, range, to the Canadian context. As Canadian physiotherapy pro- and frequency distributions) and conducted a content grammes struggle to increase clinical placement capacity, analysis of respondent comments, using representative it is crucial to address the use of an evaluation instru- quotes to support the quantitative findings. Approval for ment that discourages physiotherapists from supervising this study was granted by the Health Research Ethics students. Board of the University of Alberta. Although the CPI has been established as a con- RESULTS tributor to Canadian physiotherapists’ decisions about A total of 3,148 physiotherapists from all provinces whether to supervise students, whether a new instru- ment was needed was less clear. The findings of a pre- and territories, representing diverse practice settings and liminary study by Anderson and colleagues,10 which practice areas, responded to the survey (a response rate reported on focus groups comprising of 27 physiothera- of 17%). Of those who indicated that student evaluation pists from both public and private practice across was applicable to them (n ¼ 2,393), 70% (n ¼ 1,683) Canada, indicated a need for change in how students identified a need for a new evaluation instrument to as- on clinical placement are evaluated and suggested that sess physiotherapy students on clinical placement. On these evaluations should be based on the ECP.10 Some the basis of this clear mandate for a new instrument, all participants felt that a rating scale with clear and defined subsequent data reported here are from this subset of performance anchors was needed to enable them to pro- 1,683 respondents. Demographic data for survey respond- vide appropriate feedback to students, and others felt ents and for the national physiotherapy population are that anchor points would be restrictive and preferred presented in Table 1. a scale on a continuum or one with more intermediate rating points.10 Although this qualitative study presented Of the respondents who responded ‘‘yes,’’ 1,050 (63%) important findings, it represented the perspectives of reported having supervised between one and five stu- only a small portion of the population. dents in the previous 3 years, 292 (17%) had supervised more than five students, and 341 (20%) had not super- The purpose of the current study, therefore, was to vised any students in that period. Respondents agreed provide cross-sectional, national data on physiotherapists’ or strongly agreed (n ¼ 1,310; 78%) that a new instru- perceptions of the need to develop a new instrument to ment should be based on Canadian practice standards, evaluate Canadian physiotherapy students during CE represented by the ECP; 5% reported applying the seven experiences and, if that need exists, to determine what key roles in the evaluation of physiotherapy students, features CIs believe the new instrument should have. 20% had read the ECP and understood its application to their practice setting, a further 20% reported having only METHODS read it, and 55% had heard about the ECP but never read This study is part of a larger survey of Canadian physio- it. When asked to choose their preferred rating scale or scales from a choice of eight scales, 1,045 chose a VAS therapists that identified the contributors to their decisions with six defined anchors; 615 chose a 5-point Likert scale about whether to supervise physiotherapy students.3 All

Hall et al. Examining the Need for a New Instrument to Evaluate Canadian Physiotherapy Students during Clinical Education Experiences 153 Table 1 Demographic Data for the Full Sample, for the Subset of the Sample Who Responded ‘‘Yes’’ Regarding the Need for a New Evaluation Instrument, and for the National Physiotherapy Population https://www.utpjournals.press${contentReq.requestUri} - Horizon College Physiotherapy <[email protected]> - Monday, February 22, 2021 11:13:27 PM - IP Address:43.246.243.82 No. (%) of responses* Variable Sample Responding ‘‘yes’’ Population (n ¼ 3,148) (n ¼ 1,683) (n ¼ 18,469)† Gender 2,607 (83) 1,407 (84) 14,204 (77) Female 520 (17) 265 (16) 4,265 (23) Male 42.41 (10.62) 41.67 (10.63) 42.0 Age, y, mean (SD) 17.87 (11.36) 16.89 (11.44) N/A Experience, y, mean (SD) 1,983 (65) 1,131 (69) 8,810 (65) Work full time 1,895 (63) 1,250 (74) N/A Supervised at least 1 student in past 3 years Practice location 2,513 (80) 1,400 (84) 16,626 (90) 566 (18) 133 (8) 745 (4) Urban 49 (2) 19 (1) 713 (4) Rural Remote 1,940 (62) 953 (57) 5,606 (41) Practice area‡ 1,000 (32) 530 (32) 4,419 (32) MSK 552 (33) General practice 918 (29) 343 (20) 949 (7) Neurology 528 (17) 508 (3) Cardiorespiratory 699 (44) Practice setting‡ 1,045 (37) 367 (23) 6,288 (34) General hospital 497 (18) 389 (25) 1,437 (8) Rehabilitation facility 885 (32) 6,308 (34) Private practice *Unless otherwise indicated. As not all participants responded to every item, percentages in this table are out of total responses to each item. † Indicates values based on Canadian Institute for Heath Information (CIHI) physiotherapy population statistics for 2012.12 ‡CIHI reports primary practice area only; respondents were asked to select all that apply. N/A ¼ data not available; MSK ¼ musculoskeletal. with only the extreme anchors defined; and 410 chose a terrible, it is just terribly long, and not all components 5-point Likert scale with each anchor defined. are applicable.’’ When respondents were asked about the maximum DISCUSSION amount of time they would be willing to spend com- Our results, which reflect a national, cross-sectional pleting the evaluation instrument, the median response was 45 minutes (inter-quartile range [IQR] ¼ 30–60 min); sample, indicate that physiotherapists perceive a need the median response for the maximum time to review for a new instrument to evaluate Canadian physiotherapy the evaluation with the student was also 45 minutes students during clinical education, as highlighted initially (IQR ¼ 30–60 min). Seventy-three percent (n ¼ 1,237) of by focus-group participants.10 Our findings point to two respondents responding ‘‘yes’’ indicated that they would considerations for the development of such an instru- be comfortable or very comfortable completing the ment and an implication for the profession as a whole. evaluation online. With respect to training to use a new First, to improve measurement consistency and accuracy instrument, 40% preferred an online training module; of performance to standards in the Canadian context, 33%, an in-person workshop; 18%, an online workshop the instrument should be based on the ECP. A national or webinar; and approximately 9%, other options. Only instrument reduces the burden on CIs supervising stu- 1% felt no training was needed. dents from multiple universities, and standardisation ensures that all students are assessed against explicit A total of 337 respondents provided comments; our standards for entry-level practice.13 Australia experienced analysis revealed a consensus that a new instrument similar challenges, and the need for standardisation based on Canadian practice standards is urgently needed: fueled the development of the Assessment of Physio- for example, ‘‘Must be aligned with Canadian practice therapy Practice (APP), now in use across that country.13 standards otherwise it is irrelevant,’’ ‘‘It’s about time,’’ and ‘‘Long overdue.’’ Respondents advocated a simple, In Canada, Murphy and colleagues8 piloted the APP easy-to-use instrument—‘‘shorter, please’’—and some with students and CIs from the University of British preferred an online version: ‘‘It is time for a new elec- Columbia who worked in a range of practice settings tronic evaluation tool.’’ Some even asked that we ‘‘please and areas of practice. The APP is shorter than the CPI, hurry.’’ A few respondents also commented on the appli- uses terminology similar to that used in Canada, and cability and usability of the CPI: ‘‘The CPI is not totally uses domains of assessment that map well onto the

154 Physiotherapy Canada, Volume 68, Number 2 https://www.utpjournals.press${contentReq.requestUri} - Horizon College Physiotherapy <[email protected]> - Monday, February 22, 2021 11:13:27 PM - IP Address:43.246.243.82 ECP.8 Murphy and colleagues8 reported that CIs and CONCLUSION students preferred the APP to the CPI and considered Canadian physiotherapists perceive a need for the the APP’s completion time acceptable; they found over- whelming support for the APP, with only one CI prefer- development of a new evaluation instrument to assess ring to continue using the CPI. These findings and those students during clinical education experiences that is of Anderson and colleagues10 reinforce the results of our based on Canadian physiotherapy competency standards. survey and point to the second implication for instru- Considering the challenges associated with clinical place- ment development. Clinical instructors prefer a shorter ment capacity and the identification of student evaluation instrument with objective performance indicators that as a factor in physiotherapists’ decisions about supervis- provides clear and meaningful objective feedback to ing students,3 physiotherapists’ opinions from this study students and is less onerous to complete. A user-friendly and that of Anderson and colleagues10 should inform the Canadian instrument may persuade physiotherapists development of a new instrument. A shorter, Canadian discouraged by the length and applicability of the CPI to evaluation instrument may be helpful in recruiting phys- become involved with student CE, which may begin to iotherapists interested in physiotherapy CE and may address one of the causes of reduced clinical placement contribute to building clinical placement capacity. capacity. KEY MESSAGES Involvement beyond CIs is important during the development and field testing of a new instrument. Addi- What is already known on this topic tional input should be sought from academic partners Physiotherapists’ opinions of the Clinical Performance and other physiotherapy education stakeholders to en- sure that a new instrument meets the assessment needs Instrument and the evaluation of students have until re- of education programmes and licensing agencies. The cently been largely anecdotal. Current literature reports CPI underwent substantial pilot and field testing during findings from qualitative studies that may not be gener- its development and is considered a valid and reliable alizable to the national population. measure of student clinical performance;4 a new instru- ment will need to undergo similarly rigorous develop- What this study adds ment and testing procedures to ensure that decisions Our study provides a cross-sectional account of phys- made about student performance are also reliable and valid. iotherapists’ opinions related to the CPI and the evalua- tion of students that reinforces previous anecdotal ac- Finally, our findings have important implications for counts. We provide a foundation for the development the profession. More than 50% of respondents said they of a new evaluation instrument for Canadian clinical had heard about the ECP but had never read the docu- education experiences. ment that defines their competence as physiotherapists. This finding suggests that more work may be needed to REFERENCES familiarise and educate physiotherapists about national competency documents and the knowledge, skills, and 1. Baldry Currens JA, Bithell CP. Clinical education: listening to behaviours required of a physiotherapist practising in different perspectives. Physiotherapy. 2000;86(12):645–53. http:// Canada. dx.doi.org/10.1016/S0031-9406(05)61302-8. Two limitations should be considered when interpret- 2. Prescott-Clements L, van der Vleuten CPM, Schuwirth LWT, et al. ing our survey data, related to response rates and the Evidence for validity within workplace assessment: the Longitudinal generalizability of our results and to the use of self- Evaluation of Performance (LEP). Med Educ. 2008;42(5):488–95. report surveys. Our response rate of 17% may be con- http://dx.doi.org/10.1111/j.1365-2923.2007.02965.x. Med- sidered low; however, our sample represents diverse line:18298449 practice areas and practice settings, with respondents from each province and territory in Canada. Although 3. Hall M, Poth C, Manns P, et al. To supervise or not to supervise a our recruitment efforts attempted to include all practis- physical therapy student: a national survey of Canadian physical ing physiotherapists, our sample is likely more represen- therapists. J Phys Ther Educ. 2015;29(3):58–67. tative of supervising physiotherapists, who may be more invested in physiotherapy CE, more familiar with the 4. Roach K, Gandy J, Deusinger SS, et al.; Task Force for the Develop- CPI, and, for both of these reasons, more likely to com- ment of Student Clinical Performance Instruments, American plete the survey. Finally, self-report surveys may be Physical Therapy Association. The development and testing of APTA susceptible to bias in this situation because respondents Clinical Performance Instruments. Phys Ther. 2002;82(4):329–53. may have particularly strong views about the subject Medline:11922850 area. It remains clear that an overwhelming majority of our respondents believe that a new Canadian instrument 5. American Physical Therapy Association. A guide to physical thera- is needed. pist practice, volume I: A description of patient management. Phys Ther. 1995;75(8):707–64. Medline:7644575 6. American Physical Therapy Association. A normative model of physical therapist professional education. Version 1997. Alexandria (VA): American Physical Therapy Association; 1997. 7. National Physiotherapy Advisory Group. Essential competency pro- file for physiotherapists in Canada [Internet]. The Group; 2009 [cited 2015 Mar 16]. Available from: http://www.physiotherapyeducation. ca/Resources/Essential%20Comp%20PT%20Profile%202009.pdf. 8. Murphy S, Dalton M, Dawes D. Assessing physical therapy students’ performance during clinical practice. Physiother Can.

Hall et al. Examining the Need for a New Instrument to Evaluate Canadian Physiotherapy Students during Clinical Education Experiences 155 https://www.utpjournals.press${contentReq.requestUri} - Horizon College Physiotherapy <[email protected]> - Monday, February 22, 2021 11:13:27 PM - IP Address:43.246.243.82 2014;66(2):169–76. http://dx.doi.org/10.3138/ptc.2013-26. 11. Huq SZ, Karras BT. A proposed ontology for online healthcare Medline:24799754 surveys. AMIA Annu Symp Proc. 2003;2003:304–9. Medline:14728183 9. Creaser GA. An exploration of clinical instructors’ experiences and perceptions of the Physical Therapy Clinical Performance Instru- 12. Canadian Institute for Health Information. Physiotherapist work- ment [dissertation]. Halifax (NS): Mount Saint Vincent University; force, 2012. Ottawa: Canadian Institute for Health Information; 2013. 2006. 10. Anderson C, Cosgrove M, Lees D, et al. What clinical instructors 13. Dalton M, Davidson M, Keating J. The Assessment of Physiotherapy want: perspectives on a new assessment tool for students in the Practice (APP) is a valid measure of professional competence of clinical environment. Physiother Can. 2014;66(3):322–8. http:// physiotherapy students: a cross-sectional study with Rasch analysis. dx.doi.org/10.3138/ptc.2013-27. Medline:25125788 J Physiother. 2011;57(4):239–46. http://dx.doi.org/10.1016/S1836- 9553(11)70054-6. Medline:22093122

https://www.utpjournals.press${contentReq.requestUri} - Horizon College Physiotherapy <[email protected]> - Monday, February 22, 2021 11:13:27 PM - IP Address:43.246.243.82 EDUCATION Evidence of Reliability, Validity, and Practicality for the Canadian Physiotherapy Assessment of Clinical Performance Brenda Mori, BScPT, MSc, PhD;*† Kathleen E. Norman, BScPT, PhD;‡ Dina Brooks, BScPT, PhD;* Jodi Herold, BHSc(PT), PhD;§ Dorcas E. Beaton, BScOT, PhD¶**†† ABSTRACT Purpose: To investigate the internal consistency, construct validity, and practicality of the Canadian Physiotherapy Assessment of Clinical Performance (ACP), a descriptive measure used by physiotherapy students and their clinical instructors (CIs) at the mid- and endpoints of an internship to describe the students’ behaviours as observed in the clinical education setting relative to what might be expected of an entry-level physiotherapist. Methods: This multi-centre study piloted the ACP in 10 university physiotherapy (PT) programmes. Both CIs and students undertaking clinical internships completed the ACP and the current tool, the Physical Therapist Clinical Performance Instrument (PT-CPI; Version 1997). Results: CIs assessing PT students’ performance during internships representing a variety of areas of practice completed the ACP at the midpoint (n ¼ 132) and the endpoint (n ¼ 126) of the internship. The end-of-internship sample consisted of 55 junior, 30 intermediate, and 41 senior students. The ACP demonstrated strong internal consistency: Alpha coefficients for each role ranged from 0.94 to 0.99. Aligned items on the ACP and PT-CPI were significantly correlated (r ¼ 0.51–0.84). Senior PT students performed significantly better than intermediate students, who, in turn, performed better than junior students (p < 0.0001). Effect sizes for midpoint to final scores on the ACP ranged from medium to large (0.40–0.74). Participants were satisfied with the online education module that provided instruction on how to use and interpret the ACP, as indicated by satisfaction scores and qualitative comments. Conclusions: The ACP is a reliable, valid, and practical measure to assess and describe the PT students’ behaviours as observed during clinical education relative to what is expected of an entry- level physiotherapist. Key Words: educational measurement; internship and residency; students. RE´ SUME´ Objectif : Examiner la cohe´ rence interne, la validite´ de construit et l’utilite´ pratique de l’e´ valuation de la performance clinique (EPC) en physiothe´ rapie au Canada, une mesure employe´ e par les e´ tudiants en physiothe´ rapie et leurs instructeurs cliniques en milieu et en fin de stage clinique pour de´ crire les comportements des e´ tudiants par rapport a` ce qu’il y a lieu d’eˆ tre attendu d’un physiothe´ rapeute de´ butant. Me´ thodes : Cette e´ tude multicentrique e´ valuait la mise a` l’essai de l’EPC dans 10 programmes universitaires. Les instructeurs cliniques et les e´ tudiants en stage clinique ont re´ alise´ leurs e´ valuations a` l’aide de l’EPC et de l’outil actuel, le PT-CPI (version 1997). Re´ sultats : Des instructeurs cliniques repre´ sentant diffe´ rents champs de pratique ont comple´ te´ l’ECP en milieu (n ¼ 132) et en fin (n ¼ 126) de stage. L’e´ chantillon de fin de stage e´ tait compose´ de 55 e´ tudiants de´ butants, 30 e´ tudiants interme´ diaires et 41 e´ tudiants avance´ s. L’ECP a de´ montre´ un haut degre´ de cohe´ rence interne, soit des coefficients alpha de 0,94 a` 0,99 pour chaque roˆ le. Une corre´ lation significative a e´ te´ observe´ e entre les e´ le´ ments correspondants de l’ECP et du PT-CPI (r ¼ 0.51–0.84). Les e´ tudiants avance´ s ont obtenu des re´ sultats significativement meilleurs que les e´ tudiants interme´ diaires qui ont a` leur tour obtenu des re´ sultats significativement meilleurs que les e´ tudiants de´ butants (p < 0.0001). Les diffe´ rences constate´ es entre les notes de l’ECP en milieu et en fin de stage re´ ve` lent une taille d’effet moyenne a` forte (0,40 a` 0,74). Les notes de satisfaction et les commentaires qualitatifs indiquent que les participants e´ taient satisfaits du module de formation en ligne qui expliquait comment utiliser et interpre´ ter l’ACP. Conclusions : L’ECP est une mesure fiable, valide et pratique pour e´ valuer et de´ crire les comportements des e´ tudiants observe´ s en formation clinique par rapport a` ce qu’il y a lieu d’attendre d’un physiothe´ rapeute de´ butant. From the: *Department of Physical Therapy; §Postgraduate Medical Education; ¶Department of Occupational Science & Occupational Therapy, Faculty of Medicine, University of Toronto; †Centre for Faculty Development, Faculty of Medicine, University of Toronto, at the Li Ka Shing International Healthcare Education Centre of St. Michael’s Hospital; **Musculoskeletal Health and Outcomes Research, Li Ka Shing Knowledge Institute, St Michael’s Hospital; ††Institute for Work and Health, Toronto; ‡Physical Therapy Program, School of Rehabilitation Therapy, Queen’s University, Kingston, Ont. Correspondence to: Brenda Mori, Department of Physical Therapy, Faculty of Medicine, University of Toronto, 160–500 University Ave., Toronto, ON M5G 1V7; [email protected]. Contributors: All authors designed the study; collected, analyzed, and interpreted the data; drafted or critically revised the article; and approved the final draft. Competing interests: None declared. Acknowledgements: Use of the rating scale anchors and descriptors adapted and revised from the PT CPI | Web (Alexandria, VA: American Physical Therapy Association, 2006) is by nonexclusive license from the American Physical Therapy Association. Physiotherapy Canada 2016; 68(2);156–169; doi:10.3138/ptc.2014-43E 156

Mori et al. Evidence of Reliability, Validity, and Practicality for the Canadian Physiotherapy Assessment of Clinical Performance 157 https://www.utpjournals.press${contentReq.requestUri} - Horizon College Physiotherapy <[email protected]> - Monday, February 22, 2021 11:13:27 PM - IP Address:43.246.243.82 Assessment is an important component in the clinical ship in the curriculum. The goal of the measure is to education process, providing feedback to the learner to describe the level of performance relative to the standard reinforce areas of strength and identify areas for im- rather than to assess the magnitude of change; it is provement. Formal assessment encourages observation therefore intended to be used as a discriminative mea- of the student’s performance by the clinical instructor sure,5 with repeated descriptions at different points in (CI) and allows the student and CI to discuss the stu- time along the student’s clinical education curriculum. dent’s performance. Because the Clinical Education Guidelines for Cana- Typical development stages for a new measure in- dian University Programs specifies that each student clude item selection, item reduction, development, pre- must acquire significant clinical experience in a variety testing, and testing.1 We have described the initial of practice areas and settings,6 the ACP needed to be phases of development for the Canadian Physiotherapy useful in a variety of clinical education internships across Assessment of Clinical Performance (ACP) in two pre- Canada, ranging from acute care to community care and vious articles.2,3 The first of these articles2 described covering all areas of clinical practice (e.g., musculoskele- Phases 1 and 2. In Phase 1, we recruited an expert con- tal, cardiorespiratory, neurological). The scores on the sultant panel to participate in a study using the Delphi measure (competency and role) are out of 10, and an approach to gain consensus on the rating scale, the entry-level physiotherapist is expected to perform at a items that would make up the measure, and the number minimum of 9 of 10 on all key competencies. A list of and placement of the comment boxes. However, because items from the ACP and a sample page are included this panel was so intimately engaged with the concepts in Appendix 1. The purpose of this study was to assess of the Essential Competency Profile for Physiotherapists and report on the overall performance of the ACP— in Canada4 (ECP) and with clinical education and assess- specifically, its internal consistency, construct validity, ment, we also sought face and content validity from and practicality when completed by CIs—as detailed in future users of the measure in Phase 2, through inter- our a priori table of hypotheses (see Box 1). views with a variety of stakeholders including CIs, recent graduates, and experts in clinical education and mea- Two popular paradigms are used to assess validity: a surement. Phases 1 and 2 produced a first draft of the health measurement7 (often called a psychometric or assessment measure that included a total of 16 rating classic) approach and an educationalist8 (often called a scales, assessing seven roles, and nine comment boxes. modern) approach. Although the techniques used to Using this draft, Phase 33 investigated the face and con- establish test–retest reliability or correlation with another tent validity of the draft measure through broad con- construct are the same, the approach to validity is differ- sultation with physiotherapists across Canada who had ent: Whereas the health measurement approach typically supervised and assessed at least one student in clinical emphasizes amassing an adequate volume of data to practice in the previous 12 months. The results of Phase validate an instrument for a specific application,9 the 3 were used to develop the version of the ACP that would educationalist approach, often linked to Messick’s10 uni- be field tested. fied theory of validity, focuses on using data to support the validity of inferences about the measure. Our study The ACP is an assessment measure based on the ECP. takes a principally health measurement approach to The ECP is a national document that validity. Our intent was to build evidence that could sup- port the educationalist approach to validity, but to report describes the essential competencies (i.e., the knowledge, this evidence in a language and format familiar to the skills and attitudes) required by physiotherapists in health measurement field and, most important, to the Canada at the beginning of and throughout their career. end users of this information and our key stakeholders: It also provides guidance for physiotherapists to build clinicians measuring the competence of their students on their competencies over time. The Profile reflects the in clinical education settings. diversity of physiotherapy practice and helps support evolution of the profession in relation to the changing METHODS nature of practice environments and advances in evidence- informed practice.4(p.4) Participants A total of 10 Canadian PT university programmes The ACP is to be used by both physiotherapy (PT) stu- dents and their CIs to assess and describe the students’ agreed to participate in this prospective study assessing clinical performance behaviours as observed in the clini- the psychometrics of the ACP, which took place from cal education setting, relative to what is expected of an March through December 2013. Ethics approval was entry-level physiotherapist, across the student’s academic granted either by the programme’s internal review pro- programme. The ACP assesses seven PT roles (domains cess (3 programmes) or by the university’s ethics office and subscores), each with 1 to 8 competencies, for a total (7 programmes). We recruited both PT students and CIs of 21 competencies; it is scored as a profile across roles. to participate in the study. All students completing an The ACP is administered twice during each clinical intern- internship at a participating university during the study

158 Physiotherapy Canada, Volume 68, Number 2 Box 1 A Priori Hypotheses and Planned Analyses A priori hypothesis (and null hypothesis) Analysis https://www.utpjournals.press${contentReq.requestUri} - Horizon College Physiotherapy <[email protected]> - Monday, February 22, 2021 11:13:27 PM - IP Address:43.246.243.82 Internal consistency The ACP measure will have strong internal consistency. Cronbach’s a coefficient for each role Construct validity (Ho: internal consistency a0.90) Item-total correlation Effect sizes Practicality Students’ scores at the final point of the internship will be different (higher) than the midpoint scores. Items within the 1.0 Role of Expert will have the greatest Graph representing average scores per item effect size. Communication and professionalism will have smaller effect sizes. at midpoint and final point per stage ANOVA calculations (Ho: There will be no difference between midpoint and final point scores) Senior students will perform better than intermediate students who will perform Pearson correlations better than junior students. Spearman correlations of the ACP However, there may be exceptions to this because each clinical internship and PT-CPI total score usually represents a new practice area for a student; therefore, the profiles may Scatterplot graph of total score rank not follow a linear pattern. Scatterplot graph of total score Graphing the ACP scores for each student’s (Ho: There will be no difference among students at different stages.) internship for the cohort of 14 students who Specific ACP items will be correlated with specific PT-CPI items hypothesized appear more than once in the data set. as very strong, strong, moderate, or weak. Hypothesized correlations were established by 4 NACEP members. Descriptive statistics of the satisfaction questions at the end of the ACP. (Ho: There will be no or low [r < 0.3] correlation between ACP and CPI items.) Students who performed well or poorly on the PT-CPI will also perform well or Descriptive statistics completed at the end poorly on the ACP. of the education module (Ho: There will be no or low [r < 0.3] correlation between ACP and CPI total score ranks.) Progression of scores over internships for a person: The ACP will show higher scores over each sequential assessment within a student over multiple intern- ships, showing a logical progression of scores. ACP scores for internships that occur later in the curriculum will be higher than scores for earlier internships; however, there may be exceptions to this because each clinical internship usually represents a new practice area for a student; therefore, the profiles may not follow a linear pattern. CIs will report that they are able to use the ACP to assess their students and have insightful comments regarding things they liked about it and suggestions for improvement. (Ho: There will be no or negative opinions of the ACP.) The online education module will prepare CIs to complete the ACP. (Ho: Participants will report that they were not prepared to complete the ACP) ACP ¼ Canadian Physiotherapy Assessment of Clinical Performance; ANOVA ¼ analysis of variance; PT-CPI ¼ Physical Therapist Clinical Performance Instrument; NACEP ¼ National Association for Clinical Education in Physiotherapy; CI ¼ clinical instructor. period were invited to participate; those who consented Data collection to do so could choose to participate by (1) making their Students and their CIs were asked to participate by (1) clinical internship assessment forms available for the research study or (2) allowing their data to be used for completing the online education module for the ACP; (2) the research and completing both the Physical Therapist providing information about the internship, including Clinical Performance Instrument (PT-CPI)11 and the ACP. area of practice; (3) completing first the current assess- Once a student had given consent, we contacted the stu- ment tool used in clinical education (the PT-CPI) and dent’s CIs to invite them to participate. Participants were then the ACP at both the midpoint and the final point assigned a unique identifier to ensure that the data of the internship; and (4) answering additional validity entered and analyzed were anonymous. These identifiers and feasibility questions included as a component of also allowed us to link student–CI pairs during an intern- the ACP for the pilot. The ACP was available in English ship, identify students’ university, and follow students and French; participants were encouraged to complete and CIs throughout the study. The ACP was completed it online, but a paper copy was also available. by the same CIs at the midpoint of the internship and at the final point. This article focuses on the ACPs As part of the ACP, CIs rate students and students completed by CIs, as opposed to those completed by self-assess on each of the 21 ACP key competencies on a students. 10-point scale (Box 2) with six defined anchors ranging from ‘‘beginner’’ (1) to ‘‘with distinction’’ (10). From these, we calculated a role score for each of the seven roles

Mori et al. Evidence of Reliability, Validity, and Practicality for the Canadian Physiotherapy Assessment of Clinical Performance 159 Box 2 Coding of the ACP Rating Scale https://www.utpjournals.press${contentReq.requestUri} - Horizon College Physiotherapy <[email protected]> - Monday, February 22, 2021 11:13:27 PM - IP Address:43.246.243.82 Beginner Advanced Intermediate Advanced Entry-level With Beginner Intermediate Distinction bb b b b b b b b b Each level was coded as: 12 3 4 5 6 7 8 9 10 on the ACP. We also calculated a total score to allow ence between midpoint and final means over the pooled comparison between the ACP and the PT-CPI. For the standard deviation. Effect sizes are considered to be purposes of this study, we measured each completed small in the range of 0.2 and large in the range of 0.8.15 visual analogue scale on the PT-CPI, then divided the We anticipated small to medium effect sizes (0.2–0.5) midpoint and final measures by the total line length to for items in the Communication and Professional roles obtain a measure of the PT-CPI mid- and endpoint and medium to large effect sizes (0.5–0.8) for items in scores. the Expert role. In collaboration with each academic coordinator or Practicality director of clinical education, we categorized each intern- We used descriptive statistics to explore the useful- ship as junior, intermediate, or senior. In most cases, the first two internships of the curriculum were classified as ness of the online education module in preparing partic- junior internships, the last two as senior, and those in ipants to use the ACP, as well as the ease of use of the between as intermediate. ACP. Data analyses All data were analyzed using SAS (version 9.3; SAS In- In this study, we focused on internal consistency, stitute, Cary, NC). In cases in which data were missing, statistical analyses were performed with the existing construct validity, and practicality of the ACP completed data; when a value was missing for an item in an ACP by the CI. In designing the study, we reviewed and con- role, the denominator for that role was adjusted accord- sidered standards for methodological quality (COnsensus- ingly so as to not penalise the student for the missing based Standards for the selection of health Measurement data. For example, if the Manager role had scores for Instruments, or COSMIN12) for the internal consistency items 4.1 and 4.3 but was missing a score for item 4.2, and validity of a measure. The COSMIN is a common the scores for items 4.1 and 4.3 were added, then divided appraisal tool that identifies standards for evaluating by 20 (10 per item scored) rather than 30. Total ACP the methodological quality of studies and can be used scores were calculated using the same approach. When as a guide for designing or reporting on measurement calculating a role score, we tolerated up to 50% missing properties.12 data for that role; if more than 50% of items were miss- ing (e.g., two item scores missing for a three-item role), Internal consistency no role score was calculated. Cronbach’s a coefficient is an index of reliability that RESULTS explores the variation in the construct being assessed. An acceptable level of Cronbach’s a frequently cited Sample description in the literature is between 0.713 and 0.9014 for scores We analyzed 121 ACPs completed at both mid- and meant to be applied at an individual level (such as our tool). For each multi-item role, we calculated inter-item final points. In addition, we included 11 ACPs with only correlations and used Cronbach’s a to summarize the midpoint data and 5 ACPs with only final-point data (132 degree to which responses to the items in a given role ACPs at midpoint and 126 at the final point, respectively) score were consistent. We also calculated Cronbach’s a for a total of 137 ACPs. The sample of final ACPs came with each item removed. from 55 junior, 30 intermediate, and 41 senior students. One ACP (midpoint and final) was completed in French, Construct validity and the remainder were completed in English; 87% were As is preferred for good studies of construct validity, completed online, and the rest were completed on paper. All 10 participating universities were represented we set up several a priori hypotheses and planned statis- in the sample. tical analyses for each hypothesis (see Box 1). Our analyses focused more on the profile of scores across the roles Of the 137 ACP entries that constituted our final data and stages than on the amount of change in the scores. set for analysis, we had a total of 129 matched PT-CPIs We expected the profile of midpoint ACP scores to be (124 matched ACP and PT-CPI pairs at midpoint, and different from the profile of final ACP scores; therefore, 120 matched ACP and PT-CPI pairs at the final point). we calculated effect sizes for each item using the differ- Some ACPs did not have a matched PT-CPI because no

160 Physiotherapy Canada, Volume 68, Number 2 Table 1 Practice Area Characteristics for Internships with Completed ACPs No. of respondents https://www.utpjournals.press${contentReq.requestUri} - Horizon College Physiotherapy <[email protected]> - Monday, February 22, 2021 11:13:27 PM - IP Address:43.246.243.82 % of total responses Characteristics Junior Intermediate Senior Total in the category Continuum of care Acute care 36 13 15 64 39 21 Private practice 11 10 13 34 19 7 Rehabilitation facility 11 7 13 31 5 2 Long-term care 10 – 1 11 2 2 Community health centre 6 1 18 1 2 Community care 2 – 24 27 Hospital outpatients 2– 24 19 15 Insurance 1 – 23 15 9 Primary health care – 2 –2 6 2 Other* 2 1 14 1 1 Area of practice† 2 3 Musculoskeletal 30 13 18 61 49 Mixed practice 24 6 13 43 22 21 Neuroscience 11 11 13 35 8 Cardiorespiratory 21 6 7 34 71 17 General rehabilitation 15 4 2 21 9 3 General medicine 5 3 5 13 Oncology 3 – 14 Critical care – 2 13 Pain 3 – – 3 Other specialty‡ 1 – 45 Other area§ 3 3 17 Patient age Mixed ages 40 14 28 82 66þ y 17 11 8 36 19–65 y 15 8 12 35 0–18 y 5 4 4 13 Population density Urban 41 23 39 103 Rural 13 7 4 24 Semirural 10 1 2 13 Remote 2 2 15 *Hospital with acute/rehab mix; paediatrics (2 responses); private not-for-profit school (1 response). † Percentages total more than 100 because respondents could have more than one area of practice. ‡Amputees, women’s health, workplace injuries, burns, rheumatology. §Paediatrics (5 responses); rural and remote (2 responses). ACP ¼ Canadian Physiotherapy Assessment of Clinical Performance. PT-CPI was completed, the PT-CPI was not returned for in the missing data rate for item 4.2 (junior, 18%; inter- analysis, or the PT-CPI was incomplete. mediate, 17%; senior, 17%), but at midpoint, more intermediate students had missing data (junior, 19%; Table 1 lists the characteristics (continuum of care, intermediate, 31%; senior, 18%). We found no indica- areas of practice, age groups, and settings) of the con- tions of floor or ceiling effects in the overall data, except texts in which the internships took place. for item 4.2 at the midpoint (17.3%). Competency (item-level) results When analyzed separately, junior students’ scores were Descriptive statistics for the ACP items at the mid- moderately positively skewed, and senior students’ scores were moderately or highly negatively skewed. That is, and final points are presented in Table 2. The ACP data junior students had a higher frequency of scores at the were normally distributed at the final point, with the lower end of the rating scale (beginner performance), exception of items 2.1, 2.2, 3.1, 3.2, 4.3, 6.1, 7.1, 7.2, and and senior students had a higher frequency of scores at 7.3, which were moderately negatively skewed (i.e., the the upper end of the rating scale (entry-level performance). left tail was longer, with a greater concentration of scores on the right side of the distribution). Item 4.2 (supervis- Role score results ing support personnel) had the highest amount of missing Descriptive statistics for ACP role scores at the mid- data (17.5%) at the final point. There was no apparent relationship between missing data and stage of training. and final points, for all data and for each internship At the final point, the internship stage was not a factor

://www.utpjournals.press${contentReq.requestUri} - Horizon College Physiotherapy <[email protected]> - Monday, February 22, 2021 11:13:27 PM - IP Address:43.246.24 Table 2 Overall ACP by CIs: Item-Level Descriptive Statistics at Mid- and Final Points Expert Communica 1.1 1.2 1.3 1.4 1.5 1.6 1.7 1.8 2.1 2.2 ACP CI midpoint data (n ¼ 132) Mean (SD) 5.3 4.9 4.9 4.6 4.8 5.3 5.0 5.0 6.3 5.8 (2.2) (2.2) (2.2) (2.3) (2.3) (2.5) (2.3) (2.4) (2.2) (2.2) 1–9 1 – 10 1–9 1 – 10 1 – 10 1 – 10 1 – 10 1 – 10 12 16 Range 1–9 1–9 8 14 8 5 8 10 2 5 14 19 18 13 14 14 6 7 FC 1 46 23 17 20 19 20 17 17 19 10 20 14 16 16 14 4 8 FC 2 9 13 22 16 13 15 18 17 20 19 17 18 26 17 16 13 10 11 FC 3 21 24 22 10 18 21 18 15 27 27 12 16 14 10 16 16 FC 4 17 13 3 9 25 18 4 1 6 9 7 1 5 2 FC 5 20 21 0 1.5 0 1 1 3.8 0.0 0.0 0.0 0.09 0.0 0.0 0.0 À0.01 À0.40 À0.29 FC 6 16 16 À0.05 À0.89 0.02 À0.05 0.06 À1.10 À0.88 À0.98 À1.06 À1.05 À1.07 À1.01 FC 7 21 20 FC 8 12 12 FC 9 11 6 FC 10 00 % missing values 0.8 0.8 Skewness 0.01 0.04 Kurtosis À0.96 À1.02 ACP CI final-point data (n ¼ 126) Mean (SD) 6.7 6.5 6.5 6.3 6.4 6.8 6.6 6.5 7.5 7.0 (2.2) (2.2) (2.2) (2.2) (2.2) (2.4) (2.1) (2.1) (2.1) (2.2) 2 – 10 2 – 10 2 – 10 2 – 10 2 – 10 2 – 10 2 – 10 2 – 10 Range 2–10 2–10 0 0 0 0 0 0 0 0 3 6 6 1 3 4 1 2 FC 1 00 17 16 13 14 13 15 8 8 6 5 10 7 11 12 7 6 FC 2 15 14 17 15 15 9 14 6 18 15 11 13 13 18 11 11 11 FC 3 12 11 26 24 24 16 21 16 19 17 13 18 16 23 17 19 16 23 FC 4 14 10 24 22 24 28 24 21 43 30 6 2 4 8 8 11 15 10 FC 5 7 15 1.6 4.0 0.8 0.8 1.6 2.4 0.0 0.8 À0.32 À0.39 À0.34 À0.45 À0.36 À0.24 À0.89 À0.55 FC 6 18 13 À0.95 À0.99 À0.97 À0.95 À0.92 À1.17 À0.17 À0.68 FC 7 23 19 FC 8 15 20 FC 9 30 28 FC 10 53 % missing 0.8 1.6 Skewness À0.39 À0.44 Kurtosis À0.97 À0.93 Note: Bolded items indicate moderate skewness; bold italic items ¼ missing data >5%. * Indicates a floor effect where 17.3% were scored at 1. ACP ¼ Canadian Physiotherapy Assessment of Clinical Performance; CI ¼ clinical instructor; M ¼ midpoint data;

ator Collaborator Manager Advocate Scholarly Professional Mori et al. Evidence of Reliability, Validity, and Practicality for the Canadian Physiotherapy Assessment of Clinical Performance 4.2 5.1 practitioner 2.3 3.1 3.2 4.1 4.3 7.1 7.2 7.3 6.1 – 3 M CI 5.6 5.5 5.5 5.2 4.7 5.6 4.9 5.8 6.5 6.8 5.8 (2.4) (2.4) (2.5) (2.5) (2.6) (2.4) (2.5) (2.4) (2.3) (2.3) (2.7) 1 – 10 1 – 10 1 – 10 1 – 10 1 – 10 1 – 10 1 – 10 1 – 10 1 – 10 1 – 10 1 – 10 18* 14 5 6 9 8 3 12 6 2 1 9 7 12 11 14 8 11 17 7 4 4 7 20 18 12 20 15 22 10 14 14 11 17 11 10 15 12 21 8 9 6 4 22 8 21 18 12 9 25 16 22 21 10 20 16 10 19 9 12 8 5 5 23 14 17 14 4 13 22 23 26 21 17 18 24 14 19 14 20 15 14 11 12 17 13 16 17 13 15 13 17 38 43 23 3 12 19 6 4 3 6 3 0.0 2 1 5 1 1.5 0.8 0.8 6.8 À0.14 2 2.3 0.0 1 1 3.8 À0.24 À0.51 À0.58 À0.30 À0.99 0.0 À0.19 0.05 21.2 1.5 À0.06 À0.82 À0.88 À0.82 À1.15 À0.17 À0.99 À1.21 0.12 À0.07 À1.14 À1.03 À1.27 À1.18 F CI 7.0 7.0 6.7 6.7 6.1 7.0 6.2 7.2 7.4 7.8 7.1 (2.1) (2.2) (2.4) (2.4) (2.7) (2.2) (2.5) (2.1) (2.1) (1.9) (2.4) 2 – 10 2 – 10 1 – 10 2 – 10 1 – 10 2 – 10 1 – 10 2 – 10 2 – 10 3 – 10 1 – 10 0 0 2 0 7 0 3 0 0 0 1 1 2 4 5 6 3 9 1 1 0 3 11 12 16 12 12 8 12 9 9 6 14 6 9 3 10 3 8 8 4 2 4 2 14 11 15 13 12 15 15 15 16 9 13 13 11 10 16 11 11 16 12 6 8 7 25 18 16 19 15 21 20 18 19 19 16 13 20 16 10 6 13 11 17 11 8 14 31 33 35 28 27 34 26 31 49 53 40 12 10 7 13 5 9 5 15 10 15 10 0.0 0.0 1.6 0.0 17.5 3.2 0.8 3.2 2.4 3.2 4.8 À0.48 À0.57 À0.57 À0.32 À0.41 À0.53 À0.34 À0.58 À0.85 À1.02 À0.75 À0.76 À0.84 À0.86 À1.03 À1.04 À0.77 À0.94 À0.66 À0.36 0.08 À0.64 ; FC ¼ frequency count; F ¼ final point data. 161

162 Physiotherapy Canada, Volume 68, Number 2 Table 3 ACP by CIs: Role Score Descriptive Statistics at Mid- and Final Points 1– 2– 3– 4– 5– 6– 7– Advocate Scholarly practitioner Professional https://www.utpjournals.press${contentReq.requestUri} - Horizon College Physiotherapy <[email protected]> - Monday, February 22, 2021 11:13:27 PM - IP Address:43.246.243.82 Role Expert Communicator Collaborator Manager 1 1 3 ACP by CI midpoint data (n ¼ 132; JR ¼ 59; INT ¼ 29; SR ¼ 44) NA NA 0.93 4.9 (2.5) 5.8 (2.4) 6.4 (2.3) No. of items 8 3 2 3 1–10 1–10 1–10 0.96 3.9 1.5 0.8 Cronbach’s a 0.99 0.95 0.95 5.2 (2.4) À0.06 À0.24 À0.46 1–10 À1.14 À0.82 À0.88 Mean (SD) 5.0 (2.2) 5.9 (2.2) 5.5 (2.4) 0.8 0.03 1 1 3 Range 1–9.4 1–10 1–10 À1.25 NA NA 0.95 6.2 (2.5) 7.2 (2.1) 7.4 (2.0) % missing values 0.0 0.0 0.0 1–10 2–10 2.7–10 0.8 3.3 2.4 Overall skewness 0.01 À0.26 À0.20 À0.34 À0.58 À0.85 À0.94 À0.66 À0.34 Overall kurtosis À1.03 À0.98 À1.01 ACP by CI final point data (n ¼ 126; JR ¼ 55; INT ¼ 30; SR ¼ 41) No. of items 8 3 2 3 0.94 Cronbach’s a 0.99 0.95 0.96 6.6 (2.3) 2–10 Mean (SD) 6.5 (2.1) 7.2 (2.0) 6.8 (2.3) 1.6 À0.37 Range 2.3–10 2.3–10 1.5–10 À0.98 % missing values 0.8 0.0 0.0 Overall skewness À0.38 À0.65 À0.59 Overall kurtosis À1.03 À0.53 À0.85 ACP ¼ Canadian Physiotherapy Assessment of Clinical Performance; CI ¼ clinical investigator; JR ¼ junior internship; INT ¼ intermediate internship; SR ¼ senior internship; NA ¼ not applicable. Table 4 ACP by CIs: Effect Sizes of Mid- to Final Point Internship Data (7.45), Scholarly Practitioner (7.24), and Communicator (7.19); the lowest was Expert (6.53). Mean role scores Item Item description Effect size at the end of an internship ranged from 4.45 to 6.00 for junior students; from 6.63 to 8.28 for intermediate stu- 1.4 Expert – diagnosis and prognosis 0.74 dents; and from 8.20 to 8.86 for senior students. 1.2 Expert – objective assessment 0.73 1.3 Expert – analysis 0.73 Internal consistency 1.7 Expert – intervention effectiveness 0.73 At the final point, Cronbach’s a correlation coefficients 1.5 Expert – intervention strategy 0.70 1.6 Expert – implements intervention 0.70 were as follows: Expert, 0.99; Communicator, 0.95; Col- 1.1 Expert – subjective assessment 0.66 laborator, 0.96; Manager, 0.94; and Professional, 0.95. 3.1 Collaborator – inter-professional relationships 0.65 Alpha coefficients were not calculated for the Advocate 1.8 Expert – completes PT services 0.63 or Scholarly Practitioner roles because each contains 6.1, 6.2, 6.3 Scholarly practitioner 0.62 only 1 item. Deleting an item from the Cronbach’s a 4.1 Manager – individual practice 0.62 analysis minimally changed the a coefficient, which indi- 2.3 Communicator – effective 0.61 cates that items on the ACP are consistent in assessing 4.3 Manager – safe and effective 0.61 the same construct. Correlations of item scores to total 4.2 Manager – supervises personnel 0.56 role score were more than 0.86. 2.2 Communicator – information 0.55 2.1 Communicator – builds rapport 0.55 Validity 5.1 Advocate 0.51 To explore the ACP’s construct validity, we examined 7.3 Professional – development of PT profession 0.49 3.2 Collaborator – manages conflict 0.48 several tenets with a priori hypotheses. We expected that 7.2 Professional – respects autonomy of client 0.45 final ACP scores would be higher than midpoint scores 7.1 Professional – legal/ethical requirements 0.40 and that the difference would be greatest for items in the Expert role and least for items in the Communicator Note: ACP data (ACP midpoint, n ¼ 132; ACP final point, n ¼ 126). and Professional roles. Effect sizes ranged from 0.40 ACP ¼ Canadian Physiotherapy Assessment of Clinical Performance; for ACP item 7.1 (conducts self within legal/ethical CI ¼ clinical instructor; PT ¼ physiotherapy. requirements) to 0.74 for ACP item 1.4 (establishes a PT diagnosis and prognosis; see Table 4). The pattern of stage, are presented in Table 3. Missing role scores effect sizes was similar for the ACP and the PT-CPI: On were minimal. Overall final role scores were normally the PT-CPI, the lowest effect size, 0.28, was for item 5 distributed for Expert, Manager, and Advocate and (legal practice), and the largest effect size, 0.74 was for moderately negatively skewed for Communicator, Col- PT-CPI item 11 (establishing a diagnosis). laborator, Scholarly Practitioner, and Professional. The highest three mean final role scores were Professional We expected senior students to perform differently

Mori et al. Evidence of Reliability, Validity, and Practicality for the Canadian Physiotherapy Assessment of Clinical Performance 163 https://www.utpjournals.press${contentReq.requestUri} - Horizon College Physiotherapy <[email protected]> - Monday, February 22, 2021 11:13:27 PM - IP Address:43.246.243.82 Figure 1 ACP mean scores at mid- and final point by internship stage. Note: Symbol depicted in lighter shade represents midpoint ACP mean; darker shade represents final point ACP mean ACP ¼ Canadian Physiotherapy Assessment of Clinical Performance. than intermediate students and junior students. Figure 1 and PT-CPI total scores were also very high (Pearson’s shows the average midpoint and final scores for each r ¼ 0.85, 95% CI: 0.79, 0.89; Spearman’s r ¼ 0.89, 95% stage. Analysis of variance calculations by internship CI: 0.85, 0.92), which also supports the ACP’s validity. stage for each ACP item, as well as for ACP and PT-CPI total scores, showed significant differences at p < 0.0001. Practicality Post hoc analysis using a Tukey test revealed that all The 205 CIs and students who completed the ACP differences between senior and junior students, as well as those between intermediate and junior students, online education module evaluation indicated that the were significant at p < 0.05 for each item, but this was module took an average of 28.8 (SD 6.9) minutes to com- not the case for differences between senior and inter- plete. More than 92% agreed or strongly agreed that the mediate students for items in the Communicator, Col- module adequately prepared them to complete the ACP, laborator, and Professional roles or for items 4.1 and 4.3 was useful, and was easy to navigate. Suggested improve- in the Manager role. ments to the online education module included a pause button for the audio feed and a shorter version. The highest correlations at the end of the internship were between the ACP Expert role and the related PT- Figure 2 summarizes CIs’ responses to the feasibility CPI items (9, 10, 11, 12, 13, and 14), which supports the and satisfaction questions regarding the experience of ACP’s construct validity. Correlations between final ACP using the ACP. Mean completion time for the ACP was

164 Physiotherapy Canada, Volume 68, Number 2 https://www.utpjournals.press${contentReq.requestUri} - Horizon College Physiotherapy <[email protected]> - Monday, February 22, 2021 11:13:27 PM - IP Address:43.246.243.82 Figure 2 ACP feasibility and satisfaction scores completed by CIs (n ¼ 156). Note: 6, ‘‘The training module adequately prepared me to complete the ACP’’; 5, ‘‘The Enabling Competencies (i.e., items noted as 1.2.3 or 4.3.1) helped me to understand what the item was assessing’’; 4, ‘‘I understand what each item was asking me to assess’’; 3, ‘‘I was able to discriminate my student’s performance to grade him/her on each item with a rating scale’’; 2, ‘‘The rating scale was easy to use’’; 1, ‘‘The anchor descriptors for the rating scale were adequately explained so that I could easily rate the student’s performance.’’ ACP ¼ Canadian Physiotherapy Assessment of Clinical Performance; CIs ¼ clinical instructors. 48.8 (SD 22.87) minutes. In response to the open-ended in Canada. Areas of practice represented in our sample questions, CIs reported that they liked that the ACP were very similar to those identified in a population seemed shorter, took less time to complete, and focused study of Ontario clinical education placements,16 except on a Canadian context; that the categories were easy to that our data set included fewer internships in the interpret; that the tool had a discrete rating scale; and musculoskeletal area. Our data set was also similar to that it was accessible online. Suggested improvements the numbers given in the Canadian Institute for Health included a ‘‘not observed’’ option for items, a simpler Information’s Physiotherapists in Canada, 2010 report,17 way to save and return to an ACP not completed in one which found that musculoskeletal and general practice initial sitting, a spell-check option, and a simpler way to were the most common practice areas and general view the rating scale in a pop-up window when using the hospital and private practice were the most common scale. Participants reported some challenges with apply- places of employment. ing the rating scale anchor descriptors to some items, such as ACP item 5.1. Final ACP data were normally distributed, with the exception of some items in the Communicator (items DISCUSSION 2.1 and 2.2), Collaborator (items 3.1 and 3.2), Manager The purpose of this study was to assess and report on (item 4.3), and Professional (items 7.1, 7.2, and 7.3) roles, which were negatively skewed. These findings echo those the overall performance of the ACP. The ACP demon- of Proctor and colleagues’18 7-year longitudinal analysis strated evidence of high internal consistency and initial of the PT-CPI, published in 2010, in which students evidence of good construct validity; it is seen by users as scored higher on PT-CPI items related to communica- a practical measure to assess and describe PT students’ tion and professionalism. Higher scores on items assess- behaviours as observed during clinical education. ing communication and professionalism may indicate that university PT programmes are admitting students Our sample appeared mostly to be reflective of clini- cal education internships in Ontario and of PT practice

Mori et al. Evidence of Reliability, Validity, and Practicality for the Canadian Physiotherapy Assessment of Clinical Performance 165 https://www.utpjournals.press${contentReq.requestUri} - Horizon College Physiotherapy <[email protected]> - Monday, February 22, 2021 11:13:27 PM - IP Address:43.246.243.82 who already possess strengths in this area and therefore The ACP was able to differentiate between junior, have less potential to show change throughout the intermediate, and senior students, showing evidence of programme. strong known-groups validity. Post hoc analysis showed that for some items in the Communicator, Collaborator, Although the rate of missing data was minimal, item Manager, and Professional roles, no difference was found 4.2 (supervising support personnel) had the highest rate between intermediate and senior students. Typically, of missing data. This may have been because support these were items for which all ratings tended to be personnel were not available in all clinical environments higher than for other items; ratings on these items were (a frequent comment in the comment box for this role) high not only at the midpoint of intermediate and senior or because clinicians encouraged students to complete internships but also of junior internships. A study by all components of the treatment programme with the Norman and Booth19 similarly found, after analyzing patient themselves rather than assigning care to support 1,460 PT-CPIs, that students were most likely to receive personnel. Interestingly, the percentage of missing values credit with distinction on items relating to professional for item 4.2 is lower for final ACPs (17.5%) than for mid- behaviour (items 2 and 3) and communication (item 6). point ACPs (21.1%), which may indicate that specific Students in Canadian PT programmes may be selected opportunities to work with support personnel are tar- on the basis of strengths in communication and profes- geted for the latter half of the internship. The ACP had sionalism and thus have less opportunity for demonstra- a higher completion rate than the PT-CPI for this com- ble growth in these areas than in other PT competencies parable item, which may indicate that the ACP is more such as assessment skills, analysis, and planning and relevant to PT practice. The ACP’s relevance to Canadian delivering interventions. High professionalism scores PT practice was frequently mentioned in CIs’ responses have also been reported for the PT-CPI version 2006.20 to the feasibility and satisfaction questions. Future versions of the ACP will use methods to confirm whether Although the online education module for the ACP a rater intentionally did not respond to a certain item attempts to standardise its administration and use, rater and, if so, require that this decision be explained before bias on the part of CIs and their inherent expectations of advancing to the next page, to reduce missing data and how intermediate and senior students perform may also to better understand the reasons for missing data. have played a role in the communication and pro- fessionalism scores. It is also possible that CIs’ ratings Internal consistency of students’ communication and professionalism com- ACP values for internal consistency met levels con- petencies were high because students typically emulate their intra-professional role models. Brinkman and col- sidered important for individual-level precision.13,14 leagues21 found that medical residents were rated more Although the items on the ACP were shown to be highly highly by their physician supervisors than by nurses on related, we propose that redundancy can be tolerated in several items related to communication and profes- this measure because each item provides valuable feed- sionalism. A more representative view of students’ abilities back to students about their performance. For example, may be gained through ratings by people other than the most students’ ratings on items 1.1 and 1.2 will be highly CI; however, to our knowledge, there are no reports that correlated, but in the rare occasion on which they are this question has ever been studied with respect to PT. not, the discrepancy will be helpful in targeting plans The pattern whereby some items were typically rated for improvement. In addition, in development phase 3,3 more highly than others has implications for how pro- physiotherapists indicated that having separate rating grammes may choose to determine whether students scales for each item would be beneficial for the Expert are progressing well from one clinical education experi- role. ence to another. For example, students who are awarded a final rating of Advanced Intermediate for Expert com- Validity petency 1.5 in an intermediate internship would likely The ACP’s validity was supported by several statistical be performing comparably to their peers but, if awarded the same rating for Professionalism competency 7.2, they analyses. As anticipated, effect sizes were largest for items would likely be lagging behind most of their peers. in the Expert role (0.63–0.74), an area of students’ devel- opment in which significant improvements are made Evidence of validity was also demonstrated through during an internship, and smaller for the Professional, comparisons between the ACP and the PT-CPI. The Advocate, and Communicator roles (0.40–0.61), for which Spearman correlation coefficient for ACP and PT-CPI students generally scored higher at midpoint and thus total scores was 0.89, which indicates that students who potentially had less opportunity for improvement. Effect scored well on the PT-CPI also received high scores on sizes were similar for comparable ACP and PT-CPI items. the ACP. The strongest correlations between ACP and When we ranked ACP effect sizes from lowest to highest, PT-CPI items were found between the ACP’s Expert role we found that the highest effect sizes on both ACP and items and PT-CPI items 10 (screening), 11 (assessment), PT-CPI were for establishing a diagnosis and prognosis, 12 (determining a diagnosis), and 13 (designs a treat- analyzing assessment findings, and performing an assess- ment.

166 Physiotherapy Canada, Volume 68, Number 2 https://www.utpjournals.press${contentReq.requestUri} - Horizon College Physiotherapy <[email protected]> - Monday, February 22, 2021 11:13:27 PM - IP Address:43.246.243.82 ment plan). Although we had expected higher correla- the PT-CPI assessed each student similarly, which may tions with some items (e.g., ACP item 4.3 with PT-CPI have contributed to the similarity of results between the item 1, safe practice), we found that correlations with two measures; completing the PT-CPI first may also have PT-CPI items 1–5 had lower correlation coefficients— biased CIs’ responses on the ACP. Third, CIs were invited within the 0.5–0.6 range—which was likely an effect of to participate only if the student matched to their intern- attenuation of correlations. These PT-CPI items are ship offer had already consented to participate. There- generally scored high, with small variances, which can fore, some CIs who would have been willing to partici- lower the correlations. In other instances, the correla- pate may not have been invited to do so. Last, the CIs tions were lower than anticipated; for example, we who chose to participate in this study may be a keen, anticipated a strong correlation between ACP item 7.2 enthusiastic sample of CIs (respects the individuality and autonomy of the client) and PT-CPI item 8 (adapts care to reflect individual Future directions for the ACP include developing a differences), but in fact the Pearson correlation coeffi- more robust online platform that will allow data to be cient was 0.55 at the final point. This may be a result of pooled from across the country to continue analyses the more defined rating scale in the ACP or, perhaps, from a broader representative sample, including diverse of the order of items (item 8 on the PT-CPI vs. item 20 areas of practice as well as CIs and students who choose on the ACP). to complete the ACP in French. Pooling longitudinal data will also assist academic programmes by contributing to Practicality decisions about using the ACP as a summative pass–fail Compliance with completing education modules and measure. Although using the ACP in this way was not a component of the study, having these data would be the perceived usefulness of the module are important important to each programme to support its decision- factors in minimising bias and enhancing reliability in making process regarding assigning grades in clinical how raters complete the measure.22 The ACP online edu- education. In this study, the ACP demonstrated evidence cation module received positive ratings and, on average, of high internal consistency. Notwithstanding the chal- took less than 30 minutes to complete. The ACP measure lenges of exploring inter-rater and test–retest reliability also appears to be practical for busy front-line clinicians in the clinical setting (e.g., the changing nature of the to use. clinical practice environment, ensuring enough time has passed that the rater does not recall the student’s perfor- Methodological quality mance), these two measurement properties do need to When designing the study, we considered the COSMIN be explored. The purpose of this study was to assess and report on the ACP’s overall performance. Having data checklist.12 All aspects of the study’s design for the con- from a large, accessible database will allow for the tent and construct validity categories met the COSMIN potential to establish norm references, which will help rating of ‘‘excellent.’’ In terms of internal consistency, schools identify students whose performance requires only the factor analysis criterion was not met; the suit- attention—be it remediation or exceptional honours. ability of performing factor analysis on a measure such Moreover, the database and its digital inputs will serve as ours, which has multiple domains or roles with few as a structure that will make it feasible to undertake items in each, is debatable. Because the ACP is a profile reliability studies. measure that provides multiple scores across roles, factor analysis would be at a subscale level, and our Last, the evidence for the ACP’s reliability and validity subscales, other than the Expert role, have few items. is framed within the health measurement or classic In the future, with higher numbers, we intend to do approach to validity rather than the modern approach. confirmatory factor analysis using techniques that allow Although we chose the classic approach because it exploration of measurement models in a profile-like mea- would resonate with the ACP’s end users, as described sure. We also assessed the ACP’s measurement properties in the introduction section, it has the potential to com- as recommended by Terwee and colleagues23 and van partmentalize the psychometric properties of the measure Tulder and colleagues;24 by this standard, the ACP’s rather than view the evidence to support the inferences of ratings were positive for measurement properties and the measure in a unified approach to validity. there was strong evidence for its construct and content validity, but the level of evidence for internal consistency CONCLUSIONS was ‘‘unknown’’ (defined as positive findings from studies On the basis of the psychometric analyses performed of poor methodological quality). in this study, we found the ACP to have good internal Our study has several limitations. First, although a consistency, validity, and practicality in assessing and total of 10 universities participated in our study, 69% of describing PT students’ behaviours in the clinical educa- the ACP data set (94 out of 137 completed ACP forms) tion setting. Because of these extremely promising pre- came from only 4 of those universities. Second, CIs liminary analyses, we believe the ACP can confidently may have felt a need to ensure that both the ACP and

Mori et al. Evidence of Reliability, Validity, and Practicality for the Canadian Physiotherapy Assessment of Clinical Performance 167 https://www.utpjournals.press${contentReq.requestUri} - Horizon College Physiotherapy <[email protected]> - Monday, February 22, 2021 11:13:27 PM - IP Address:43.246.243.82 be used to assess clinical education practice in Canada 10. Messick S. Validity. In: Linn RL, editor. Educational measurement. with the goal of continuing to collect data to enable 3rd ed. New York: American Council on Education/Macmillan; 1989. additional psychometric testing of the ACP. p. 13–103. KEY MESSAGES 11. American Physical Therapy Association (APTA). Physical Therapist Clinical Performance Instrument. Alexandria (VA): The Association; What is already known on this topic 1997. Assessment is a valuable component of the clinical 12. Terwee CB, Mokkink LB, Knol DL, et al. Rating the methodological education process. A reliable, valid, and practical tool that quality in systematic reviews of studies on measurement properties: reflects Canadian physiotherapy practice is required. A a scoring system for the COSMIN checklist. Qual Life Res. new assessment tool based on the Essential Competency 2012;21(4):651–7. http://dx.doi.org/10.1007/s11136-011-9960-1. Profile for Physiotherapists in Canada, the Canadian Medline:21732199 Physiotherapy Assessment of Clinical Performance (ACP), has been developed for this purpose. 13. Nunnaly J. Psychometric theory. New York: McGraw-Hill; 1978. 14. McHorney CA, Tarlov AR. Individual-patient monitoring in clinical What this study adds The ACP is a reliable, valid, and practical measure to practice: are available health status surveys adequate? Qual Life Res. 1995;4(4):293–307. http://dx.doi.org/10.1007/BF01593882. assess and describe physiotherapy students’ behaviours Medline:7550178 as observed during clinical education relative to what is 15. Cohen J. Statistical power analysis for the behavioral sciences. New expected of an entry-level physiotherapist. York: Academic Press; 1969. 16. Norman KE, Booth R, Chisholm B, et al. Physiotherapists and REFERENCES physiotherapy student placements across regions in Ontario: A descriptive comparison. Physiother Can. 2013;65(1):64–73. http:// 1. Guyatt GH, Bombardier C, Tugwell PX. Measuring disease-specific dx.doi.org/10.3138/ptc.2011-63. Medline:24381384 quality of life in clinical trials. CMAJ. 1986;134(8):889–95. 17. Canadian Institute for Health Information. Physiotherapists in Medline:3955482 Canada, 2010. Ottawa: The Institute; 2011. 18. Proctor PL, Dal Bello-Haas VP, McQuarrie AM, et al. Scoring of 2. Mori B, Brooks D, Norman KE, et al. Development of the Canadian the physical therapist clinical performance instrument (PT-CPI): Physiotherapy Assessment of Clinical Performance: a new tool to analysis of 7 years of use. Physiother Can. 2010;62(2):147–54. http:// assess physiotherapy students’ performance in clinical education. dx.doi.org/10.3138/physio.62.2.147. Medline:21359047 Physiother Can. 2015;67(3):281–9. http://dx.doi.org/10.3138/ 19. Norman KE, Booth R. Observations and performances ‘‘with distinc- ptc.2014-29E. tion’’ by physical therapy students in clinical education: analysis of check-boxes on the Physical Therapist Clinical Performance Instru- 3. Mori B, Norman KE, Brooks D, et al. Canadian Physiotherapy ment (PT-CPI) over a 4 year period. Physiother Can. 2015;67(1):17– Assessment of Clinical Performance: face and content validity. 29. http://dx.doi.org/10.3138/ptc.2013-64. Medline:25931650 Physiother Can. 2016;68(1):64–72. http://dx.doi.org/10.3138/ 20. Roach KE, Frost JS, Francis NJ, et al. Validation of the revised Physi- ptc.2015-35E. cal Therapist Clinical Performance Instrument (PT CPI): Version 2006. Phys Ther. 2012;92(3):416–28. http://dx.doi.org/10.2522/ 4. National Physiotherapy Advisory Group. Essential competency ptj.20110129. Medline:22135710 profile for physiotherapists in Canada. Toronto: Canadian Physio- 21. Brinkman WB, Geraghty SR, Lanphear BP, et al. Evaluation of therapy Association; 2009. resident communication skills and professionalism: a matter of perspective? Pediatrics. 2006;118(4):1371–9. 5. Guyatt GH, Feeny DH, Patrick DL. Measuring health-related quality 22. Williams RG, Klamen DA, McGaghie WC. Cognitive, social and of life. Ann Intern Med. 1993;118(8):622–9. http://dx.doi.org/ environmental sources of bias in clinical performance ratings. Teach 10.7326/0003-4819-118-8-199304150-00009. Medline:8452328 Learn Med. 2003;15(4):270–92. http://dx.doi.org/10.1207/ S15328015TLM1504_11. Medline:14612262 6. National Physiotherapy Advisory Group. Entry-to-practice physio- 23. Terwee CB, Bot SD, de Boer MR, et al. Quality criteria were proposed therapy curriculum: A companion document. Clinical education for measurement properties of health status questionnaires. J Clin guidelines for Canadian university programs. Council of Canadian Epidemiol. 2007;60(1):34–42. http://dx.doi.org/10.1016/j. Physiotherapy University Programs; 2011. jclinepi.2006.03.012. Medline:17161752 24. van Tulder M, Furlan A, Bombardier C, et al.; Editorial Board of the 7. Streiner DL, Norman GR. Health measurement scales: a practical Cochrane Collaboration Back Review Group. Updated method guide to their development and use. 4th ed. Oxford, UK: Oxford guidelines for systematic reviews in the Cochrane Collaboration University Press; 2008. http://dx.doi.org/10.1093/acprof:oso/ Back Review Group. Spine. 2003;28(12):1290–9. http://dx.doi.org/ 9780199231881.001.0001. 10.1097/01.BRS.0000065484.95996.AF. Medline:12811274 8. Kane MT. Validating the interpretations and uses of test scores. J Educ Meas. 2013;50(1):1–73. http://dx.doi.org/10.1111/jedm.12000. 9. Beaton DE, Boers M, Tugwell P. Assessment of health outcomes. In: Firestein GS, Budd RC, Gabriel SE, et al, editors. Kelley’s textbook of rheumatology. 9th ed. Philadelphia: Saunders; 2012. p. 462–75.

168 Physiotherapy Canada, Volume 68, Number 2 APPENDIX 1 https://www.utpjournals.press${contentReq.requestUri} - Horizon College Physiotherapy <[email protected]> - Monday, February 22, 2021 11:13:27 PM - IP Address:43.246.243.82 ACP description of items, rating scale and scoring Items scored with Expert role—focus on assessment a rating scale 1.1 Consults with the client to obtain information about his/her health, associated history, previous health interventions, and Rating scale associated outcomes. Scored 1.2 Collects assessment data relevant to the client’s needs and physiotherapy practice. 1.3 Analyzes assessment findings. Expert role—focus on analysis 1.4 Establishes a physiotherapy diagnosis and prognosis. 1.5 Develops and recommends an intervention strategy. Expert role—focus on intervention 1.6 Implements intervention. 1.7 Evaluates the effectiveness of interventions. 1.8 Completes physiotherapy services. Communicator role 2.1 Develops, builds, and maintains rapport, trust, and ethical professional relationships through effective communication. 2.2 Elicits, analyzes, records, applies, conveys, and shares information. 2.3 Employs effective and appropriate verbal, nonverbal, written, and electronic communications. Collaborator role 3.1 Establishes and maintains inter-professional relationships, which foster effective client-centred collaboration. 3.2 Collaborates with others to prevent, manage, and resolve conflict. Manager role 4.1 Manages individual practice effectively. 4.2 Manages and supervises personnel involved in the delivery of physiotherapy services. 4.3 Participates in activities that contribute to safe and effective physiotherapy practice. Advocate role 5.1 Works collaboratively to identify, respond to, and promote the health needs and concerns of individual clients, populations, and communities. Scholarly Practitioner role (all assessed with one rating scale) 6.1 Uses a reflective approach to practice. 6.2 Incorporates lifelong learning and experiences into best practice. 6.3 Engages in scholarly inquiry. Professional role 7.1 Conducts self within legal/ethical requirements. 7.2 Respects the individuality and autonomy of the client. 7.3 Contributes to the development of the physiotherapy profession. Discrete boxed adjectival rating scale with 6 anchors and 10 boxes Profile score across roles

Mori et al. Evidence of Reliability, Validity, and Practicality for the Canadian Physiotherapy Assessment of Clinical Performance 169 https://www.utpjournals.press${contentReq.requestUri} - Horizon College Physiotherapy <[email protected]> - Monday, February 22, 2021 11:13:27 PM - IP Address:43.246.243.82 1.0 EXPERT—FOCUS ON ASSESSMENT As experts in function and mobility, physiotherapists integrate all of the physiotherapist roles to lead in the promotion, improvement, and maintenance of the mobility, health, and well-being of Canadians. 1.1 Consults with the client to obtain information about his/her health, associated history, previous health inter- ventions, and associated outcomes. 1.1.1 Collects and reviews background information relevant to the client’s health. 1.1.2 Determines the client’s expectations related to physiotherapy services. 1.1.3 Collects and reviews health information about the client from other sources (e.g., other sources may include previous health records, other health care practitioners, professional colleagues, or family). 1.1.4 Collects and reviews information related to the client’s prior functional abilities, physical performance, and participation. 1.1.5 Identifies the client’s personal and environmental factors affecting his/her functional abilities, physical performance, and participation. Beginner Advanced Intermediate Advanced With Beginner Intermediate Entry Level Distinction Midterm b b b b b b b b b b Final b b b b b b b b b b 1.2 Collects assessment data relevant to the client’s needs and physiotherapy practice. 1.2.1 Selects quantitative and qualitative methods and measures based on evidence-informed practice. 1.2.2 Informs the client of the nature and purpose of assessment as well as any associated significant risk. 1.2.3 Safely performs a physiotherapy assessment, taking into account client consent, known indications, guidelines, limitations, and risk–benefit considerations. 1.2.4 Monitors the client’s health status for significant changes during the course of assessment and takes appropriate actions as required. Beginner Advanced Intermediate Advanced With Beginner Intermediate Entry Level Distinction Midterm b b b b b b b b b b Final b b b b b b b b b b Midterm Comments: Final Comments: Please check if you have significant concerns with the student’s performance on these items. k Midterm k Final

https://www.utpjournals.press${contentReq.requestUri} - Horizon College Physiotherapy <[email protected]> - Monday, February 22, 2021 11:13:27 PM - IP Address:43.246.243.82 Clinician’s Commentary on Mori et al.1 The preparation of future health care providers—doctors, their practice setting, and may have a US bias.’’4(p.282) In a nurses, pharmacists, physiotherapists, and others—is considered recently published article, Hall and colleagues5 from the Uni- a high-stakes responsibility in civil society because these gradu- versity of Alberta confirmed this sense of dissatisfaction after ates become autonomous health care providers entrusted with conducting a survey of 3,148 English- and French-speaking the care of patients and families. In Canada, we place our collec- Canadian physiotherapists (both those supervising and those tive trust in university programmes to prepare physiotherapists not supervising students) from each Canadian province and for practice at the master’s-degree level. Whenever an institution territory working in a wide range of practice areas and clinical is entrusted with a high-stakes educational endeavour, we (as a settings. Their findings provided empirical evidence that ‘‘dis- society) expect and assume that student competency is sys- like of the assessment instrument [PT-CPI]’’5(p.60) is one factor tematically assessed using valid and reliable measures and that contributing to Canadian physiotherapists’ decision not to super- students safely and effectively demonstrate required learning vise PT students. outcomes during their course of study. Over the past 3 years, and after rigorous research and devel- In professional, entry-level Master of Physiotherapy (MPT) opment, Mori and colleagues have accomplished the remark- programmes, graduating physiotherapy (PT) students complete able feat of creating and testing the new, unique ACP, which about one-third of their total academic credit units engaging possesses inherent validity because it is based on the Essential in full-time clinical courses in PT practice settings. The other Competency Profile for Physiotherapists in Canada.6 We now two-thirds of MPT programmes consist of relevant academic have a tool tailored to the Canadian PT practice context. and applied courses such as anatomy, pathology, neurosciences, exercise physiology, biomechanics, musculoskeletal practice, re- Mori and colleagues have earned the gratitude of many in search methods, and so on. In Canadian universities, academic the PT world, including courses at the master’s level must include standard academic requirements, including course descriptions, learning objectives, e PT students, who are grateful for a reliable, valid, and prac- and valid methods of assessment to determine credit (i.e., tical measure to assess and describe behaviours observed pass) or non-credit (i.e., fail) for each student. For example, in during clinical education relative to what is expected of an an academic course such as Professional Practice for Physio- entry-level physiotherapist in Canada. therapists, the summative grade for each student completing the course might reflect the cumulative assessment of student e CIs, who are grateful that the ACP seems shorter, takes less learning in several components, such as a group project (30%), time to complete, includes easy-to-interpret categories and an individual student assignment (20%), a self-reflection paper discrete rating scales, is accessible online, and reflects a (10%), and a final written exam (40%). Canadian context. On the clinical education side of the curriculum, we are e MPT programmes and clinical faculty, who value the new fortunate to have dedicated PT clinicians who serve as teachers, online platform because it will potentially allow national referred to as preceptors or clinical instructors (CIs), of MPT clinical education data to be pooled and analyzed to estab- students engaged in full-time clinical practice courses. These lish normative values in several parameters, including sum- courses possess the same mandatory components as academic mative pass–fail scores for various levels of internship. courses, including credit-unit designations, stated student learn- ing outcomes, and defined pass–fail criteria for various levels of e PT regulators and, by extension, the Canadian public, internship. The assessment of student performance in each clin- who are grateful for a Canadian tool that demonstrates evi- ical placement must, therefore, be measured and reported using dence of high internal consistency, construct validity, and a practical, reliable, and valid instrument—not only from the practicality. student’s perspective, but from the CI’s perspective as well. Hence, there is a lot resting on each clinical placement. In fact, We extend our congratulations to Mori and her Toronto- many provincial PT regulatory bodies, given their mandate of based team of researchers for producing an evidence-based protecting the public interest, publish practice guidelines per- and robust instrument, the new Canadian ACP, which will assist taining to clinical supervision of student learners, acknowledg- us all in the high-stakes endeavour of producing safe and effec- ing that quality clinical education is vital to the future provision tive entry-level PT practitioners. of relevant, safe, and effective PT services by entry-level practi- tioners.2 Peggy L. Proctor, BSc(PT) Clinical Associate Professor, School of Physical Therapy, Over the past 15 years in Canada, virtually all PT education programmes have used the Physical Therapist Clinical Per- University of Saskatchewan formance Instrument (PT-CPI),3 which was developed by the American Physical Therapy Association. However, there has REFERENCES always been a certain degree of dissatisfaction with the PT-CPI in Canada, and, in their article regarding the development of 1. Mori B, Norman KE, Brooks D, Herold J, Beaton DE. Evidence of the Canadian Assessment of Clinical Performance (ACP), Mori reliability, validity, and practicality for the Canadian Physiotherapy and colleagues4 outlined ‘‘concerns voiced by Canadian CIs Assessment of Clinical Performance. Physiother Can. that the PT-CPI is time consuming, does not always apply to 2016;68(2);156–69. http://dx.doi.org/10.3138/PTC.2014-43E 2. Saskatchewan College of Physical Therapists. Clinical supervision of student learners (SCPT practice guideline no. 22) [Internet]. 170

Clinician’s Commentary on Mori et al. 171 https://www.utpjournals.press${contentReq.requestUri} - Horizon College Physiotherapy <[email protected]> - Monday, February 22, 2021 11:13:27 PM - IP Address:43.246.243.82 Saskatoon: The College; 2015 [cited 2015 Nov 29]. Available from: 5. Hall M, Poth C, Manns P, et al. To supervise or not to supervise a http://www.scpt.org/images/ClinicalPractice_Guideline_22 physical therapist student: A national survey of Canadian physical SupervisionofStudentLearners.pdf. therapists. J Phys Ther Educ. 2015;29(3):58–67. 3. American Physical Therapy Association. Physical Therapist Clinical Performance Instrument. Alexandria (VA): The Association; 1997. 6. National Physiotherapy Advisory Group. Essential competency 4. Mori B, Brooks D, Norman KE, et al. Development of the Canadian profile for physiotherapists in Canada. Toronto: Canadian Physio- Physiotherapy Assessment of Clinical Performance: A new tool to therapy Association; 2009. assess physiotherapy students’ performance in clinical education. Physiother Can. 2015;67(3):281–9. http://dx.doi.org/10.3138/ DOI:10.3138/ptc.2014-43-CC ptc.2014-29E.

https://www.utpjournals.press${contentReq.requestUri} - Horizon College Physiotherapy <[email protected]> - Monday, February 22, 2021 11:13:27 PM - IP Address:43.246.243.82 EDUCATION Physiotherapy Students’ Attitudes toward Psychiatry and Mental Health: A Cross-Sectional Study Joanne Connaughton, BAppSc (Physio), DPhysioRes; William Gibson, BSc(Hons), MPhty, PhD ABSTRACT Purpose: A cross-sectional exploration of Notre Dame Australia physiotherapy students’ attitudes toward psychiatry and mental illness, students’ percep- tions regarding preparation in this area for general clinical practice, and a cross-sectional investigation of current mental health—and psychiatry-related content in physiotherapy curricula across Australia and New Zealand. Methods: A questionnaire including demographic details, level of exposure to mental illness, and the Attitudes Toward Psychiatry–30 items (ATP-30) was completed by pre-clinical and clinically experienced physiotherapy students from the University of Notre Dame Australia. Students with clinical experience were asked additional questions about preparedness for practice. Staff of 10 of 17 physiotherapy programmes across Australia and New Zealand responded to an online questionnaire investigating relevant content and quantity of learning experiences in mental health. Results: Student response rate was 89%. Students generally had a positive attitude about psychiatry and mental health. Women were significantly more positive than men, and students who had completed clinical experience had a significantly more positive attitude. Physio- therapy program responses (response rate ¼ 59%) highlighted disparate approaches to psychiatry and mental health learning opportunities in terms of quantity and content. Conclusion: Entry-level physiotherapy students who have clinical experience generally have a more positive attitude toward psychiatry and people with mental illness. Given the prevalence of mental health problems and the increase in physical and mental health comorbidities, it is imperative that future clinicians have positive educational experiences in psychiatry. A coherent, integrated approach to mental illness and psychiatry is suggested for entry-level physiotherapy programmes in Australia and New Zealand. Key Words: attitudes; curriculum; mental health; psychiatry; students. RE´ SUME´ Objectif : Analyse transversale des attitudes des e´ tudiants en physiothe´ rapie de l’Universite´ Notre Dame, en Australie, a` l’e´ gard de la psychiatrie et des maladies mentales, perceptions des e´ tudiants au sujet de la pre´ paration dans ces domaines en vue de la pratique clinique ge´ ne´ rale et analyse transversale du contenu courant portant sur la sante´ mentale et la psychiatrie dans les cursus en physiothe´ rapie en Australie et en Nouvelle-Ze´ lande. Me´ thodes : Des e´ tudiants en physiothe´ rapie chevronne´ s aux niveaux pre´ clinique et clinique de l’Universite´ Notre Dame, en Australie, ont rempli un questionnaire com- portant des de´ tails de´ mographiques, le niveau d’exposition aux maladies mentales et 30 questions sur les attitudes a` l’e´ gard de la psychiatrie (ATP-30). On a pose´ aux e´ tudiants qui avaient de l’expe´ rience clinique d’autres questions sur la pre´ paration a` la pratique. Dix des 17 programmes de physiothe´ rapie de l’Australie et de la Nouvelle-Ze´ lande ont re´ pondu a` un questionnaire en ligne portant sur le contenu pertinent et le volume des expe´ riences d’appren- tissage en sante´ mentale. Re´ sultats : Le taux de re´ ponse des e´ tudiants s’est e´ tabli a` 89%. Les e´ tudiants avaient en ge´ ne´ ral une attitude positive a` l’e´ gard de la psychiatrie et de la sante´ mentale. Les femmes e´ taient beaucoup plus positives que les hommes et l’on a constate´ une attitude beaucoup plus positive chez les e´ tudiants qui avaient de l’expe´ rience clinique. Les re´ ponses des programmes de physiothe´ rapie (taux de re´ ponse ¼ 59%) ont mis en e´ vidence des approches disparates des possibilite´ s d’apprentissage en psychiatrie et en sante´ mentale sur les plans de la quantite´ et du contenu. Conclusion : Les e´ tudiants en physiothe´ rapie au niveau de´ butant qui ont de l’expe´ rience clinique ont en ge´ ne´ ral une attitude plus positive a` l’e´ gard de la psychiatrie et des personnes vivant avec une maladie mentale. E´ tant donne´ la pre´ valence des proble` mes de sante´ mentale et l’augmentation des comorbidite´ s en sante´ physique et mentale, il est impe´ ratif que les futurs cliniciens aient des expe´ riences d’e´ ducation positives en psychiatrie. On sugge` re une approche cohe´ rente et inte´ gre´ e de la sante´ mentale et de la psychiatrie pour les programmes de physiothe´ rapie au niveau de´ butant en Australie et en Nouvelle-Ze´ lande. People with serious mental health issues generally Broadly speaking, many of these physical health comor- bidities fall under the scope of musculoskeletal, cardior- have poorer physical health than people without mental espiratory, and neurological conditions for which phys- illness,1–4 and the majority of these people may not receive iotherapy is recognized as either the treatment of choice appropriate primary care for these comorbidities.5–7 From the School of Physiotherapy, The University of Notre Dame Australia, Fremantle, Western Australia. Correspondence to: A/Prof Joanne Connaughton, School of Physiotherapy, University of Notre Dame Australia, PO Box 1225, Fremantle, WA 6959 Australia; [email protected]. Contributors: All authors designed the study; collected, analyzed, and interpreted the data; drafted or critically revised the article; and approved the final draft. Competing interests: None declared. Some preliminary findings from this study were presented at the Australian Physiotherapy Association Conference 2013. Physiotherapy Canada 2016; 68(2);172–178; doi:10.3138/ptc.2015-18E 172

Connaughton and Gibson Physiotherapy Students’ Attitudes toward Psychiatry and Mental Health: A Cross-Sectional Study 173 https://www.utpjournals.press${contentReq.requestUri} - Horizon College Physiotherapy <[email protected]> - Monday, February 22, 2021 11:13:27 PM - IP Address:43.246.243.82 or an essential component of management. Thus, physi- e What are physiotherapy students’ perceptions of otherapists can arguably play a key role in the delivery of mental health content in their physiotherapy course? health care to people with a mental illness.8–10 e What are the current characteristics of mental health Entry-level physiotherapy programmes in Australia learning experiences offered in Australian and New prepare graduates with the knowledge and skills to Zealander tertiary entry-level courses? assess and treat musculoskeletal conditions and chronic pain, acute and chronic cardiorespiratory conditions, a We anticipate that the results may help shape curric- wide range of neurological conditions, and many pre- ulum planning in Australian physiotherapy programmes. ventable diseases such as diabetes.8–10 Students are also taught to look for and recognize potential bio- METHODS psychosocial factors that may affect treatment in all of these areas of practice, but they receive minimal training Design in how to address and accommodate mental health Our cross-sectional study was conducted in the entry- comorbidities. Previous work involving nurses has dem- onstrated that students and new graduates rely on their level program of the School of Physiotherapy, University own personal experiences and beliefs about people with of Notre Dame Australia. All students who had not yet mental illness to shape their attitudes and guide provi- undertaken any clinical placements (Years 1 and 2) and sion of care;11 we do not currently know, however, all final-year students nearing the completion of their whether the same is true for physiotherapy students and clinical placements (Year 4) were included. Students new graduates. Research conducted in Belgium that were asked to complete a hard-copy questionnaire that examined physiotherapy students’ attitudes toward psy- included the ATP-30, age, gender, program year, and chiatry demonstrated scores deemed to be moderately previous experience with mental illness, including per- positive when assessed with the Attitudes Toward Psy- sonal mental health problems or contact with family chiatry–30 items (ATP-30),12 but little is currently known members or patients with mental health problems. Stu- about Australian physiotherapy students’ perceptions and dents were asked how often they thought they might attitudes toward psychiatry and people with mental ill- treat people with comorbid mental illness after gradua- ness. This is important to address because health care tion. Final-year students were also asked to complete providers who have negative attitudes and prejudices are two open-ended questions asking what they now knew unlikely to provide optimal health care and appropriate about psychiatry that they had not known before starting emotional support to people in this situation. their clinical placements and what they would have liked to have known. Education can foster positive attitudes toward people presenting with mental health comorbidities;13 several The ATP-30 was developed in Canada in the 1980s studies have shown that medical and allied health stu- to measure medical and occupational therapy students’ dents’ attitudes toward psychiatry can be influenced by attitudes toward psychiatry and demonstrate changes in the psychiatric content in their undergraduate courses.12–14 attitude after exposure to psychiatry through curriculum. Limited curricula in physiotherapy programmes in the The tool has been shown to be both reliable and valid.14 area of mental health and psychiatry could imply a lack It consists of 30 statements (14 positively phrased and 16 of insight by the profession into the prevalence of mental negatively phrased) about mental illness and treatment; health comorbidities and the role of psychiatry and high- psychiatric patients, psychiatric institutions, and psy- light that physiotherapists may not graduate with the chiatrists; and teaching, knowledge, and career choice. necessary skills to deliver best holistic practice in the Participants are asked to rank their response to the state- treatment of all their patients. Graduates who have not ments using a 5-point Likert-type scale (ranging from been adequately trained may then rely on their own 1 ¼ strongly agree to 5 ¼ strongly disagree). Scores for personal experiences, stereotypes, and prejudices to guide the positively phrased questions are reversed by subtract- interactions and aid holistic management for their ing them from 6; all item scores are summed to arrive at patients with mental health comorbidities. a total score out of 150. Higher scores indicate more positive attitudes; a score of 90 is considered neutral.14 We collected data from physiotherapy students from There is currently no specific tool for physiotherapy, but the University of Notre Dame Australia on their self- the ATP-30 is considered general rather than dedicated reported attitudes toward mental illness and psychiatry to a specific group. It was used in a Flemish study of and their self-reflections on education received in this area. physiotherapy students in 2010.12 Our research aimed to answer the following questions: We also developed an online course-content ques- e What are physiotherapy students’ attitudes toward tionnaire to collect data on course learning content and psychiatry and mental health? experiences in the area of psychiatry and mental illness in entry-level physiotherapy programmes across Australia e How often do physiotherapy students believe they and New Zealand. will treat patients with mental illness? Does this change once students start clinical placements? Ethics approval for this study was granted by the Human Research Ethics Committee of the University of Notre Dame Australia (HREC No. 013037F).

174 Physiotherapy Canada, Volume 68, Number 2 Table 1 Student Attitudes toward Psychiatry and Mental Health https://www.utpjournals.press${contentReq.requestUri} - Horizon College Physiotherapy <[email protected]> - Monday, February 22, 2021 11:13:27 PM - IP Address:43.246.243.82 Students Female Mean (SD) ATP-30 score* Total Students, no. (%) 115 (71) Male 162 (100) All students 104.7 (11.1) 103.1 (11.5) Clinically experienced students 110.0 (9.0) 47 (29) 107.8 (9.3) Students with no clinical experience 101.9 (11.1) 99.0 (11.6) 100.6 (11.8) 102.4 (7.9) *Unless otherwise indicated. 97.2 (12.9) ATP-30 ¼ Attitudes Toward Psychiatry–30 items. Figure 1 Comparison between pre-clinical and clinically experienced RESULTS students’ expected frequency of contact with patients with a comorbid mental illness in clinical practice. Student questionnaire Of 193 students, 172 completed the questionnaire, for Participants Students in Years 1 and 2 who had not yet been on a response rate of 89%. However, 10 questionnaires were incomplete, leaving a total of 162. The sample was 71% clinical placements (n ¼ 130) were invited to participate female (n ¼ 115), a good representation of gender distri- at the end of a lecture in April 2013. Students in Year 4 bution in the physiotherapy program (see Table 1); ages (n ¼ 63) were recruited while on campus undertaking ranged from 17 to 37 years, with a median of 20 years. clinical preparation for their final two placements. All students were asked to complete a hard copy of the ATP-30 scores were generally positive (see Table 1). A questionnaire and place their completed questionnaire t-test found more positive attitudes among female stu- in a box. Surveys were anonymous, and consent was dents than among male students (t150 ¼ 2.85, p ¼ 0.005). implied by returning the survey. The ATP-30 scores of students who had been on clinical placement were significantly more positive than those of In April 2014, we sent a link to the online course- students who had not (t151 ¼ À3.83, p < 0.005). Women content questionnaire by email to all members of the who had completed clinical placements were more posi- Physiotherapy Clinical Education Managers Australia tive than women who had not (t106 ¼ À3.8, p < 0.001), and New Zealand, asking that they pass it along to the but we found no significant differences among men most appropriate staff member for response. Of the 17 (p ¼ 0.16). universities contacted, staff members at 10 (59%) com- pleted the questionnaire. In response to the question ‘‘How often do you think you’ll be treating patients with a comorbid mental illness Analysis once you are graduated and working in clinical practice?’’ Attitudes toward psychiatry and mental health were clinically experienced students predicted more contact with patients with a comorbid mental illness than those determined using descriptive and inferential analysis. without clinical experience (see Figure 1). Among clini- Below we present results of the ATP-30 as means and cally experienced students, 93% (52 of 56) expected to standard deviations; we compared these scores by student have contact with patients with a comorbid illness at group and gender using independent t-tests (a ¼ 0.05). least three to four times per week, versus 62% (66 of Answers to the open-ended questions were coded and 106) of pre-clinical students. sorted into themes for thematic analysis. Of 56 students with clinical experience, 46 (82%) re- sponded to the two open-ended questions on psychiatry and mental health (see Table 2). Students stated that the clinical experience highlighted issues surrounding the prevalence, complexity, and management of people with comorbid mental health problems (Table 2). Students also identified a desire for more knowledge of the side effects of medications used in psychiatry and patho- physiology of common mental health problems and more education on signs and symptoms of depression and the roles of mental health practitioners, including psychiatrists, before undertaking their clinical place- ments (see Table 2).

Connaughton and Gibson Physiotherapy Students’ Attitudes toward Psychiatry and Mental Health: A Cross-Sectional Study 175 https://www.utpjournals.press${contentReq.requestUri} - Horizon College Physiotherapy <[email protected]> - Monday, February 22, 2021 11:13:27 PM - IP Address:43.246.243.82 Table 2 Students’ Responses to Open-Ended Questions What do you wish you had already known about psychiatry and people with What have you learned about psychiatry and people with mental illness mental illness before starting clinical placement? since doing clinical placements? Communication strategies to engage the person with comorbid mental The prevalence of comorbid mental illness is higher than expected. illness Medication and its side effects Patients with comorbid mental illness are more complex than expected. More information about different areas of psychiatry The approach to management of the physical health issues of the person with comorbid mental illness should change. More education on signs, symptoms, pathophysiology, and management of The effects mental illness have on physical illness mental health conditions How to identify depression The influence of mental health medications on client management More information about psychiatry treatments Table 3 Universities’ Responses to Course Content Questionnaire (n ¼ 10) Learning experience Responses Does your entry-level program have a specific unit that addresses mental illness and psychiatry? 0% have a specific unit. Do you have explicit learning experiences within your course that provide a basic overview of common mental 80% have some coverage. illnesses such as depression, anxiety, bipolar affective disorder, or eating disorders? 20% have no coverage. Do you have explicit learning experiences within your entry-level course that inform students on the prevalence 70% have course content in this area. of common mental illnesses such as depression, anxiety, bipolar affective disorder, or eating disorders? Do you have explicit learning experiences on pathophysiology of mental illness or psychiatric conditions in your 50% have no coverage. entry-level program? 30% have <3 course h. 20% have b12 course h. Do you have explicit learning experiences within your entry-level course that inform students on issues on 30% reported coverage. pharmacological management of mental illness? Do you have explicit learning experiences within your entry-level course that inform students on the impact 90% reported coverage. mental illness can have on an individual’s physical well-being and recovery? Do you have explicit learning experiences within your entry-level course that inform students on communication 20% have no coverage. strategies to engage the person with a comorbid mental illness or psychiatric condition? 80% reported 2–12 course h. Course content questionnaire was approximately neutral. This finding is consistent Of the 17 universities approached, staff members at with those of previous research comparing attitudes of male and female medical students.13 When we compared 10 completed the survey, for a response rate of 59%. clinically experienced respondents with pre-clinical re- Table 3 identifies these responses. The themes of addi- spondents, we found that women who had been on tional responses are summarized by the following com- clinical placements showed significantly more positive ments: ‘‘This is an increasingly critical area of curricula attitudes than pre-clinical female students; the same for physiotherapy students,’’ ‘‘it can be a significant comparison for male students suggested a trend in the comorbidity in our clients,’’ and ‘‘it is hard to know same direction, but the difference was not statistically where to fit it in though other than as an embedded significant. Our findings are based on cross-sectional product.’’ data, however, and further studies would be needed to determine whether and why clinical experience truly DISCUSSION has a more profound effect on attitudes toward psychia- Our findings demonstrate that, overall, undergraduate try and mental health among women than among men. students in the University of Notre Dame Australia’s Probst and Peuskens’12 study of Flemish physiotherapy entry-level physiotherapy program have generally positive students’ attitudes toward psychiatry and mental health attitudes toward psychiatry and mental health problems. produced very similar results: The female:male ratio of This is an important finding, given the prevalence of their sample was the same, as were their sample’s ATP- mental illness and physiotherapists’ role in health care 30 scores (the overall positive attitude scores of 103.1 for people with comorbid mental and physical con- and 103.9 were almost identical). These two studies ditions. Female physiotherapy students had a more were conducted independently of each other and with positive attitude toward psychiatry and mental health problems than their male counterparts, whose attitude

176 Physiotherapy Canada, Volume 68, Number 2 https://www.utpjournals.press${contentReq.requestUri} - Horizon College Physiotherapy <[email protected]> - Monday, February 22, 2021 11:13:27 PM - IP Address:43.246.243.82 no collaboration, which suggests that there may be some chiatry and mental health. The majority, including the common personality characteristics across physiotherapy program at Notre Dame Australia, include a basic over- students. view of the most commonly seen comorbidities, anxiety and depression. Although this information may help stu- Unlike previous studies that have compared students’ dents identify these common problems in their patients, attitudes before and after exposure to people with diag- it will not help them with patient management. Inter- nosed mental illness,15 our study examined attitudes estingly, despite these overviews, many Notre Dame after diverse general practice in which students may Australia students reported that they wanted more infor- have had exposure to people with comorbid mental ill- mation on how to recognize depression in their patients. ness. Our findings suggest that clinical experience has an effect on attitude, especially among female students. Pre-clinical students’ perceptions of frequency of con- It is acknowledged this study used a cross-sectional de- tact with people with comorbid mental illness were sign and that causal relationships cannot be established; significantly lower than predicted by students who had however, as an initial finding, further investigation ap- general clinical experience. Although most programmes pears warranted. educate students on the prevalence of mental health problems, the information is likely to be based on pre- To help inform curriculum, we elicited students’ valence in the total population and thus may not accu- thoughts regarding their knowledge of mental health rately reflect what students will see in practice. It is and psychiatry before clinical placements and what they known that poor mental health and poor physical health perceived would have helped them with management of are linked,1–4 so it is not unrealistic to expect that people their clients. Probst and Peuskens12 demonstrated that with mental health issues may make up a larger pro- education specific to psychiatric rehabilitation had a portion of admissions or outpatient appointments in positive impact on physiotherapy students’ attitudes, general hospital settings than of the general population. and a qualitative study by Matteo (2013) highlighted an Further research should be undertaken to determine the increase in empathy among psychology students after actual prevalence of comorbid mental illness among education on mental illness and stigma.15 Kuhnigk et al.’s16 physiotherapy patients. and other studies7,13,17,18 have found that a positive edu- cation experience fosters a positive attitude toward psy- Students expressed some surprise at the added com- chiatry in students, which translates into better patient plexity of management of and recovery from physical care. It may be that more education before clinical expe- health problems when comorbid mental health issues rience would result in further improvement in attitudes are present. The majority of entry-level programmes toward psychiatry and mental health issues. Currently, appear to cover the impact of mental health problems physiotherapy students at the University of Notre Dame on physical health and recovery, but further investi- Australia receive 4 hours of lectures dedicated to patho- gation would be needed to determine what format this physiology and management of clients with mental ill- education takes. As previously suggested, a brief over- ness. References to depression and how it may affect view of mental health problems may help physiotherapy treatment are also embedded in the musculoskeletal students recognize these comorbidities, but, as in other and neurological elements of the curriculum. Hodgins health professions, better training is needed to help et al.17 argued that information alone is not enough to physiotherapy students manage these issues.20 Students engender a more positive attitude and that a practice- may not be able to fully appreciate the impact of mental based educational model is required in which students health comorbidities before exposure to treating patients can develop and practise skills at the same time as they in clinical practice. receive education on patient management. This is an element to be considered in future curriculum develop- We can assume that an understanding of pathophy- ment for any physiotherapy program. siology will help students to understand the manifesta- tions of mental health conditions and, to some extent, Our finding that students who have been on general the impact of these conditions on a person’s physical clinical placements have a more positive attitude is health. The students we surveyed requested more infor- interesting and suggests that students may be accessing mation on pathophysiology, highlighting their need to positive information by observing more experienced better understand these mental health problems so as clinicians during their clinical experience. In another to better manage their patients. Of the physiotherapy study, psychology students developed a more positive programmes surveyed, 50% (n ¼ 5) included in their attitude after having personal contact with people with curricula learning experiences that address the patho- mental health problems,19 and perhaps the same phe- physiology of some mental health conditions. nomenon is experienced by physiotherapy students on general clinical placement. Students also indicated that they wanted more educa- tion and learning experiences dedicated to communica- Our survey of physiotherapy programmes across tion strategies for engaging this client group. Of the Australia and New Zealand uncovered little coherence programmes surveyed, 80% (n ¼ 8) included strategies in terms of content and quantity of information on psy- for communicating with people with mental illness, but

Connaughton and Gibson Physiotherapy Students’ Attitudes toward Psychiatry and Mental Health: A Cross-Sectional Study 177 https://www.utpjournals.press${contentReq.requestUri} - Horizon College Physiotherapy <[email protected]> - Monday, February 22, 2021 11:13:27 PM - IP Address:43.246.243.82 the course time devoted to this topic ranged from 2 forces the psychological in the bio-psychosocial approach hours to 12 hours over the duration of the program, advocated for physiotherapeutic management. This inte- which may not be enough time or offer enough learning grated approach is used in all participating programs sur- opportunities to allow students to move from accessing veyed that indicated they included mental health content knowledge to developing skills and, finally, applying those in their course. skills in this area. Physiotherapy students spend many hours practising their practical clinical skills in tutorials Similarly, Kuhnigk and colleagues (2009)13 advocated and laboratory sessions, and they arguably need the this integrated approach for medical students, and Bell same practice to perfect the communication skills used and colleagues (2006)23 for pharmacy students. Kuhnigk to engage people with mental health comorbidities.17 and colleagues13 also highlighted how attitudes toward psychiatry and mental health problems improved after Side effects of medications used to manage mental students had undertaken specific units about mental health issues can have a significant impact on physical health and recommended the integration of training in health and well-being, and our student respondents this area throughout the whole course. identified this as the second most important element lacking in their education. Only 30% of programmes Our study had several limitations. First, the students surveyed (n ¼ 3) reported having learning experiences in invited to participate were from only one university and this aspect of mental health, and it is obvious that this thus are representative only of that program. Further needs to be addressed. studies should include all programs in Australia to pro- vide better representation of Australian physiotherapy Physiotherapy programmes in Australia are accredited students as a whole. Second, the cross-sectional study through the Australian Physiotherapy Council, whose design limits our ability to draw causal connections Australian Standards for Physiotherapy include quite from our data. A prospective study could further explore strict guidelines on curriculum content. The Chartered the impact of clinical experience on attitudes toward Society of Physiotherapists21 has acknowledged mental psychiatry. health comorbidity and a need for more education on managing the increasing number of clients with mental CONCLUSION health issues so that future physiotherapists can deliver Physiotherapy students from the University of Notre best-practice treatment to and management for their patients. Dame Australia who have some clinical experience generally have a more positive attitude toward psychiatry Respondents to our program survey identified a than those who have not yet begun clinical placements. common problem across physiotherapy curricula in Students without clinical experience predicted a low fre- Australia: the increasing demand to fit more content quency of contact with people with comorbid mental into already content-heavy entry-level programmes.22 health problems compared with students who had clini- Respondents saw the prospect of introducing more con- cal experience. Communication strategies; medication tent related to mental health as a significant challenge, side effects; and signs, symptoms, and pathophysiology one that may require removing other curriculum con- of common mental health problems were identified by tent; this could have serious implications for accredita- students as the key areas in which they would have liked tion and therefore requires very careful consideration. training before undertaking clinical placement. Learning Clearly, this situation is problematic and calls for a re- experiences in physiotherapy entry-level programmes view of the standards to identify whether all current across Australia and New Zealand are varied and in- standards are relevant to the changing face of health consistent. care. Examining students’ attitudes at different universities Clarification and consensus on what constitutes men- could give insight into the impact of these differing learn- tal health and psychiatry content within physiotherapy ing experiences on students’ attitudes toward mental courses may be an area for further investigation. For health. Students from different universities undertake a example, the physiotherapy program at the University of similar suite of clinical placements, and a comparison of Notre Dame Australia includes education on managing students’ attitudes would be linked more to the indi- chronic pain, complex regional pain syndrome, and vidual and the program than to the clinical experience. chronic fatigue syndrome within musculoskeletal and This would allow us to evaluate the impact on attitude pain management units, but not under the umbrella of of the actual mental health learning experiences within mental health; neurology lectures include information each program and to establish a benchmark. about the common side effects of depression and anxiety; and developmental problems and physiotherapy involve- KEY MESSAGES ment in eating disorders are included in pediatric course content. Arguably, integrating psychiatric content through- What is already known on this topic out the physiotherapy curriculum accurately reflects the A negative attitude and prejudices toward people with prevalence of mental health issues in society and rein- mental illness is unlikely to result in optimal health care delivery. If students and graduates rely on their own

178 Physiotherapy Canada, Volume 68, Number 2 https://www.utpjournals.press${contentReq.requestUri} - Horizon College Physiotherapy <[email protected]> - Monday, February 22, 2021 11:13:27 PM - IP Address:43.246.243.82 personal experiences, stereotypes, and prejudices to guide physiotherapy: position statement [Internet]. Camberwell (VIC): The interaction with and management of their patients with Association; 2008 [cited 2012 Aug 17]. Available from: http:// mental health comorbidities, it is imperative that they www.physiotherapy.asn.au/DocumentsFolder/Advocacy_Position_ have a positive attitude. Education can foster positive atti- Primary_Health_Care_2008.pdf tudes toward people with mental health comorbidities. 11. MacNeela P, Scott PA, Treacy M, et al. A risk to himself: attitudes toward psychiatric patients and choice of psychosocial strategies What this study adds among nurses in medical-surgical units. Res Nurs Health. Australian physiotherapy students from one univer- 2012;35(2):200–13. http://dx.doi.org/10.1002/nur.21466. Medline:22334254 sity have a positive attitude toward psychiatry. General 12. Probst M, Peuskens J. Attitudes of Flemish physiotherapy students clinical experience may have a positive effect on attitude. towards mental health and psychiatry. Physiotherapy. Students recognized that mental health problems added 2010;96(1):44–51. http://dx.doi.org/10.1016/j.physio.2009.08.006. complexity to the management and recovery of people’s Medline:20113762 physical health problems. Physiotherapy programs across 13. Kuhnigk O, Hofmann M, Bo¨thern AM, et al. Influence of educational Australia and New Zealand have a very disparate programs on attitudes of medical students towards psychiatry: approach to psychiatry and mental health learning oppor- Effects of psychiatric experience, gender, and personality dimen- tunities, with the majority including basic information sions. Med Teach. 2009;31(7):e303–10. http://dx.doi.org/10.1080/ on anxiety and depression. 01421590802638048. Medline:19811138 14. Burra P, Kalin R, Leichner P, et al. The ATP 30—a scale for measur- REFERENCES ing medical students’ attitudes to psychiatry. Med Educ. 1982;16(1):31–8. http://dx.doi.org/10.1111/j.1365- 1. Coghlan R, Lawrence D, Holman C, Jablensky A. Duty to care: 2923.1982.tb01216.x. Medline:7057722 physical illness in people with mental illness. Crawley: University of 15. Matteo E. A qualitative study of undergraduates’ conceptualizations Western Australia; 2001. of mental illness. J Prev Intervent Community. 2013;41(4):231–43. http://dx.doi.org/10.1080/10852352.2013.818482. Medline:24010560 2. Richardson CR, Faulkner G, McDevitt J, et al. Integrating physical 16. Kuhnigk O, Strebel B, Schilauske J, et al. Attitudes of medical stu- activity into mental health services for persons with serious mental dents towards psychiatry: effects of training, courses in psychiatry, illness. Psychiatr Serv. 2005;56(3):324–31. http://dx.doi.org/10.1176/ psychiatric experience and gender. Adv Health Sci Educ Theory appi.ps.56.3.324. Medline:15746508 Pract. 2007;12(1):87–101. http://dx.doi.org/10.1007/s10459-005- 5045-7. Medline:17115252 3. Galletly CA, Foley DL, Waterreus A, et al. Cardiometabolic risk 17. Hodgins G, Judd F, Davis J, et al. An integrated approach to general factors in people with psychotic disorders: the Second Australian practice mental health training: the importance of context. Australas National Survey of Psychosis. Aust N Z J Psychiatry. 2012;46(8):753– Psychiatry. 2007;15(1):52–7. http://dx.doi.org/10.1080/ 61. http://dx.doi.org/10.1177/0004867412453089. Medline:22761397 10398560601083019. 18. Thomas S, Pai N, Dawes K, et al. Updating medical school psychiatry 4. Morgan VA, Waterreus A, Jablensky A, et al. People living with curricula to meet projected mental health needs. Australas Psychiatry. psychotic illness in 2010: the Second Australian National Survey of 2013;21(6):578–82. http://dx.doi.org/10.1177/1039856213500092. psychosis. Aust N Z J Psychiatry. 2012;46(8):735–52. http:// Medline:23996665 dx.doi.org/10.1177/0004867412449877. Medline:22696547 19. Matteo EK, You D. Reducing mental illness stigma in the classroom. Teach Psychol. 2012;39(2):121–4. http://dx.doi.org/10.1177/ 5. Griswold KS, Homish GG, Pastore PA, et al. A randomized trial: are 0098628312437720. care navigators effective in connecting patients to primary care after 20. Arvaniti A, Samakouri M, Kalamara E, et al. Health service staff’s psychiatric crisis? Community Ment Health J. 2010;46(4):398–402. attitudes towards patients with mental illness. Soc Psychiatry http://dx.doi.org/10.1007/s10597-010-9300-x. Medline:20204511 Psychiatr Epidemiol. 2009;44(8):658–65. http://dx.doi.org/10.1007/ s00127-008-0481-3. Medline:19082905 6. Griswold KS, Zayas LE, Pastore PA, et al. Primary care after 21. Chartered Society of Physiotherapy. Recovering mind and body: psychiatric crisis: a qualitative analysis. Ann Fam Med. 2008;6(1):38– a framework for the role of physiotherapy in mental health and 43. http://dx.doi.org/10.1370/afm.760. Medline:18195313 wellbeing [Internet]. London: The Society; 2008 [cited 2012 Aug 17]. Available from: http://www.csp.org.uk/publications/recovering- 7. van der Kluit MJ, Goossens PJJ. Factors influencing attitudes of mind-body nurses in general health care toward patients with comorbid mental 22. McMeeken J, Webb G, Krause K-L, et al. Learning outcomes and illness: an integrative literature review. Issues Ment Health Nurs. curriculum development in Australian physiotherapy education. 2011;32(8):519–27. http://dx.doi.org/10.3109/01612840.2011.571360. Melbourne: University of Melbourne; 2005. Medline:21767254 23. Bell JS, Johns R, Chen TF. Pharmacy students’ and graduates’ attitudes towards people with schizophrenia and severe depression. 8. Australian Physiotherapy Association. Chronic disease and physio- Am J Pharm Educ. 2006;70(4):77–77. http://dx.doi.org/10.5688/ therapy: position statement [Internet]. Camberwell (VIC): The aj700477. Medline:17136196 Association; 2009 [cited 2012 Aug 17]. Available from: http:// www.physiotherapy.asn.au/DocumentsFolder/Advocacy_ Position_Chronic_Disease_2009.pdf 9. Australian Physiotherapy Association. Pain management [Internet]. Camberwell (VIC): The Association; 2012 [cited 2012 Aug 17]. Available from: http://www.physiotherapy.asn.au/DocumentsFolder/ Advocacy_Position_Pain_Management_2012.pdf 10. Australian Physiotherapy Association. Primary health care and

https://www.utpjournals.press${contentReq.requestUri} - Horizon College Physiotherapy <[email protected]> - Monday, February 22, 2021 11:13:27 PM - IP Address:43.246.243.82 EDUCATION Interviewers’ Experiences with Two Multiple Mini- Interview Scoring Methods Used for Admission to a Master of Physical Therapy Programme Ina van der Spuy, PhD; Angela Busch, PhD; Julia Bidonde, PhD ABSTRACT Purpose: To describe participants’ attitudes, beliefs, and experiences with the use of two methods of scoring the Multiple Mini-Interview (MMI) for admis- sion to a Master of Physical Therapy program: a rank-based scoring system (RBS; used from 2007 to 2013) and a criterion-based scoring system (CBS; tested in 2014). The MMI uses short independent assessments to obtain an aggregate score of candidates’ professionalism and interpersonal skills, based on behavioural questions within scenarios that assess one attribute at a time. Method: This qualitative descriptive inquiry sought to capture the experiences of 18 MMI interviewers primarily through semi-structured interviews. Interviews were transcribed verbatim, and the data were analyzed using thematic analysis. The results were validated by theoretical and investigator triangulation and member checking. Results: One major theme, scoring systems, and two sub-themes, CBS and RBS, emerged across all data. Participants unanimously agreed that CBS is a more fair and objective way to score candidates’ interviews. Conclusions: CBS was well accepted by participants, and the majority preferred it over RBS. Participants felt that CBS presented a more accurate depiction of candidates. Key Words: evaluation studies as topic; qualitative research; students. RE´ SUME´ Objectif : De´ crire les attitudes, les croyances et les expe´ riences des participants face a` l’utilisation de deux me´ thodes d’attribution d’une note a` la mini- entrevue multiple (MEM) pour l’admission a` un programme de maıˆtrise en physiothe´ rapie: un syste` me de classement selon la note rec¸ ue (RBS, utilise´ de 2007 a` 2013) et un syste` me de classement selon des crite` res (CBS, teste´ en 2014). La MEM utilise de bre` ves e´ valuations inde´ pendantes pour produire, au sujet du professionnalisme et des techniques de relations interpersonnelles des candidats, une note agre´ ge´ e base´ e sur des questions relatives au comportement dans les sce´ narios qui e´ valuent une qualite´ a` la fois. Me´ thode : Cette recherche descriptive qualitative visait a` saisir les expe´ riences de 18 intervieweurs MEM principalement graˆ ce a` des entrevues semi-structure´ es. Les entrevues ont e´ te´ transcrites textuellement et l’on a analyse´ les donne´ es en appliquant une analyse the´ matique. Les re´ sultats ont e´ te´ valide´ s par triangulation entre la the´ orie et les chercheurs et par une ve´ rification aupre` s des membres. Re´ sultats : Un the` me de premier plan, soit les syste` mes de notation, et deux sous-the` mes, soit les syste` mes RBS et CBS, ont e´ merge´ de toutes les donne´ es. Les participants ont convenu a` l’unanimite´ que le classement selon des crite` res constitue une fac¸ on plus e´ quitable et objective d’attribuer une note aux entrevues des candidats. Conclusion : Les participants ont bien accepte´ le classement base´ sur des crite` res qu’ils pre´ fe´ raient en majorite´ au classement selon la note rec¸ ue. Les participants e´ taient d’avis que le classement base´ sur des crite` res repre´ sentait plus fide` lement les candidats. For many years, admissions for medical and physical tors not relevant to candidates’ suitability.1,2,4 Inter- therapy (PT) programmes have used the traditional inter- viewer variability accounts for fully 56% of the variance view.1,2 Recent studies have demonstrated, however, that in traditional interview scores.5 traditional interviews may not be sufficient to select the best candidates.3 The main criticism of traditional inter- The Multiple Mini-Interview (MMI) addresses these views is that they may not be reliable or valid because concerns through its structure of multiple stations, inter- they are context specific; that is, outcomes rely heavily viewers, and questions.2–4 In this way, the MMI expands on the interviewers, the questions being asked, and fac- on the traditional interview and addresses context spe- cificity.1,3 Both the MMI and the traditional interview From the School of Physical Therapy, University of Saskatchewan, Saskatoon. Correspondence to: Ina van der Spuy, School of Physical Therapy, College of Medicine, University of Saskatchewan, 1121 College Dr., Saskatoon, SK S7N 0W3; [email protected] Disclaimers: The views expressed in this article are those of the authors and not the official position of the School of Physical Therapy, University of Saskatchewan. Competing interests: None declared. Acknowledgements: The authors thank Julie Penner, BScPT, MSc, for help with research on the background section of the article. Physiotherapy Canada 2016; 68(2);179–185; doi:10.3138/ptc.2015-24E 179

180 Physiotherapy Canada, Volume 68, Number 2 https://www.utpjournals.press${contentReq.requestUri} - Horizon College Physiotherapy <[email protected]> - Monday, February 22, 2021 11:13:27 PM - IP Address:43.246.243.82 aim to assess professionalism and interpersonal skills.1,2 accommodate the 96 selected candidates, four cycles of Six characteristics are assessed in the MMI: communica- MMIs were completed. Three stations per circuit in all tion and interpersonal skills; ethical and moral reason- four cycles in the morning and afternoon were desig- ing; critical thinking and logical reasoning; preparation nated as research stations, and the same three questions and motivation to study in the designated field; leader- were used throughout the day. During their orientation ship and teamwork; and integrity and honesty.3 Eva and on MMI day, candidates were informed about the re- colleagues’2 original 2004 study identified six benefits of search process and were instructed to ignore the second the MMI over the traditional interview: The MMI allows person in the room while being interviewed at the re- multiple views into candidates’ abilities; dilutes exam- search stations. iner bias and chance; is structured so that all candidates respond to the same scenarios and questions; allows for Each station presented a scenario designed to evaluate flexibility in what is assessed at particular stations; gives factors such as the candidate’s empathy, integrity, ethical candidates an opportunity to start over with a new inter- judgment, and overall communication skills. Each candi- viewer if they have difficulty with any particular station; date had 10 minutes at each station: 2 minutes to read and may require fewer resources. Several studies have the scenario and 8 minutes to respond. The interviewer confirmed that the MMI is reliable, valid, and suitable then scored the candidate using CBS; at the research sta- for assessing candidates’ skills in non–fact-based knowl- tions, observers simultaneously scored candidates using edge categories such as communication.1,4,6 RBS. Rank-based scoring (RBS) is used to score candidates’ Participants MMI for the medical programmes at McMaster Uni- We used an opportunistic sampling strategy, and all versity, where the MMI was first developed. The RBS approach determines candidates’ scores relative to the participants were drawn from the pool of 27 admissions performance of a pool of candidates.7 interviewers (PT faculty, PT clinicians, and 2nd-year MPT students).12 A letter of invitation was mailed to all Scores on interviews can also be based on predeter- potential participants. mined standards for specific performance of objectives or skills, also known as criterion-based scoring (CBS).8,9 Research ethics approval was obtained from the Uni- CBS is reported to be better for testing the mastery of versity of Saskatchewan, certifying that the study met skills and to easily encompass items with a greater range national standards for research involving human partici- of difficulty or ease. When scoring the performance of pants, and informed consent was obtained from each individuals in a homogeneous group, CBS has better participant. Each participant was paired and stationed curricular validity than norm-referenced tests, and this with another participant; one served as the active inter- approach also performs well in heterogeneous groups.8,10,11 viewer for the candidate and the other served as an observer for each of the 96 candidates interviewed. Between 2007 and 2013, the Master of Physical Therapy Training for both interviewer and observer roles at the (MPT) program at the University of Saskatchewan used research stations was provided before MMI day. The the RBS system with the MMI; in 2014, we tested the training session included information on the admissions CBS system. The purpose of this study, therefore, was to and MMI processes, detailed information on CBS, and describe participants’ attitudes, beliefs, and experiences practice in the use of both CBS and RBS. with these two methods of scoring the MMI. Scoring of the Multiple Mini-Interview METHODS Our study used a qualitative descriptive inquiry approach Criterion-based scoring Participants scored candidates’ performance in three that was reflective and interpretive in nature to describe the participants’ experiences with scoring the MMI using categories (critical thinking, communication, and pro- RBS and CBS. The appeal of a descriptive inquiry per- fessionalism) on a 10-point scale (ranging from 1– spective is in the pedagogical reflections and the com- 2 ¼ unsatisfactory/unacceptable to 9–10 ¼ exceptional; see prehensive summary of events that this approach elicits, detailed rubric in Appendix 1 online). A detailed rubric, which we considered necessary to understand partici- rubric summary, and worksheet were provided to help pants’ experiences with the two scoring methods. participants with scoring. Context Rank-based scoring This study was conducted during the admissions pro- Participants assigned each candidate a single score on cess for the MPT program at the University of Saskatche- the basis of his or her overall performance on the main wan. The MMI involves a circuit of multiple stations focus of the case relative to the seven other candidates through which the candidates rotate. The MMI used in that circuit. Participants were encouraged to assign for this study had three circuits consisting of the same preliminary scores, which they revised later as they eight stations (seven scenarios and one rest station); to assessed the remainder of the candidates in the circuit.

van der Spuy et al. Experiences with Two Multiple Mini-Interview Scoring Methods for Master of PT admission 181 https://www.utpjournals.press${contentReq.requestUri} - Horizon College Physiotherapy <[email protected]> - Monday, February 22, 2021 11:13:27 PM - IP Address:43.246.243.82 Data collection we also contacted participants by email and asked them The primary data source was semi-structured one-on- to review their transcripts for accuracy, after which all participants signed a transcript release form. They were one interviews conducted with the participants, by also asked to comment on whether they saw their expe- phone or in person, by two of the investigators (see riences in the themes that emerged from the analysis. All Interview Guide in Appendix 2 online). The interviews participants responded favourably to the themes, and were recorded and transcribed verbatim. Participants none suggested changes. were given ample opportunity for input, allowing them to elaborate on their experiences and opinions related Finally, theoretical triangulation contributed to the to the MMI scoring methods. Open-ended questions study’s credibility because the research question was such as ‘‘Could you describe your experience scoring situated within the current literature on admissions to the MMI this year?’’ and ‘‘What do you think about the health professional programmes. Confirmability of our accuracy of the interview scores?’’ were used to increase findings was supported by the use of multiple data the breadth of responses.13 The existing literature on sources (interview transcripts, assessor feedback data, MMI admissions experiences helped in the development and investigators’ detailed notes). To further reduce of the interview guide. investigator bias, we used investigator triangulation.12 Together, we possess knowledge of PT, admissions pro- All admissions interviewers completed assessor feed- grammes, and qualitative inquiry; to bring dependability back forms at the end of the MMI. Data from the partic- to our interpretation of the data, we coded the data in- ipants’ feedback forms were transcribed verbatim and dependently and determined the emergent themes col- used to further probe into questions addressed during laboratively. As noted, we maintained an audit trail (i.e., the one-on-one interviews.14 The full assessor feedback notes) recording methodological decisions, contextual form is available from the lead investigator. notes, analytic comments, and information on the pro- cesses and progress of the study.18 The transferability of To create an audit trail, detailed notes were recorded the findings to other contexts depends on similarities to at every stage and included descriptions of the research the program and to the context and nature of the process steps taken, from the start of the project to the develop- described.12 A detailed list of strategies used to ensure ment and reporting of findings and preliminary thoughts trustworthiness is available in Appendix 3 online. about emerging experiences. These descriptions per- mitted a conceptual return to the setting during the RESULTS data analysis.15 Of the 27 potential participants, 18 confirmed their Data analysis interest in participating in the research project by sub- Each investigator completed an inductive line-by-line mitting signed consent forms, for a response rate of 67%. The 18 study participants (6 faculty members, 6 thematic analysis using QDA Miner 4 Lite (Provalis clinicians, and 6 second-year students) were aged be- Research, Montreal). We read the transcripts and notes tween 23 and 62 years; 12 had prior experience and 6 numerous times.14 Particularly revealing phrases were had no experience with admissions and administering the highlighted, coded, and assigned meaningful labels. In MMI. All faculty and clinicians held at least a bachelor’s the event of discrepancies in codes or variance in the degree in PT; in addition, 2 held master’s degrees and 4 labels, a discussion ensued until we reached a common held doctoral degrees. understanding. The data analysis continued by constantly comparing labels and phrases to determine whether they One major theme, scoring systems, and two sub- should be classified separately or whether they belonged themes, CBS and RBS, were identified across all data. to an existing code.16 We then determined the essential Within these themes, overlapping themes and category or invariant themes, that is, those that gave fundamental weighting were also discussed. Pseudonyms (element meaning to the experiences, as identified by their pattern symbols) are used to identify participants instead of regularities.14 names and are indicated in parentheses at the end of the quote (i.e., Li). Trustworthiness The truth value or credibility of the data was en- Participants were unanimous in recognizing that both interpersonal and intellectual characteristics are im- hanced via data saturation, that is, by repeating infor- portant qualities for physical therapists to possess. They mation and verifying previously collected data through also agreed that the MMI consistently captured the pro- subsequent interviews. The first two investigators estab- fession’s expectations and emphasized that only through lished rapport with the participants during the MMI the interview can candidates’ personal characteristics training and admissions sessions; they also knew the be identified. The interview was regarded as a tool to participants through their prior involvement with the capture essential characteristics of interest and weed program. As Guba17(p.85) has stated, member checking is out candidates not suitable for the profession: ‘‘the single most important action inquirers can take, for it goes to the heart of the credibility criterion’’; therefore,

182 Physiotherapy Canada, Volume 68, Number 2 https://www.utpjournals.press${contentReq.requestUri} - Horizon College Physiotherapy <[email protected]> - Monday, February 22, 2021 11:13:27 PM - IP Address:43.246.243.82 I think interviews are good and important because I think Furthermore, participants were confident that CBS the things that you measured are important . . . they may not only portrayed candidates more accurately but had have worked their tails off and had a good GPA but if they the ability to discriminate between excellent, good, and are not able to have good critical thinking, then they may poor candidates: not make a good physical therapist because there will be situations that come up that aren’t in the textbook. (Li) Candidates who were exceptional would likely excel in all three and likewise on the other end if they were un- The printed materials (worksheet and rubric) provided acceptable, then they were likely low in all three. But in were considered useful and appropriate for the tasks at the middle is where it really clarified things. (Mg) hand. Some participants used these materials more than others; reasons for these variations were interviewer However, the accuracy of CBS in discriminating be- experience, reliance on memory (e.g., particular features tween the categories assessed was not always clear. Pro- such as t-shirt colour), and use of a personal system (e.g., fessionalism, in particular, was challenging: Although sticky notes). most participants defined professionalism as capturing what it means to be a professional and what is expected The participants had mixed views on whether suffi- of them, the concept is complex and has several dimen- cient time was allocated to each station. Although some sions. The difficulty may be rooted in personal beliefs participants considered an 8-minute interview with 2 about what constitutes professionalism: minutes to finish off scoring as just right, others saw it as restricting their ability to make an appropriate judg- I think the professionalism needs to be perhaps fine- ment. Participants reflected on the nature of the station tuned. (Al) (i.e., some cases were more complex than others), which scoring system they were using, their own experience I think that professionalism is extremely important and I with interviewing, and the adoption of a tracking method think it is great that it has its own category. (Si) as key to using the time wisely and effectively. Communication was considered key to the profession. Most participants experienced some degree of physical Participants’ underlying assumption was that candidates or mental fatigue over the course of the day, depending would be highly articulate, with strong communication on their role. Participants reported that being an ob- skills and fluency in English, and have the ability to share server or an interviewer required more or less concentra- and express ideas in individual or group situations, orally tion, and each role brought particular challenges. Multi- or in writing. Language and communication were con- tasking was complex or even unfamiliar, and new sidered bearers of culture and identity, a powerful in- participants reported that the process took more time strument and a key ingredient in a candidate’s success. and energy than they had expected it would. In addition, One participant noted that communication is far more some candidates, or sets of candidates, were more de- than a neutral medium for conveying ideas and that manding than others, requiring more concentration or mastering communication will have an impact on how effort (e.g., prompting, engaging) from the interviewer, physical therapists are perceived by others. Communi- which added to an already demanding job. cation, including interpersonal and relationship skills (i.e., professionalism), was without hesitation considered Scoring systems essential to the profession. Criterion-based scoring Overlapping areas Participants agreed that the three categories—critical However, participants also saw communication as thinking, communication, and professionalism—appro- overlapping with professionalism. Collectively, profession- priately captured the necessary outcomes for the profes- alism and communication demonstrated candidates’ sion. For this reason, CBS was assessed as fair in terms credibility and suitability for the program. One inter- of its capacity to evaluate candidates appropriately and viewer suggested either amalgamating these categories accurately: or breaking them down to better tell them apart: I thought it was much more accurate. I thought you could Professionalism I felt overlapped a lot with particularly get a better feel for the person in the three areas. . . . I feel the communication one, and a lot of times I judged pro- that the CBS does give a more accurate representation of fessionalism based on the communication delivered, so if the candidates. (Na) I am thinking of some of the really poor candidates, they seemed unprofessional because they didn’t communicate This scoring system ensured greater objectivity, struc- well. Their body language was poor; they spoke in a really ture, and consistency. Participants emphasized that CBS high, rapid tone, without any interaction with the inter- enabled them to link these properties with the desirable viewer to see how I was receiving their information. (Cl) and sought-after qualities or characteristics. Many partic- ipants felt that CBS was both a fair and an effective tool Critical thinking was also considered essential to the to evaluate affective characteristics. profession; participants described it as a process, a way

van der Spuy et al. Experiences with Two Multiple Mini-Interview Scoring Methods for Master of PT admission 183 https://www.utpjournals.press${contentReq.requestUri} - Horizon College Physiotherapy <[email protected]> - Monday, February 22, 2021 11:13:27 PM - IP Address:43.246.243.82 of thinking, understanding, and expressing oneself. Crit- A secondary problem was that most candidates were ical thinking shows that one can accurately and carefully forced into the middle. Participants felt that RBS limited analyze ideas, observations, and experiences and explore the interviewer’s flexibility and was more of an obstacle the evidence. Participants felt that the critical thinking than a help in selecting appropriate candidates: category was well defined and a stand-alone area; at the same time, however, it was hard to evaluate because par- There were some candidates that obviously did very well ticipants were looking for specific answers: and some that did poorly, but the majority were in the middle. (Ni) It was hard. I think the critical thinking component is really important but it was harder to judge. (Ar) An important requirement of a scoring system is that it be able to discriminate among poor, good, and excel- I think that is a very well-defined criterion in that tool lent candidates. Participants considered RBS both in- and I think that it quite stands alone (Ca). accurate and unable to discriminate; in general, they disliked RBS: Category weighting Suggesting that CBS categories be weighted generated I think it was good at either end, being the poor or kind of unsatisfactory versus the exceptional, and it got a little bit interesting comments. Assigning weight would imply more difficult with the in-between kind of ones. (Cu) agreement on the categories to be evaluated and reflect the importance of each category. Some participants RBS did not allow interviewers to discriminate among opposed attaching a weight to any category, arguing factors such as communication, critical thinking, and pro- that all are equally important; some were uncertain fessionalism. Personal preferences swayed their answers: about weighting the categories, arguing that they could Whereas communication was more important for one not decide which was most important: interviewer, critical thinking was paramount for others: I think it would be very hard to pick which should be It was definitely a lot tougher. . . . It was hard because weighted more. I think they are all very important when some students may have been really good in one regard, you are looking at candidates trying to get into the School like problem solving or critical thinking, but not as good of PT . . . you are going to get a really good overall picture in communication, so then they got chopped to compare if you weight them the same. (Sc) them to students who may have had different strengths. I found it tough to make those comparisons, and then it I probably would put the most weight on the communi- was up to me which one I valued more out of the three. cation and critical thinking. (Cr) (Zn) Rank-based scoring DISCUSSION Although RBS was intended to help participants dis- This study explored participants’ attitudes, beliefs, and criminate among candidates and choose those who experiences related to two methods of scoring the MMI, present themselves as right for the profession—a worthy both in detail and in depth. The interviews were not re- objective—participants often felt that the cure was worse stricted to specific questions, although they were guided than the disease. Overall, they expressed concerns about by the investigator. a lack of fairness, difficulties with multitasking, and ranking a small group of candidates. Participants were Participants unanimously agreed that CBS was a more generally discontent with the performance of RBS. suitable method for selecting candidates for admission to the MPT program. Their main concern about using There was a strong perception that RBS was not fair to RBS was the difficulty of accurately ranking the small all candidates. Some experienced participants questioned pool of candidates (eight per circuit). RBS also hampered the validity of the scoring system because of the need to their ability to determine a performance-based indi- rely on a gut feeling: vidual score for comparison. CBS, however, provided an objective structure and allowed each candidate to be Needed to base the score on a ‘‘gut feeling,’’ which would assessed individually. Participants considered CBS to be difficult to justify if one had to justify the score you perform better as a whole and to produce a more accu- gave. RBS ends up being quite subjective. We had a group rate representation of candidates. of stronger candidates followed by a group of weaker candidates. The ranking of a mediocre student among a The literature supports the use of non–fact-based group of strong students would result in a low score as categories in making admissions decisions.19–21 Accord- compared to ranking a mediocre student among a group ing to the literature, the categories selected in this of weaker candidates. (Fe) case—professionalism, critical thinking, and communi- cation—are appropriate for the PT profession.22–28 The Some participants firmly believed that RBS did not findings of this study confirmed that the CBS categories assess everyone according to their abilities. The number are valid; however, they are also overlapping and inter- of candidates pooled—only eight per circuit—was brought related, especially communication and professionalism, up as a particular problem; eight is too small a number and this question merits further investigation. to do an accurate job.