fletcher2003

The Emerald Research Register for this journal is available at The current issue and full text archive of this journal is available at http://www.emeraldinsight.com/researchregister http://www.emeraldinsight.com/0268-3946.htm Assessing self-awareness: Assessing some issues and methods self-awareness Clive Fletcher 395 Goldsmiths College, University of London, London, UK, and Caroline Bailey University of Surrey, Guildford, UK Keywords Self-assessment, Performance, Feedback Abstract Multi-source processes have been increasingly adopted by organisations in recent years and most projections suggest this trend will continue. As a developmental technique, one underlying rationale to such systems is their potential impact on target managers’ self-awareness; increasing self awareness is thought to enhance performance. The main theme of this paper relates to the potential of 360-degree assessment for yielding measures of self-awareness and the different ways of deriving indices of this variable. The relationship between self-awareness indices and measures of performance are discussed in light of research findings. It is concluded that different self-awareness measures used in the research literature are not equivalent, and may have differential relationships to performance. It is argued that self-awareness should be assessed in selection and other settings using a variety of methods, not necessarily utilizing the kinds of measures typically associated with multi-source feedback systems. In recent years there has been a rapid increase in the popularity of multi-source Journal of Managerial Psychology multi-rater systems, sometimes referred to as 360-degree feedback systems. Vol. 18 No. 5, 2003 These usually involve a process whereby a target manager is rated on various pp. 395-404 behavioural dimensions or competencies by one or more bosses, peers, q MCB UP Limited subordinates and sometimes customers. The dimension ratings are usually 0268-3946 aggregated and compared against the target’s self-rating in a written report; the process may be paper-based or completed via specifically designed DOI 10.1108/02683940310484008 software packages. Multi-source multi-rater systems have chiefly been oriented towards the target manager’s development and have taken place in the context of management development or leadership courses. While they have not been in use for very long (they were first introduced to the UK in the early 1990s), 360-degree and other multi-source multi-rater feedback systems have spread quickly across a whole range of public- and private-sector organisations in the UK (Fletcher and Baldry, 2000). This widespread adoption of 360-degree feedback is based on the perceived benefits of such processes. These are: . It is inherently fairer and has greater accuracy in representing performance because it offers a more rounded assessment of the individual, not just the top-down perspective of conventional appraisal. . It is an empowering mechanism, in that it allows subordinates to exert some influence over the way they are managed; the same is true for peers,

JMP who can reflect back and perhaps alter the way a colleague performs as a 18,5 member of the team. 396 . It enhances awareness of the organisation’s competency framework, because staff completing the questionnaires may (depending on how the questionnaire is presented) become familiar with the competencies and the behaviours associated with each one. . It has powerful development and learning potential – if used sensitively and with the right kind of support mechanisms. The impact of this kind of feedback is, as anyone who has used it knows, quite strong and can motivate changes in behaviour. . It brings about a culture change in organisations, whereby individuals become more ready to seek, give and accept feedback in a constructive manner, and so enhance communication and openness. . It increases self-awareness, that is the extent to which an individual’s self-assessment of performance is congruent with how colleagues perceive that performance. It is the last point “facilitating self-awareness”, that multi-source multi-rater feedback is seen to make a fundamental contribution to performance outcomes for individuals. There is a general belief that increasing self-awareness will have a positive effect on performance. The extent to which self-ratings are congruent with the assessments made of the individual by others, has been taken as a measure of the degree to which individuals understand their own strengths and weaknesses. Evidence from various settings has demonstrated an association between self-awareness and managerial success and leadership effectiveness (Fletcher, 1997). High self-awareness has been found to be associated with higher performance ratings in various settings (Bass and Yammarino, 1991; Furnham and Stringfield, 1994; Atwater et al., 1998). Nasby (1989) found that individuals with high self-awareness are more able to incorporate comparisons of behaviour into their self-perception, and that their self-perceptions are both more reliable and valid. Conversely, people with low self-awareness are more likely to ignore or discount feedback about them, suffer career derailment and have negative attitudes towards work (Ashford, 1989). Other studies have found that high levels of self-other congruence are associated with good working relationships (Wexley et al., 1980) and more attentiveness to feedback (Wohlers and London, 1989). Moreover, it seems a universal phenomenon across industry, as indicated by Church (1997) and his investigation of self-other congruence as predictors of performance across a variety of organisational contexts. Measuring self-awareness Given the relationship between self-awareness and performance outcomes for the individual, establishing individual’s level of self-awareness (for example as

a criterion in initial selection, promotion or on-going appraisal) can be seen to Assessing have practical benefits for the organisation, in that it may enable predictions to self-awareness be made of that individual’s performance. Unfortunately, though, the measurement of self-awareness is not a straightforward matter; despite the 397 link between self-other congruence and performance being substantiated in empirical investigations across a range of organisational contexts, researchers have used different ways of representing self-other congruence. As yet there is no consensus among researchers in this field as how to best represent self-awareness conceptually or statistically (Baldry and Fletcher, 2000). To examine conceptual issues first, although there is general agreement for the broad definition of self-awareness (that self-awareness is the extent to which an individual sees themselves as others see him or her), within the context of self-awareness of competence (i.e. 360-degree and other developmental feedback systems) there are a number of ways this self-other congruence can be conceptualised. Common to most conceptualisations is the notion of having an index of self-other similarity, i.e. the measure will result in each individual having a self-awareness “score” ascribed to them. This is in contrast to very recent approaches to the study of self-awareness within the context of multi-source multi-rater feedback that have considered polynomial regression and within- and between-analysis techniques (see Atwater and Yammarino, 1997 for a review). However, it is important to note that these complex measures do not identify self-other congruence at the individual level, which is the issue of interest here. One way in which self-awareness has been conceived is that of the extent to which the self- and other-raters agree on the level of competence the focal individual (or “target”) attains. Using the rating-scale language commonly applied in multi-source multi-rater feedback, this is represented by the extent to which both the self and others agree the target is “effective” or perhaps has “a development need” for a particular behaviour, certain competency, or overall. This type of conceptualisation has traditionally been represented statistically by difference scores (Church, 1997) and in certain 360-degree feedback systems may be described as a “gap analysis”; within 360-degree literature this has also been referred to as “congruence-d” (Warr and Bourne, 1999). Congruence-d is obtained by subtracting the average “other” score from the self-rating for each feedback questionnaire item, and then dividing by the pooled standard deviation o fthose scores (Warr and Bourne, 1999). Level of self-awareness is signified by increasingly small “d” scores, with d ¼ 0 signifying complete agreement between self- and other-ratings for all items, i.e. “self-awareness”. Proponents consider this index to indicate both the extent to which the individual’s level of competence is, on average, seen as at the same level by others, as well as similarity in perceptions of relative strengths and weaknesses. However, despite being one of the most common ways in research literature for self-other congruence to be portrayed (Church, 1997), it is also the measure that, most recently, has attracted most criticism.

JMP Johns (1981) concluded in a review of the difference score that it was a 18,5 somewhat dubious paradigm, noting among a number of criticisms that “d” is potentially umeliable and without construct validity. Telling a manager they 398 have a certain “d” score is of no use unless we can provide information on what that amount means in real terms – if an organisation is keen for the individual to reduce the “gap” between the self-assessment and ratings given by others, more must be known about the significance of the size of gap as a predictor of certain outcomes. Edwards (1993, 1994) has also offered a critique of “d”, noting how, in its various guises (algebraic, absolute or squared differences), a difference score conceals the source of disagreement between self- and other-ratings (i.e. where in the feedback profile discrepancy lies for ratings of particular behaviours or competencies) and the type of difference (i.e. overestimation or underestimation by the focal individual). It is for these reasons that Atwater and Yammarino (1997) suggest researchers have had to rethink the statistical representation of self-awareness. The current alternative is to see self-awareness as the extent to which individuals agree on the relative strengths and weaknesses of the target individual. Research has generally found that, typically, individuals have a tendency to overestimate their own level of competence (Harris and Schaubroeck, 1988; Conway and Huffcut, 1997) – yet, while an individual may not share the same view of their current level of competence (as described above), as long as this overestimation (or underestimation) is consistent across all behaviours rated, agreement with others can be achieved for the pattern of competence. Again, using multi-source multi-rater feedback terms, this conceptualisation of self-awareness reflects the extent to which the target individual and their raters agree on the target’s least – and most effective – behaviours, rather than level of competence per se. This type of conceptualisation has been labelled within multi-source multi-rater literature as “relative self-awareness” (London and Wohlers, 1991; Wohlers and London, 1989) or “congruence-r” (Warr and Bourne, 1999), and is represented statistically by correlation between self- and other-ratings across all items comprising the feedback profile. Increasingly, large positive correlations are indicative of higher levels of self-awareness, small effect sizes and increasingly large negative correlations are indicative of lack of self-awareness. The main criticism of this conceptualisation lies in the fact this index can reflect simply the extent to which self- and other-ratings covary, rather than actual self-awareness. Consider the example of an individual who views their own competency as achieving or exceeding criterion levels of competence for all behaviours, whereas the manager’s colleagues view their competency as below the criterion for all behavioural indicators. The individual does not consider that they have any development needs whereas colleagues do not consider the individual to have any noticeable competency. However, if both the self- and other-ratings covary, it is possible

that a perfect correlation could be obtained between the self- and Assessing colleague-ratings, and the individual would be considered to be self-aware, self-awareness even though in real terms, one would say they are not. 399 Finally, the importance of incorporating the favourability of ratings in self- and other-assessments into an index of self-awareness has been raised by a small number of researchers. It has been noted in various models of self-assessment that having an accurate self-image (congruence with others) may not necessarily lead to positive outcomes, if that image is actually negative (Ashford, 1989; Atwater and Yammarino, 1997; London and Smither, 1995). Hence, the type of self-awareness that should be seen as the objective of developmental feedback is not simply where the self- and other-ratings are congruent, but that also ratings are favourable (i.e. the individual has few, if any, development needs). Yammarino and Atwater (1997) formulated a categorical measure of self-awareness that does just this, taking into account both the nature and degree of congruence. However, as with other profile similarity indices, this form of self-awareness has not generally been adopted by research (for reasons given previously). Some findings on the relationship of self-awareness measures to each other and to performance Having reviewed the two different indices of self-awareness mainly used by researchers (congruence-r and congruence-d), the issue then is, whether there is any kind of overlap between the measures, to what extent could they be interchangeable, or at least equally predictive of certain individual and organisational outcomes? From a practical standpoint, these issues are important for the facilitation of development through feedback, and the utilisation of self-awareness as a performance criterion in itself. For example, should one emphasise to the individual receiving feedback the difference in the favourability of their self-assessments to ratings ascribed by others (i.e. congruence-d) or whether different strengths and weaknesses are perceived (i.e. congruence-r)? If used as a criterion in selection, to what extent would congruence-r, for example, be a better predictor of candidate’s job performance than congruence-d ? No published research yet exists evaluating these indices in this way. However, the authors of this paper have investigated these issues in a field study conducted within a large private-sector organisation in the UK. The two different profile similarity index (PSI) measures of self-awareness (congruence-d and congruence-r) were computed relative to each external rating constituency (immediate line managers, direct subordinates and indirect subordinate staff) involved in the organisation’s developmental feedback system. Six indices were produced in total for each of the 104 target managers: self-awareness relative to each rater source (to line manager, to direct report and to indirect report) first by congruence-d then by congruence-r. These

JMP indices were examined in relation to each other, by “within-rater” comparisons 18,5 (e.g. congruence-d to report vs congruence-r to report), to determine the equivalence of the two measures; as correlates of favourability of feedback 400 received (higher self-awareness was expected to be positively associated to favourable feedback, given the link found in previous research between high self-awareness and higher levels of performance); and for predictive validity, as predictors of supervisor’s ratings ascribed in the organisation’s annual appraisal process (both across type of measure, and type of rater in self-other comparison) at three time points – the same year feedback ratings were ascribed, then two successive ratings provided at approximately 12-month intervals. Two performance criteria were assessed by the supervisor in the annual appraisal system: (1) PDR-Com – the extent to which the manager had met the requirements of the role as indicated by effectiveness ratings of “soft” management skills; and (2) PDR-Obj – the extent to which the manager had met performance objectives, as indicated by attainment of “hard” output targets. Comparing the relationship between self-awareness conceptualisations, a manager’s level of congruence-r was not found to be related to their level of congruence-d for either self-awareness to line manager nor for immediate reports; within all self-other comparisons congruence-r was only found to be significantly related to congruence-d for self-awareness to indirect subordinates, and even then the coefficient observed was small (r ¼ 0:27). From this it was concluded that self-awareness measures were largely not equivalent; if a person was found to have a high degree of self-awareness by congruence-d to their line manager or their immediate reports they would not necessarily also have a high level of self-awareness by congruence-r, a finding of some significance given the previous links made between self-awareness and performance. Correlating self-awareness indices for a particular rating constituency to the “average feedback rating” from that constituency, the link between higher self-awareness and higher performance was supported for direct and indirect subordinates ratings when self-awareness to these constituencies was operationalised as congruence-d. The relationship between congruence-d to immediate reports and the favourability of feedback from immediate reports was observed as r ¼ 20:28 (n ¼ 99, p , 0:01 one-tailed); the relationship between congruence-d to indirect reports and the favourability of their feedback was observed as r ¼ 20:40 (n ¼ 73, p , 0:001 one-tailed). It is important to note that, in the case of congruence-d, a negative correlation is indicative of positive association to performance (as higher self-awareness is signified by increasingly small “d” scores). No significant correlations were observed for any rater group for congruence-r. Thus it seems that the form of

self-awareness more relevant to performance is having a shared perception of Assessing one’s overall level of effectiveness, rather than holding an image of one’s own self-awareness relative strengths and weaknesses that is similar to others. 401 Correlation analyses indicated that self-awareness indices had limited association to supervisor’s annual appraisal ratings; by simple linear regression analyses, no support was found for their predictive validity. Only two self-awareness indexes were found to be associated to appraisal ratings, and only those appraisal ratings ascribed the same year as feedback ratings (thus providing no support for the long-term predictive validity of such measures). Moreover, one relationship was in the opposite direction to that expected: self-awareness to direct subordinates by congruence-d was found to be positively significantly related to appraisal ratings of the extent to which the individual met the requirements of the role (r ¼ 0:27, n ¼ 50, p , 0:05 one-tailed). So, contrary to the self-awareness-performance link hypothesised as a target’s self-awareness to their direct reports increased, supervisor’s annual appraisal ratings of that individual decreased. Only self-awareness to “boss” by congruence-r was found to be positively associated to the higher appraisal ratings of “extent to which objective performance targets were met” (r ¼ 0:29, n ¼ 57, p , 0:05 one-tailed), the greater target’s self-awareness in terms of similarity in perceptions of relative strengths and weaknesses between their self-assessment and boss’ feedback, the higher their performance. A possible interpretation of this finding is that agreeing with the boss’ view of you is likely to lead to a better appraisal! Overall though, it can only be concluded that self-awareness measures in this instance did not appear to have clear functional value as predictors of performance criteria. To the authors’ knowledge this study constitutes the only investigation currently available on these issues; as such, these inferences should be regarded tentatively and in need of further replication before conclusions are made. Moreover, given the problems known to be associated to supervisor’s appraisal ratings – leniency, halo and other idiosyncratic rating errors, etc. – it may be that findings here are indicative of weakness in the criteria rather than the predictors. Thus it is suggested that future research considers additional measures of performance such as rate of promotion or productivity. Conclusions On the basis of the findings briefly reported above, it seems that different ways of measuring self-awareness used in the research literature are neither equivalent nor necessarily related to formal organisational appraisal of performance. However, it should be kept in mind that one of the reasons for the emergence of multi-source feedback systems was the dissatisfaction with conventional, top-down appraisal as a way of reflecting performance levels (Fletcher and Baldry, 2000). Thus, as a criterion for measuring the relationship of self-awareness to performance, appraisal ratings leave something to be

JMP desired. That said, given the problems associated with the measurement of 18,5 self-awareness and the lack of guidance in multi-source multi-rater literature to date (even from a theoretical perspective) as to which measure has greatest 402 functional value, there is some utility in considering alternative ways and contexts in which to establish an individual’s level of self-awareness. Despite the capacity of 360-degree feedback systems for readily yielding indices of self-other congruence, it may be that measures of self-awareness derived from other assessment contexts – where there is greater similarity between the assessment criteria and the performance criteria – are better predictors of performance. Arguably, multi-source multi-rater developmental feedback is not about identifying present or future performance, but more about identifying the current competency profile, and there is, thus, a fundamental difference between the purposes and criteria involved. To take the context of selection, a number of techniques commonly employed could provide other direct measures of self-other congruence. For example: . by self-assessment in application forms, which could then be compared with structured references from previous employers (Jones and Fletcher, 2002); . in assessment/development centres, by comparing self-assessments against both assessor’s and peer’s ratings (Halman and Fletcher, 2000). Alternatively, there may be some utility in adopting an indirect approach to the measurement of self-awareness – by evaluating other individual differences that theoretically should relate to one’s level of self-awareness – rather than using “direct” measures of self-other rating congruence. For example: . by using interview questions to assess previous incidence of feedback seeking behaviour, capacity for self-criticism, emotional intelligence and awareness of others’ perceptions (Fletcher, 1997); . by psychometric assessment of personality traits and other individual differences related to self-awareness (Fletcher and Baldry, 2000). Such techniques would have the advantage of avoiding the complex statistical problems associated with current measures based on self-other rating congruence, may be seen to have a stronger theoretical rationale than statistical operationalisations and, as such, perhaps higher face-validity to those involved in such assessments. However, while different approaches to assessing self-awareness may be utilized, the general nature of the construct and of its relationship with performance also needs more exploration. The focus of this article has largely been on self-awareness defined in terms of the relationship of self-assessments to assessments made of others. It is possible to look at it different ways; for example, the extent to which an individual can accurately identify how others perceive them (i.e. by asking “how do you think your boss rates you on this competency”), irrespective of whether this accords

with their own self-view. In addition, more research is needed on the exact Assessing nature of the relationship of different forms or measures of self-awareness to self-awareness varying performance outcomes. References 403 Ashford, S.I. (1989), “Self-assessments in organizations: a literature review and integrative model”, Research in Organizational Behavior, Vol. 11, pp. 133-74. Atwater, L.E. and Yammarino, F.I. (1997), “Self-other rating agreement: a review and model”, Research in Personnel and Human Resources Management, Vol. 15, pp. 121-74. Atwater, L.E., Ostroff, C., Yammarino, F.I. and Fleenor, I.W. (1998), “Self-other agreement: does it really matter?”, Personnel Psychology, Vol. 51 No. 3, pp. 577-98. Baldry, C. and Fletcher, C. (2000), “The distribution and measurement of managerial self- awareness in the context of multi-source feedback: findings from a field study”, paper presented at the BPS Division of Occupational Psychology Annual Conference, Brighton. Bass, B.M. and Yammarino, F.I. (1991), “Congruence of self and others’ leadership ratings of naval officers for understanding successful performance”, Applied Psychology: An InternationalReview, Vol. 40 No. 4, pp. 437-54. Church, A.H. (1997), “Managerial self-awareness in high-performing individuals in organizations”, Journal of Applied Psychology, Vol. 82 No. 2, pp. 281-92. Conway, J.M. and Huffcut, A.I. (1997), “Psychometric properties of multisource performance ratings: a meta-ana1ysis of subordinate, supervisor peer and self-ratings”, Human Performance, Vol. 10 No. 4, pp. 331-60. Edwards, J.R. (1993), “Problems with the use of profile similarity indices in the studyof congruence in organizational research”, Personnel Psychology, Vol. 46, pp. 641-65. Edwards, J.R. (1994), “The study of congruence in organizational behavior research: critique and proposed alternative”, Organizational Behavior and Human Decision Processes, Vol. 58, pp. 51-100. Fletcher, C. (1997), “Self-awareness – a neglected attribute in selection and assessment?”, International Journal of Selection and Assessment, Vol. 5 No. 3, pp. 183-7. Fletcher, C. and Baldry, C. (2000), “A study of individual differences and self awareness in the context of multi-source feedback”, Journal of Occupational and Organizational Psychology, Vol. 73, pp. 303-19. Furnham, A. and Stringfield, P. (1994), “Correlates of self and subordinate ratings of managerial practices as a correlate of supervisor evaluation”, Journal of Occupational and Organizational Psychology, Vol. 67 No. 1, pp. 57-67. Halman, F. and Fletcher, C. (2000), “The impact of development centre participation and the role of individual differences in changing self assessments”, Journal of Occupational and Organizational Psychology, pp. 423-42. Harris, M.M. and Schaubroeck, J. (1988), “A meta-analysis of self-supervisor, self-peer and peer-supervisor ratings”, Personnel Psychology, Vol. 41, pp. 43-62. Johns, G. (1981), “Difference score measures of organizational behavior variables: a critique”, Organizational Behavior and Human Peiformance, Vol. 27, pp. 443-63. Jones, L. and Fletcher, C. (2002), “Self-assessment in a selection situation: an evaluation of different measurement approaches”, Journal of Occupational & Organizational Psychology, Vol. 75 No. 2, pp. 145-62.

JMP London, M. and Smither, J.W. (1995), “Can multi-source feedback change perceptions of goal 18,5 accomplishment, self-evaluations and performance related outcomes? Theory-based applications and directions for research”, Personnel Psychology, Vol. 48, pp. 803-39. 404 London, M. and Wohlers, A.J. (1991), “Agreement between subordinate and self-ratings in upward feedback”, Personnel Psychology, Vol. 44, pp. 375-90. Nasby, W. (1989), “Private self-consciousness, self-awareness and the reliability of self-reports”, Journal of Personality and Social Psychology, Vol. 56 No. 6, pp. 950-7. Warr, P. and Bourne, A. (1999), “Factors influencing two types of congruence in multi-rater judgements”, Human Performance, Vol. 12 No. 3/4, pp. 183-210. Wexley, K.N., Alexander, R., Greenwalt, J. and Couch, M. (1980), “Attitudinal congruence and similarity as related to interpersonal evaluation in manager-subordinate dyads”, Academy of Management Journal, Vol. 23, pp. 320-30. Wohlers, A.J. and London, M. (1989), “Ratings of managerial characteristics –evaluation difficulty, co-worker agreement and self-awareness”, Personnel Psychology, Vol. 42, pp. 235-61. Yammarino, F. and Atwater, L.E. (1997), “Do managers see themselves as others see them? Implications of self-other rating agreement for human resource management”, Organizational Dynamics, Spring, pp. 35-44. Further reading Antonioni, D. (1996), “Designing an effective 360 degree appraisal feedback process”, Organizational Dynamics, Vol. 25 No. 2, pp. 24-38. Cleveland, I.N. and Murphy, K.R. (1992), “Analyzing performance appraisal as goal-directed behavior”, Research in Personnel and Human Resources Management, pp. 121-85. Mathews, B.P. and Redman, T. (1997), “The attitudes of service industry managers toward upward appraisal”, Career Development International, Vol. 2 No. 1, pp. 46-53. Pollack, D.M. and Pollack, L.J. (1996), “Using 3608 feedback in performance appraisal”, Public Personnel Management, Vol. 25 No. 4, pp. 507-28. Redman, T. and Snape, E. (1992), “Upward and onward – can staff appraise their managers”, Personnel Review, Vol. 21 No. 7, pp. 32-46. Romano, C. (1994), “Conquering the fear of feedback”, HRFocus, Vol. 71 No. 3, pp. 9-19. VanVelsor, E., Taylor, S. and Leslie, J.B. (1993), “An examination of the relationships among self-perception accuracy, self-awareness, gender and leader effectiveness”, Human Resource Management, Vol. 32 No. 2-3, pp. 249-63.