Methodologies for educational research Controls and causality it is largely a matter of postulating hypotheses accord- ing to one’s personal preference. The investigator Other characteristics of ex post facto research become begins with certain data and looks for an interpretation apparent when it is contrasted with true experimental consistent with them; often, however, a number of research. Kerlinger (1970) describes the modus oper- interpretations may be at hand. Consider again the andi of the experimental researcher. (‘If x, then y’ in hypothetical increase in road accidents in a given town. Kerlinger’s usage. We have substituted X for x and O A retrospective search for causes will disclose half-a - for y to fit in with Campbell and Stanley’s (1963) con- dozen plausible ones. ventions throughout the chapter.) Kerlinger hypothe- Experimental studies, by contrast, begin with a spe- sizes: if X, then O; if frustration, then aggression. cific interpretation and then determine whether it is Depending on circumstances and his own predilections congruent with externally derived data. Frequently, in research design, he uses some method to manipulate causal relationships seem to be established on nothing X. He then observes O to see if concomitant variation, more substantial than the premise that any related event the variation expected or predicted from the variation occurring prior to the phenomenon under study is in X, occurs (see also Chapter 6). If it does, this is evi- assumed to be its cause – the classical post hoc, ergo dence for the validity of the proposition ‘if X, then O’. propter hoc fallacy (‘after this, therefore because of Note that the scientist here predicts from a controlled X this’); just because one variable precedes another in to O. To help him achieve control, he can use the prin- time, it does not follow that the first variable causes the ciple of randomization and active manipulation of X second: I may drink coffee and then have a sleepless and can assume, other things being equal, that O is night, but it does not follow that drinking the coffee varying as a result of the manipulation of X. caused the sleepless night – there may have been other In ex post facto designs, on the other hand, O is causes. Even when we do find a relationship between observed. Then a retrospective search for X ensues. An two variables, we must recognize the possibility that X is found that is plausible and agrees with the hypoth- both are individual results of a common third factor esis. Due to lack of control of X and other possible Xs, rather than the first being necessarily the cause of the the truth of the hypothesized relation between X and O second. cannot be asserted with the confidence of the experi- As mentioned earlier, there is also the real possibil- mental researcher. Basically, then, ex post facto inves- ity of reverse causation, for example, that a heart condi- tigations have, so to speak, a built-in weakness: lack of tion promotes obesity rather than the other way around, control of the independent variable or variables. As or that they encourage each other. The point is that the Spector (1993, p. 43) suggests, it is impossible to evidence simply illustrates the hypothesis; it does not isolate and control every possible variable, or to know test it, since hypotheses cannot be tested on the same with absolute certainty which are the most crucial data from which they were derived. The relationship variables. noted may actually exist, but it is not necessarily the The most important difference between experi only relationship, or perhaps the crucial one. Before we mental and ex post facto designs is control. In the can accept that smoking is the primary cause of lung experimental situation, investigators at least have cancer, we have to rule out alternative hypotheses. manipulative control; they have as a minimum one Further, a researcher may find that playing computer active variable. If an experiment is a ‘true’ experiment, games correlates with poor school performance. Now, they can also exercise control by randomization. They it may be there is a causal effect here: playing compu- can assign subjects to groups randomly; or, at the very ter games causes poor school performance; or there least, they can assign treatments to groups at random. may be reverse causality: poor school performance In the ex post facto research situation, this control of causes students to playing computer games. However, the independent variable is not possible, and, perhaps there may be a third explanation: students who, for more important, neither is randomization. Investigators whatever reason (e.g. ability, motivation), do not must take things as they are and try to disentangle do well at school also like playing computer games; it them, though having said this, they can make use of may be the third variable (the independent variable of selected procedures that will give them an element of ability or motivation) that is causing the other two control in this research. These we shall touch upon outcomes (playing computer games or poor school shortly. performance). By their very nature, ex post facto experiments can We cannot conclude from what has just been said provide support for any number of different, perhaps that ex post facto studies are of little value; many even contradictory, hypotheses; they are so flexible that important investigations in education and psychology 422
Experiments are ex post facto designs. There is often no choice in OO The PISA database (www.oecd.org/pisa/pis- the matter: an investigator cannot cause one group to aproducts; https://nces.ed.gov/surveys/pisa); become failures, delinquent, suicidal, brain-damaged or dropouts. Research must of necessity rely on existing OO The World Bank (statistics section); groups. On the other hand, the inability of ex post facto OO The TIMSS database (http://nces.ed.gov/timss/ designs to incorporate the basic need for control (e.g. through manipulation or randomization) makes them datafiles.asp). vulnerable from a scientific point of view and the pos- sibility of their being misleading should be clearly Advantages and disadvantages of ex post acknowledged. Indeed, ex post facto designs are proba- facto research bly better conceived more circumspectly, not as experi- ments with the greater certainty that these denote, but Among the advantages of the approach are the more as surveys, useful as sources of hypotheses to be following: tested by more conventional experimental means at a later date. OO ex post facto research meets an important need of the researcher where the more rigorous experimental Occasions when appropriate approach is not possible; Ex post facto designs are appropriate in circumstances OO the method yields useful information concerning the where the more powerful experimental method is not nature of phenomena – what goes with what and possible. These arise when, for example, it is not possi- under what conditions. Here ex post facto research ble to select, control and manipulate the factors neces- is a valuable exploratory tool; sary to study cause-and-effect relationships directly; or when the control of all variables except a single inde- OO improvements in statistical techniques and general pendent variable may be unrealistic and artificial, pre- methodology have made ex post facto designs more venting the normal interaction with other influential defensible; variables; or when laboratory controls for many research purposes would be impractical, costly or ethi- OO in some ways and in certain situations the method is cally undesirable. more useful than the experimental method, espe- Ex post facto research is particularly suitable in cially where the setting up of the latter would intro- social, educational and psychological contexts where duce a note of artificiality into research proceedings; the independent variable or variables lie outside the researcher’s control. Examples of the method abound OO ex post facto research is particularly appropriate here: the research on cigarette-smoking and lung when simple cause-a nd-effect relationships are cancer, for instance; or studies of teacher characteris- being explored; tics; or studies examining the relationship between political and religious affiliation and attitudes; or inves- OO the method can give a sense of direction and provide tigations into the relationship between school achieve- a fruitful source of hypotheses that can subsequently ment and independent variables such as social class, be tested by the more rigorous experimental ethnicity, gender and intelligence. Such investigations method. may be large scale or small scale ex post facto. For educational research, public domain databases Among the limitations and weaknesses of ex post facto and data sets can be used for conducting ex post facto designs are the following: educational research, for example, databases and data sets produced by: OO there is the problem of lack of control in that the researcher is unable to manipulate the independent OO government agencies (e.g. www.gov.uk/government/ variable or to randomize her subjects; statistics); OO one cannot know for certain whether the causative OO research agencies (e.g. www.data-a rchive.ac.uk); factor has been included or even identified; OO consortia (e.g. www.socsciresearch.com/r6.html); OO organizations, for example: OO it may be that no single factor is the cause; OO a particular outcome may result from different OO The OECD: http://stats.oecd.org/index.aspx; OO UNESCO (Institute for Statistics); causes on different occasions; OO when a relationship has been discovered, there is the problem of deciding which is the cause and which the effect; the possibility of reverse causation must be considered; OO the relationship of two factors does not establish cause and effect; OO the ex post facto hypothesis is generated after the data have been collected, so it is not possible to dis- confirm it (Babbie, 2010, p. 462); 423
Methodologies for educational research OO classifying into dichotomous groups can be innovation, loss of teacher status, decline in student problematic; motivation, modifications to the school catchment area or the appointment of a new headteacher. These could OO there is the difficulty of interpretation and the danger then be checked against a measure of prevailing teach- of the post hoc assumption being made, that is, ers’ attitudes (O), thus providing the researcher with believing that because X precedes O, X causes O; some leads at least as to possible causes of current discontent. OO as the researcher attempts to match groups on key Here the causal-c omparative model may be repre- variables, this leads to shrinkage of sample (Spector, sented schematically as: 1993, p. 43). (Lewis-Beck (1993, p. 43) reports an example of such shrinkage from a sample of 1,194 Group Independent variable Dependent variable to 46 after matching had been undertaken); E X O1 C O2 OO it often bases its conclusions on too limited a sample or number of occurrences; Using this model, the investigator hypothesizes the independent variable and then compares two groups, an OO it frequently fails to single out the really significant experimental group (E) which has been exposed to the factor or factors, and fails to recognize that events presumed independent variable X and a control group have multiple rather than single causes; (C) which has not. (The dashed line in the model shows that the comparison groups E and C are not equated by OO as a method it is regarded by some as too flexible; random assignment.) Alternatively, she may examine OO it lacks nullifiability and confirmation. two groups that are different in some way or ways and then try to account for the difference or differences by Designing an ex post facto investigation investigating possible antecedents. We refer the reader to Chapter 6 on effect-to-cause investigations. We earlier referred to the two basic designs embraced The basic design of causal-c omparative investiga- by ex post facto research – the co‑relational (or causal) tions is similar to an experimentally designed study. model and the criterion group (or causal-comparative) The chief difference resides in the nature of the inde- model. As we saw, the causal model attempts to iden- pendent variable, X. In a truly experimental situation, tify the antecedent of a present condition and may be this will be under the control of the investigator and represented thus: may therefore be described as manipulable. In the causal-c omparative model (and also the causal model), Independent variable Dependent variable however, the independent variable is beyond her X O control, having already occurred. It may therefore be described in this design as non-m anipulable. Although one variable in an ex post facto study cannot be confidently said to depend upon the other as Procedures in ex post facto research would be the case in a truly experimental investigation, it is nevertheless usual to designate one of the variables Ex post facto research is concerned with discovering as independent (X) and the other as dependent (O). The relationships among variables in one’s data; and we left to right dimension indicates the temporal order, have seen how this may be accomplished by using though having established this, we must not overlook either a causal or causal-c omparative model. We now the possibility of reverse causality. In a typical investi- examine the steps involved in implementing a piece of gation of this kind, then, two sets of data relating to the ex post facto research. We begin by identifying the independent and dependent variables respectively are problem area to be investigated. This stage will be fol- gathered. As indicated earlier, the data on the independ- lowed by a clear and precise statement of the hypothe- ent variable (X) will be retrospective in character and sis to be tested or questions to be answered. The next as such will be prone to the kinds of weakness, limita- step is to make explicit the assumptions on which the tions and distortions to which all historical evidence is hypothesis and subsequent procedures will be based. A subject. review of the research literature follows. This enables For example, imagine a secondary school in which the investigator to ascertain the kinds of issues, prob- it is hypothesized that low staff morale (O) has come lems, obstacles and findings disclosed by previous about as a direct result of school reorganization some two years earlier. A number of key factors distinguish- ing the new organization from the previous one can be identified. Collectively these could represent or contain the independent variable X and data on them could be accumulated retrospectively, for example, curricular 424
Experiments studies in the area. There then follows the planning of factors are, that is, the factors that may be related to the the actual investigation and this consists of three broad dependent variable. Further, there is the possibility of stages – identification of the population and samples; losing those subjects who cannot be matched, thus the selection and construction of techniques for collect- reducing one’s sample. ing data; and the establishment of categories for classi- As an alternative procedure for introducing a degree fying the data. The final stage involves the description, of control into ex post facto research, the researcher can analysis and interpretation of the findings. build the extraneous independent variables into the Drawing on Lord (1973, p. 6), we can set out several design and then use an analysis of variance technique. stages in conducting an ex post facto piece of research: For example, if intelligence is a relevant extraneous variable but it is not possible to control it through Stage 1: Define the problem and survey the literature. matching or other means, then it could be added to the Stage 2: State the hypotheses and the assumptions or research as another independent variable, with the par- premises on which the hypotheses and research proce- ticipants being classified in terms of intelligence levels. dures are based. Through analysis of variance techniques the dependent Stage 3: Select the subjects (sampling) and identify the variable measures would then be analysed to reveal the methods for collecting the data. main and interaction effects of intelligence, indicating Stage 4: Identify the criteria and categories for classify- any statistically significant differences or effect sizes ing the data to fit the purposes of the study and which between the groups on the dependent variable, even are as unambiguous as possible and which will enable though no causal relationship between intelligence and relationships and similarities to be found. the dependent variable could be assumed. Stage 5: Gather data on those factors which are always Yet another procedure which may be adopted for present in which the given outcome occurs, and discard introducing a measure of control into ex post facto the data in which those factors are not always present. design is that of selecting samples that are as homoge- Stage 6: Gather data on those factors which are always neous as possible on a given variable. For example, if present in which the given outcome does not occur. intelligence were a relevant extraneous variable, its Stage 7: Compare the two sets of data (i.e. subtract the effects could be controlled by including participants former (Stage 5) from the latter (Stage 6)), in order to from only one intelligence level. This would disentan- be able to infer the causes that are responsible for the gle the independent variable from other variables with occurrence or non-o ccurrence of the outcome. which it is commonly associated, so that any effects Stage 8: Analyse, interpret and report the findings. found could be associated justifiably with the independ- ent variable. One has to bear in mind that the evidence illustrates Finally, control may be introduced into an ex post rather than tests the hypothesis here (Lord, 1973, p. 7). facto investigation by stating and testing any alternative It was noted earlier that the principal weakness of ex hypotheses that might be plausible explanations for the post facto research is the absence of control over the empirical outcomes of the study. A researcher has to independent variable influencing the dependent variable beware of accepting the first likely explanation of rela- in the case of causal designs or affecting observed dif- tionships in an ex post facto study as necessarily the ferences between dependent variables in the case of only or final one. A well-k nown instance to which ref- causal-c omparative designs. Although the ex post facto erence has already been made is the presumed relation- researcher is denied not only this kind of control but ship between cigarette smoking and lung cancer. Health also the principle of randomization, she can neverthe- officials were quick to seize on the explanation that less utilize procedures that provide some measure of smoking causes lung cancer. Tobacco firms, however, control in her investigation; it is to some of these that put forward an alternative hypothesis – that both we now turn. smoking and lung cancer were possibly the result of a One of the commonest means of introducing control third factor, i.e. the possibility that both the independ- into this type of research is that of matching the sub- ent and dependent variables were simply two separate jects in the experimental and control groups where the results of a single common cause. design is causal-comparative. Matched pair designs are careful to match the participants on important and rele- 20.13 Conclusion vant characteristics that may have a bearing on the research (for an example of this, see Leow, 2009). This chapter has introduced a range of different types There are difficulties with this procedure, however, for of experiment. Starting with the randomized controlled it assumes that the investigator knows what the relevant trial, the ‘true’ experiment’, it held this up as the 425
Methodologies for educational research clearest example of a full experiment, as it abides by described, a design study, Internet experiments and all the features of an experiment that is intended to their limitations, and a full overview of ex post facto yield evidence of ‘what works’. The strengths and lim- research. itations of the true experiment and the randomized controlled trial were set out. Further variants of a true Notes experiment were set out. Moving further out of the laboratory and into the ‘real world’, the chapter then 1 Maynard and Chalmers (1997); Brown et al.’s (2011) study presented a discussion of different types of quasi- of bullying prevention; Slavin et al.’s (2009) study of a experiment, i.e. those kinds of experiment in which not middle school cooperative reading programme; Tracey et all the requirements of a true experiment were met or, al.’s (2010) study of cooperative learning’s effects on stu- in the case of ex post facto experiments or those which dents’ mathematics achievement; Madden et al.’s (2011) could not be justified on ethical or practical grounds, study of cooperative writing; Buckingham et al.’s (2012) where the requirements of a true experiment were study of a small group intervention for older low-p rogress impossible to meet. Rendering an experiment a quasi- readers; Jennings’s et al.’s (2013) study of cultivating experiment rather than a true experiment was seen to awareness and resilience in education; the list is endless. lie not only in design matters, but also in issues of sampling and controls. The chapter introduced design 2 Morrison (2001); Shadish et al. (2002); Maxwell (2004); experiments, or, as was argued to be more fittingly Hammersley (2005, 2008, 2015b); Biesta (2007, 2010b); Frueh (2009); Hutchison and Styles (2010); Sullivan (2011); Bouguen and Gurgand (2012); Cartwright and Hardie (2012); Goldacre (2013); Camburn et al. (2015). Companion Website The companion website to the book includes PowerPoint slides for this chapter, which list the structure of the chapter and then provide a summary of the key points in each of its sections. These resources can be found online at www.routledge.com/cw/cohen. 426
Meta-a nalysis, CHAPTER 21 systematic reviews and research syntheses Harsh Suri This chapter will explore: analyses have been conducted to examine the impact of individual interventions on student achievement. OO meta-a nalysis and the different stages of research Making rigorous comparisons across the findings of a synthesis number of meta-a nalyses is a complex endeavour for which useful guidelines have been provided (Hattie et OO systematic reviews al., 2014). Hattie’s visible learning series, based on the OO methodologically inclusive research syntheses synthesis of more than 800 meta-a nalyses (Hattie, 2009), has generated significant interest among teach- 21.1 Introduction ers and policy makers (see visible-learning.org). While meta-a nalyses facilitate comparisons across In contemporary educational research, most issues or quantitative studies by bringing them on a common interventions tend to be examined in a variety of con- metric called effect size, they are not suitable for syn- texts utilizing a diverse range of methodological thesizing qualitative research. In a seminal research approaches. Making sense of such complex domains of monograph, Noblit and Hare (1988) proposed meta- literature to inform policy, practice or further research ethnography as an appropriate method for synthesizing can be challenging for decision makers and practition- a small number of qualitative research reports selected ers. As evidence-b ased education gathers pace, research through the logic of purposeful sampling. Distinguish- syntheses are increasingly gaining prominence as valid ing features of their approach include an emphasis on methods for knowledge generation in their own right being ‘interpretive rather than aggregative’ (p. 11); (Suri, 2014). Most high-ranking educational research being inductive rather than using an a priori conceptual journals have become open to publishing quality framework; employing purposeful sampling rather than research syntheses. Many educational research journals exhaustive sampling for selecting primary research also specifically focus on quality research syntheses. studies; being consciously aware of one’s own subjec- These include Review of Educational Research, tivity; and paying attention to the target audience’s dis- Australian Education Review, Educational Research course (Noblit and Hare, 1988). The literature on Review, Research Synthesis Methods and Review of guidelines for rigorous synthesis of qualitative research Education. in education was relatively sparse until the early 2000s. Glass (1976) coined the term ‘meta-analysis’ to dis- In this century, excellent guidelines have been pub- tinguish between three forms of analysis: primary lished by educational researchers for publishing quali- analysis, secondary analysis and meta-a nalysis (p. 3). tative research (e.g. Major and Savin-B aden, 2010; Primary research involves collecting and analysing Suri, 2014). fresh data; secondary analysis involves re-a nalysing Following the popularity of systematic reviews of data collected for primary research to answer different research in medicine, evidence-based education and questions; and meta-a nalysis involves rigorous statisti- systematic reviews also became popular in education in cal integration of findings reported across a number of the last two decades. In the past two decades, several primary research studies. Meta-analysts employ explicit large-s cale centres have been established to support protocols to enhance consistency and objectivity systematic reviews (e.g. Campbell Collaboration, n.d.; through all stages. In the last four decades, popularity EPPI-Centre, n.d.). Early proponents of systematic of meta-a nalyses has grown exponentially (Glass, 2006; reviews in education were meta-a nalysts who used Cooper and Hedges, 2009). randomized controlled trials as the gold standard In popular areas of research, such as the effect of for rigour. In recent years a number of systematic different interventions on student achievement, Hattie reviewers have included qualitative and mixed methods et al. (2014) note that on average, one primary research research. report gets published every hour. A number of meta- 427
Methodologies for educational research Educational researchers espousing critical and inter- OO extracting relevant information from selected pretive orientations have made another important con- studies; tribution to methods of research synthesis by questioning the rhetoric of systematic reviews and OO integrating findings across studies; aggregative reviews and raising a concern over this OO presenting the findings as a scientific report. rhetoric which undermines the contributions of other forms of conceptual reviews (Hammersley, 2001; Meta-a nalysts have systematized the entire process of Clegg, 2005; Kennedy, 2007). research synthesis by identifying the main tasks in each Research synthesis is an umbrella term which phase, highlighting critical decision points within each includes a range of styles of bringing together into a phase and allowing discussion of the relative merits of single expert review or report several studies and sum- different choices at each decision point. They advocate maries on a particular topic. The evidence, methodo- explicit statement and justification of the decisions logical perspectives and techniques employed in a made at each stage of the research synthesis from research synthesis can be qualitative, quantitative or a hypothesis formulation, data selection, evaluation, anal- combination of both. The purpose of a research synthe- ysis and interpretation, to public presentation. There are sis is to produce new knowledge by making explicit also several types of sensitivity analyses that can connections and relations between individual reports examine the dependence of the findings on the assump- that were not visible before. It involves purposeful tions made about the nature of the data. Over the past selection, review, analysis and synthesis of previously three decades, meta-a nalysts have conducted numerous published reports on a similar topic to draw conclusions investigations to examine the robustness of their tech- that enable recommendations to be made for policy, niques and have explored ways of refining these tech- practice and further research. In a rigorous synthesis, niques, as well as examining many substantive uses in readers are provided with sufficient information about the field of education. Meta-a nalysis now has become the synthesis process so that they can make informed but one (important) method in integrative research decisions about the extent to which the synthesis find- synthesis. ings may be adapted to their own contexts (Cooper and Hedges, 2009; Suri, 2014). Formulating a problem This chapter introduces key issues in the fields of meta-a nalysis, systematic reviews and methodologi- Meta-a nalyses seek to discover both consistencies in cally inclusive research syntheses as part of the push similar-a ppearing primary studies and also to account towards evidence-b ased policy and practice in for the variability found between them, leading to gen- education. eralizations within the limits and contexts of the research studies used (Cooper and Hedges, 2009). The 21.2 Meta-analysis purpose of a meta-a nalysis is formulated in terms of a clear hypothesis with conceptual and operational defi- Early meta-a nalysts criticized intuitive narrative nitions of key constructs, independent variables and reviews for not being comprehensive in their coverage; dependent variables. Keeping in mind the intended overly relying on significance tests and subjective audience, the contextual variation covered within the judgements; being prone to Type II error and inconclu- scope of the synthesis is explicitly stated. Meta-analysts sive findings; overlooking the magnitude of the effect note that statistical significance is easier to achieve with sizes; and overlooking contextual variables that poten- large samples than with small samples. Hence, they tially moderate effect size (Jackson, 1980; Cook et al., integrate findings across a number of primary research 1992). studies examining a conceptually similar hypothesis, by To reduce unstated subjectivity in aggregating find- bringing them to a common metric called an effect size ings from a range of separate and disparate primary (Lipsey and Wilson, 2001). research reports examining a similar concept or inter- In education, for example, a large number of vention, meta-a nalysts recommend an explicit adher- primary research studies examines the effects of spe- ence to scientific rigour in each of the following stages cific interventions on student achievement through of research synthesis (Lipsey and Wilson, 2001; Glass, experimental or quasi-experimental designs. Meta- 2006; Cooper and Hedges, 2009): analyses are particularly suitable for synthesizing effect sizes across a range of contexts reported in different OO formulating a problem; studies examining a similar intervention, to estimate OO searching for relevant literature; the cumulative effect size, the associated confidence intervals and the potential moderators of the effect size (Hattie et al., 2014). 428
Meta-analysis, systematic reviews and research syntheses Searching for relevant literature Meta-a nalytic literature contains sophisticated dis- cussions of different types of effect sizes suitable for In a meta-analysis, the criteria for inclusion and exclu- different study designs; formulae that allow conversion sion of primary research are explicitly stated. Compre- between different effect size indices; guidelines for hensive searches are conducted for relevant studies estimating an effect size when some information is not with explicit delineation of search protocols. While reported in the primary research report; and appropri- some meta-a nalysts argue that studies with relatively ateness of various effect indices for analysing different weak study designs ought to be excluded (Slavin, types of data (Borenstein, 2009; Fleiss and Berlin, 2008), others insist that all research reports which meet 2009). the substantive selection criteria ought to be included in the synthesis, and an empirical examination should then Integrating findings across studies be conducted of how different study design features moderate the effect size (Glass et al., 1981). Reasoning that findings from studies with large samples Meta-a nalysts have identified a number of publica- are more precise, often meta-a nalysts compute the com- tion biases and search biases along with strategies for posite effect by averaging the relevant d-s tatistics taking these biases into account. A publication bias weighted by the reciprocal of their respective variances. exists when the chances of a study being published Confidence intervals associated with the cumulative depends on the nature of its methodological orientation effect are also computed (Hedges and Olkin, 1985). or findings. A search bias exists when certain types of Within each category of conceptually similar effect studies are more likely to be retrieved through common sizes, the homogeneity statistic between the studies search channels, such as key databases. Studies with (QB) is estimated by assuming that QB has an approxi- large sample sizes are more likely to attract research mate chi-s quare distribution with m − 1 degrees of funding, being submitted for publishing and getting freedom, where m is the number of studies within each published in reputable journals. Research that does not category. In rare cases, when the QB value is non- report marked differences between individual groups or significant (indicating a consistency of outcomes across sub-groups examined within a study is less likely to be studies), the composite effect size is taken as a conclu- published (Rothstein et al., 2004). ‘Sub-g roup reporting sive result representative of the within-category find- bias’ exists when several sub-g roups are compared but ings. However, often the QB value is significant, which only comparisons with interesting or statistically signif- indicates a considerable inconsistency across findings. icant findings get published. Similarly, ‘time-lag bias’ In these cases, the composite effect size does not ade- exists when certain types of studies, such as those with quately describe the studies, since the magnitudes and large sample sizes or effect sizes, take less time to get perhaps the directions of the findings are very different published (Sutton, 2009, p. 448). Unaccounted publica- from each other. These categories are analysed further tion biases and search biases can lead to Type I error, to account for the differences in individual outcomes. leading to erroneous reporting of a large overall effect. At this stage of the analysis, an outlier diagnosis is performed by visually plotting all the conceptually Extracting relevant information from similar effect sizes to identify any effect size that mark- selected studies edly differs from the remaining effect sizes. If the study design of the outlier markedly differs from all the Meta-analysts follow explicit procedures for extracting remaining studies, then the outlier is isolated and the relevant information from each study by developing difference in study design noted as a potential modera- protocols for coding contextual and outcome variables. tor of the effect size. Findings of individual studies are then converted to The remaining studies are subjected to categorical a common metric called an effect size, typically model testing, which is analogous to analysis of vari- expressed as the difference of means between the ance (ANOVA), to account for the heterogeneity of experimental and control groups divided by the stand- outcomes of different studies by identifying potential ard deviation. Algebraically, moderators of the effect. The studies are divided into sub-g roups based on a study characteristic. Within each (Me – Mc)/SD class, composite effect size and the within-g roup homo- geneity statistic, QW, is estimated by assuming QW to where Me is the mean of the experimental group, Mc is have an approximate chi-s quare distribution with k − 1 the mean of the control group and SD is the pooled degrees of freedom, where k is the number of studies standard deviation. An effect size of d = 0.0 is indicative within each sub-group. A non-s ignificant QW value of no change, while an effect size of d = 1.0 is typically indicates consistency of outcomes within a class. The regarded as a blatantly obvious change (Cohen, 1988). 429
Methodologies for educational research between-g roup homogeneity statistic (QB) is also esti- findings using appropriate statistical and visual tech- mated where a significant QB indicates that the study niques; and finally they interpret and contextualize their characteristic under consideration significantly moder- findings (Cooper and Hedges, 2009). ates the effect size (Hedges and Olkin, 1985). Meta-a nalysts sometimes conduct sensitivity analy- 21.3 Systematic reviews ses to ‘assess robustness and bound uncertainty’ (Orwin and Vevea, 2009, p. 196). In this procedure, the meta- In an age of evidence-b ased education, systematic analyst constructs connected understandings from the reviews are increasingly used methods of investigation, same data by making different assumptions about the bringing together different studies to provide evidence data, including the missing data. The relative match to inform policy making and planning (Gough et al., between these constructed understandings demonstrates 2012). Several large centres have been established to the degree to which these understandings are dependent support production and dissemination, for example: on the initial assumptions made about the data. For instance, meta-a nalysts often compute inter-rater relia- OO the EPPI-C entre (Evidence for Policy and Practice bility in terms of multiple measures of inter-rater agree- Information and Co-o rdinating Centre) at the Uni- ment; isolate outlier cases; isolate variables with low versity of London (http://eppi.ioe.ac.uk/cms); confidence ratings; use different formulae for comput- ing or transforming effect sizes (e.g. Borman et al., OO the Social, Psychological, Educational and Crimino- 2003); and compare generalizations to the study sample logical Controlled Trials Register (SPECTR) based on a fixed effects model with generalizations to a (Milwain, 1998; Milwain et al., 1999), later trans- large population based on a random effects model (e.g. ferred to the Campbell Collaboration (www.campbell- Sirin, 2005). Meta-analysts employ various strategies collaboration.org), a parallel to the Cochrane to examine a potential publication bias (Sutton, 2009), Collaboration in medicine (www.cochrane.org), which such as: funnel plot method of plotting sample sizes of undertakes systematic reviews and meta-a nalyses of, individual studies against their effect sizes; computing typically, experimental evidence in medicine; a fail-s afe n (Rosenthal, 1980) to estimate the number of studies with insignificant findings that would have to OO the Curriculum, Evaluation and Management (CEM) be added to the analysis to make the cumulative effect centre at the University of Durham (www.cemcen- size insignificant. Sensitivity analyses facilitate ratifica- tre.org); tion and validation of conclusions geared for consensus and convergence. OO the What Works Clearinghouse in the US (http://ies. There are several useful and cost-e ffective software ed.gov/ncee/wwc); programs for meta-a nalysis which offer many features including user-friendly menus to input information OO Centre for Reviews and Dissemination (www.york. about an individual study’s characteristics and data for ac.uk/crd); and computing effect sizes; compute and transform individ- ual effect sizes; calculate cumulative effect sizes after OO Best Evidence Encyclopedia (www.bestevidence. appropriately adjusting relevant effect sizes; conduct org). homogeneity analyses, outlier diagnoses and categori- cal testing for identifying moderator variables; and Like meta-a nalyses, systematic reviews use several conduct sensitivity analyses for comparing results techniques to minimize bias. They follow explicit pro- based on different assumptions and analytic paths. See tocols and criteria for searching for relevant primary, Shadish (n.d.) for a list of commonly used software for usually empirical studies, for example: their inclusion meta-a nalysis. and exclusion; the standards for acceptable methodo- logical rigour; their relevance to the topic in question; Presenting the findings as a scientific report the scope of the studies included; team approaches to reviewing in order to reduce bias; the adoption of a Meta-a nalysts typically employ the scientific reporting consistent and clearly stated approach to combining format with an explicit discussion of the critical deci- information from across different studies; drawing sion points in the process. It typically has four distinct careful, relevant conclusions and recommendations sections, i.e. Introduction, Methods, Results and Dis- (Evans and Benefield, 2001). cussion. The meta-analyst begins with an identification In addition, systematic reviewers make noteworthy and contextualization of the problem; describes their contributions to the methodology of research by embed- attempts to find the solutions; then they describe their ding the following features in systematic reviews (Gough et al., 2012): OO according a greater agency of control to stakehold- ers in making decisions about how the synthesis should proceed; 430
Meta-analysis, systematic reviews and research syntheses OO intention to regularly update their reviews with the resource-intensive and may not be viable. To improve new relevant studies in the field; efficiency, some systematic reviewers recommend starting with an a priori conceptual framework and OO intention to reduce duplication through explicit modifying it as new themes emerge from the data international collaborations; (Carroll et al., 2011; Dixon-W oods, 2011). Some systematic reviewers argue that in preference OO providing methodological support to groups inter- to quantitative or qualitative research syntheses, often ested in conducting systematic reviews; ‘mixed methods research syntheses’ are more suitable for providing ‘more complete, concrete and nuanced OO development of useful databases of intervention answers’ to complex synthesis questions (Heyvaert, studies and systematic reviews to facilitate their dis- 2013, p. 671). Particularly noteworthy is Pawson’s semination and access; and (2006) method of ‘realist synthesis’ to develop theory from successful as well as non-s uccessful implementa- OO utilizing technology strategically to update and dis- tions of a programme. Rather than making global gen- seminate relevant information widely. eralizations, realist reviews seek to explain how different aspects of a programme are likely to work in There are two methodologically distinct perspectives different circumstances. The reviewer begins by identi- prevalent among systematic reviewers. The first group fying the key theories underlying the specific phenome- is dominated by meta-analysts who recommend that non to formulate a more refined theory. Then the ideally a systematic review must hold randomized con- reviewer applies this theory successively to explain a trolled trials (RCTs) as the gold standard for individual number of successful and unsuccessful cases. With studies (e.g. Campbell Collaboration, n.d.). The second each application, the reviewer refines the theory. The group of systematic reviewers have engaged with salient features of Pawson’s method include: purpose- various issues associated with including qualitative and ful sampling; including studies with diverse qualitative mixed methods research in systematic reviews: devel- and qualitative designs; involvement of stakeholders in oping efficient search strategies; appraisal criteria and identifying the purpose; and tentative findings which synthesis techniques for research from different meth- inform decision makers of the likely implications of odological traditions; engaging various stakeholders in different decisions made in different situations rather formulating and critiquing their reviews’ questions, than what works (Pawson, 2006). protocols and summary reports; and stressing that sys- tematic reviews must be complemented with other 21.4 Methodologically inclusive forms of reviews to facilitate informed decision making research syntheses by different stakeholders (e.g. Petticrew and Roberts, 2006; Pope et al., 2007; Gough et al., 2012). Meta-a nalyses and systematic reviews have made a Since the mid-1 990s, there has been a growing inter- substantial contribution towards advancing methods of est in systematic reviews of qualitative studies, espe- research synthesis and have their own domains of cially in the areas of health care and public policy. A applicability. Nonetheless, many systematic reviewers variety of methods have been proposed for synthesizing exclude a large proportion of research on the grounds qualitative research from interpretive and critical-realist of poor methodological quality, using evaluation crite- perspectives which vary along several dimensions ria that are biased against certain paradigmatic orienta- (Barnett-P age and Thomas, 2009). Some methods have tions. Such an unacknowledged bias raises serious been developed to facilitate a fuller understanding of a questions about the validity and generalizability of phenomenon (Jensen and Allen, 1996) but others are review findings (Pawson, 2006). Even in their inclusion aimed at generating mid-range theory (Eastabrooks et of qualitative research, systematic reviewers often al., 1994; Zimmer, 2006) or ‘lines-of-action’ (Hannes include only interpretive qualitative research and seek and Lockwood, 2011a, p. 1633) to inform practical ideologically neutral evidence. The rhetorical effect decision making. While some systematic reviewers rec- of terms like ‘evidence-b ased practice’, ‘systematic ommend purposeful sampling for selecting studies, reviews’, ‘clarity’, ‘comprehensive’, ‘reliable’, ‘object others recommend comprehensive searches and inclu- ivity’, ‘replicable’ not only might discredit opposition, sion criteria. Some recommend including epistemologi- but also may have the political impact of favouring cally similar qualitative studies in a synthesis; others post-positivism. Ironically, these key terms that are recommend including studies from diverse epistemolo- associated with systematic reviews are operationalized gies. And while many qualitative systematic reviewers differently by different groups of systematic reviewers. recommend a grounded theory-like approach of axial coding for identifying themes emerging across studies, others note that a grounded theory-like approach to synthesizing research can sometimes become very 431
Methodologies for educational research The problem here is not the subjectivity associated with Suri (2014) has identified the following three these terms, but the systematic reviewers’ denial of general guiding principles for a quality research subjectivity itself (Hammersley, 2004). synthesis: Many systematic reviewers uncritically value objec- tivity and transparency of process, a priori protocols 1 informed subjectivity and reflexivity; and exhaustive searches. Accordingly, advantages of 2 purposefully informed selective inclusivity; emergent synthesis designs and purposeful sampling 3 audience-a ppropriate transparency. are less discussed in this body of literature. In reality, transparency itself is always subjective, partial and pur- Noting that each guiding principle will be enacted dif- posefully informed, where each way of showing is mir- ferently depending on the overarching epistemological rored by a way of concealing, which may or may not be and teleological orientation of the synthesis, Suri deliberate. Prescribing a priori rules to enhance objec- (2014) has clustered key decisions associated with the tivity, transparency and clarity could reduce the quality process of a research synthesis into six phases: of reviews by discouraging reflection on important process decisions (MacLure, 2005; Kennedy, 2007). 1 identifying an appropriate epistemological orientation; Hammersley (2003) observes that the phrase systematic 2 identifying an appropriate purpose; reviews makes the other forms of reviews appear 3 searching for relevant literature; ‘unsystematic’, which can be misleading. Rather than 4 evaluating, interpreting and distilling evidence from categorizing reviews as ‘systematic/unsystematic’, he urges the educational research community to develop selected reports; an appropriate typology for distinguishing between 5 constructing connected understandings; reviews with different foci (p. 5). 6 communicating with an audience. Recognizing the need to move beyond post- positivist reviews that often focus on constructing a In the remainder of this chapter, key decisions associated coherent understanding of a field, there have been calls with each of these phases of research synthesis are dis- for reviews that challenge prevalent understandings, cussed from a methodologically inclusive perspective. illuminate variations across contexts and highlight the tensions inherent in our understanding about a phenom- Phase one: identifying an appropriate enon (Eisenhart, 1998); reviews which reflexively inter- epistemological orientation rogate the inequalities associated with educational research and practice (Meacham, 1998); and post- There is no best-fit orientation for all research synthe- structural reviews that focus on constructing multiple ses. The overarching orientation of the synthesis ought understandings with a critical awareness that any to be guided by the anticipated utility of the synthesis, understanding is inherently partial and situated within a the nature of primary research in the field and the syn- particular perspective and time (Lather, 1999). thesist’s methodological expertise. The synthesist must In critiquing the hegemonic dominance of systematic attempt to make explicit the reflexive relationship reviews, critical scholars alert us to several contentious between the synthesis findings and the synthesist’s own issues, such as the problematics of formalization and research disposition. In Table 21.1, an illustrative systematization of research synthesis processes (e.g. framework of four epistemological orientations is used Gallagher, 2004; MacLure, 2005), and power issues to demonstrate how syntheses with different paradig- influencing how evidence is used (Clegg, 2005, p. 425). matic orientations can serve varied, albeit equally Many of these criticisms apply to most formal research useful, purposes. synthesis methods and not just systematic reviews. The first row in Table 21.1 illustrates distinct onto- Suri (2014) draws upon the diverse literature on logical positions of a synthesis. The second row illus- primary research methods and research synthesis trates different purposes that a synthesis can serve. The methods to expand possibilities within research synthe- third row illustrates potential relationships that a syn- sis methods from a perspective of methodological thesist can have with participants and authors of inclusivity. Noting that contemporary educational primary research. Evidence included in a research syn- research is marked by diversity, complexity and rich- thesis is interpreted and represented first by the partici- ness of purposes, methods and perspectives, she illumi- pants in primary research, then by the authors of nates the variety of ways in which we can accommodate primary research and finally by the research synthesist. and reflect such variety and complexity at the level of Both participants and authors of primary research serve synthesizing educational research. as informants for a research synthesist. The fourth row of Table 21.1 illustrates a range of strategies for search- ing and distilling relevant evidence and constructing 432
Meta-analysis, systematic reviews and research syntheses TABLE 21.1 RESEARCH SYNTHESES WITH DIFFERENT EPISTEMOLOGICAL ORIENTATIONS Post-positivist syntheses Interpretive syntheses Participatory syntheses Critical syntheses Ontological Objective factual world World is constructed Individuals and groups Relativistic and through meanings that construct their own transitional world views position is out there individuals and groups world views through reflective of dominant attribute to events participation power structures Amenable Objectively explain, Construct deeper and Understand and/or Problematize prevalent purposes predict or describe in more comprehensive improve ourselves and metanarratives to terms of probabilistic, understanding about our local world deconstruct and/or Informant– generalizable laws, phenomena as experientially through transform dominant synthesist facts or relations experienced critical engagement discourses relationship between measurable subjectively by different constructs and stakeholders variables Sensitive and reflective Critical, selective and Self-doubting and Objective distancing of understanding with creative understanding, reflexive understandings an unbiased expert minimal power emphasizing realistic of perspectives imbalance transferability to inform represented in, and local practice missing from, primary research literature Common Exhaustive sampling; a Purposive sampling; Purposive sampling; Openly ideological, strategies priori protocol and dialogic, dialectic coding sheets; emergent design; holistic emergent design; selection and analysis of statistical variable- evidence, emphasis on oriented analysis case-oriented analysis; eclectic data analysis; historical and structural insights summary sheets, meta- emphasis on practical matrices, reciprocal and experiential translations, etc. knowledge Quality Validity and reliability Deep and authentic Empower participants to Catalytic validity or criteria understanding Scientific reporting improve locally crystallization Suitable format genres Comprehensive Interactive reporting Nuanced texts narrative with thick celebrating descriptions intertextuality Source: Adapted from Suri (2013, p. 892) connected understandings from the distilled evidence. Post-p ositivist syntheses: Often post-p ositivist syn- The fifth row illustrates various quality criteria suitable thesists, like meta-a nalysts, seek to synthesize research for evaluating syntheses. The last row illustrates objectively with minimal researcher bias, by designing common genres that synthesists could employ to com- a priori synthesis protocols to minimize biases intro- municate with their audiences. duced by the synthesist’s subjective preferences; defin- The four columns of Table 21.1 illustrate distinc- ing conceptually and operationally all key constructs in tions between research syntheses aligned with four dis- behavioural terms at the outset; and employing exhaus- tinct epistemological orientations: post-positivist tive sampling in order to be representative of the entire syntheses; interpretive syntheses; participatory synthe- population of studies. Sometimes post-positivist syn- ses; critical syntheses. Boundaries between different thesists blind primary research reports to reduce biases orientations are blurred and rigid adherence to any introduced in judging the quality of individual reports single perspective is neither prescribed nor recom- by preconceived notions about the source of the publi- mended. Nonetheless, synthesists should be critically cation or the author of the individual primary research aware of the implications of the choices they make, report. Also, they measure inter-rater reliability to where some of these choices are likely to involve judge the degree of objectivity and reliability associ- drawing from more than one paradigm. ated with the key decisions in the synthesis process 433
Methodologies for educational research (Lipsey and Wilson, 2001; Orwin and Vevea, 2009; Participatory syntheses: Participatory synthesists Wilson, 2009). encourage critical thinking through engaged participa- Post-positivist synthesists commonly employ variable- tion with those whose practices and experiences are oriented statistical analyses to reduce Type II error and being researched. A complementary collaborative to enhance objectivity in the process of analysis and model, where the distinct skill sets and expertise of indi- synthesis; target global decision makers and research- vidual collaborators are valued, is suitable for encourag- ers as their audience; utilize the scientific reporting ing participation of different stakeholders in a research format; and adapt Cook and Campbell’s (1979) con- synthesis, without burdening them with a heavier work- structs of validity and reliability to address issues of load (Ritchie and Rigano, 2007; Yu, 2011). Rather than rigour in research synthesis. Sophisticated discussions ironing out the differences, a participatory synthesis have been published about ways of reducing threats to involves paying careful attention to learning opportuni- internal validity, external validity, internal reliability ties that arise from the differences in language, perspec- and external reliability within post-positivist syntheses tives and experiences of individual co-synthesists (e.g. Petticrew and Roberts, 2006; Matt and Cook, (Paugh and Robinson, 2011). A participatory synthesis 2009; see also Chapter 14 of the present volume). can become a site for teachers where they collabora- Interpretive syntheses: In the last two decades, inter- tively reflect on their own practice using published pretive syntheses have been discussed under various research as a mirror to develop ‘actionable knowledge’ names, such as meta-ethnography (Noblit and Hare, about their own practice (Torrance, 2004, p. 198). 1988), exploratory case-study oriented review of multi- Participatory synthesists value practical experience, vocal literatures (Ogawa and Malen, 1991), cross-case local knowledge and serendipitous leaps of intuitive analysis (Miles and Huberman, 1994), aggregated anal- understanding. Such participants could be: the authors ysis (Eastabrooks et al., 1994), meta-a nalysis of quali- of the primary research reports being synthesized; tative research (Jensen and Allen, 1994), qualitative members of stakeholder groups who participated in meta-s ynthesis (Zimmer, 2006), interpretivist-o riented those studies; or stakeholders wishing to engage criti- reviews (Eisenhart, 1998), meta-s ynthesis (Bair, 1999), cally with the literature to inform their own decisions. meta-s tudy (Paterson et al., 2001), thematic synthesis Academic synthesists collaborate with these partici- (Thomas and Harden, 2008) and framework synthesis pants in order to co-s ynthesize the relevant body of (Carroll et al., 2011). research through a process of reciprocal learning and Contesting an objective reality that is out there, co-constructing connected understandings. A participa- interpretive synthesists hold that the world is socially tory synthesis of action research reports authored by constructed in terms of the meanings we attribute to teacher-researchers or reflexive practitioners, on how events. Typical questions addressed by an interpretive they effected changes within their contextual con- synthesist include: How do different stakeholders in straints, can provide useful information to policy different contexts experience a phenomenon? How do makers and other practitioners. Identifying patterns the contextual particularities interact with the percep- across these individual reports can provide useful input tions of different groups and individuals? How do indi- from this group of action researchers towards theory- vidual primary research reports on a topic reinforce, building. contradict or augment each other? A participatory synthesis can involve: cycles of Interpretive synthesists begin by acknowledging the reflection to formulate synthesis purpose; conducting the tacit knowledge, values and experiences that they bring research synthesis; implementing changes as suggested to the synthesis process. They recognize that each by the implications of the synthesis; evaluating the primary research report is the author’s interpretation of implemented change and comparing these evaluations the research participants’ interpretation of the phenom- with the relevant research literature. Using emergent, enon being studied. By engaging in iterative negotia- pragmatic and eclectic designs, participatory synthesists tions between multiple meanings constructed at each can employ purposeful sampling strategies for selecting layer of interpretation and representation, they try to studies which illuminate aspects of a phenomenon that reveal the multiple perspectives of different stakehold- are of immediate interest to the participant co- ers with a sensitive understanding. They seek evidence synthesists. They can use the delphi-technique for col- that contests, reinforces or augments their emerging lecting, analysing and building collective understandings understanding of the phenomenon. Refraining from the of research to involve a homogeneous or heterogeneous tendency to construct one coherent grand metanarra- group of participants with a common interest in the tive, they remain open to constructing multiple under- research topic. Participatory synthesists can construct standings of the phenomenon (Paterson et al., 2001). critical, selective and creative understandings with 434
Meta-analysis, systematic reviews and research syntheses realistic transferability to inform practice in local con- prioritized? Whose questions have received little atten- texts of the participants. They can employ an interac- tion from primary researchers? How are the answers to tive reporting format to encourage a participative such questions intertwined? audience. The synthesis can be evaluated in terms of Critical synthesists could construct self-d oubting the progress in thinking and transformation of the con- and reflexive understandings of not only the perspec- texts of the individuals and the communities of those tives represented in the primary research literature but engaged in the synthesis process (see also Chapter 3 of also those missing from the published primary research the present volume). A practical example of some of to illuminate how some groups have become invisible these ideas can be seen in the work of Bassett and in the field with little representation. Critical synthesists McGibbon (2012), who designed a critical, participa- can also collaborate with the groups who have been rel- tory and collaborative method for scoping literature atively silenced in the primary research in order to with individuals who had been actively drawing atten- identify how the body of primary research has failed tion to barriers to health and well-b eing in rural, Abo- to adequately represent their interests (see, e.g., riginal and African Canadian communities in Canada. Warschauer and Matuchniak, 2010). Rather than defer- Critical syntheses: Critical synthesists can reveal ring to the authority of the author, postmodernist criti- how certain forms of knowledge get privileged and reg- cal synthesists would recognize an author as someone ulate the norms within discussions of policy, practice who is in the process of making sense, a sense which is and research in a field (Aronowitz and Giroux, 1991; partial and temporal (Lather, 1999; Richardson, 2001). Clegg, 2005; see also Chapter 3 of the present volume). By paying attention to the presence and absence of Phase two: identifying an appropriate various issues in the primary research reports, critical purpose synthesists can disrupt conventional thinking to con- struct spaces for new ways of talking about policy, An emphasis on purposefully informed selective inclu- practice and research (Eisenhart, 1998; Segall, 2001; sivity necessitates that synthesists carefully identify a Kress, 2011). A good example of a critical synthesis is purpose that takes into account: Windschitl’s theoretical analysis of research to illumi- nate the uncertainties and tensions experienced by OO potential stakeholders and collaborations; teachers along with the compromises they make as they OO the nature of the substantive area; implement a constructivist pedagogy in practice (Wind- OO the intended audience and utility; schitl, 2002, p. 131). OO pragmatic constraints; Through critical interrogation of the very text being OO ethical considerations. synthesized and by constantly questioning the assump- tions we make in constructing our understanding, criti- All these factors might influence, and be influenced by, cal synthesists could open new ways of thinking about the synthesist’s contextual positioning and the over- a phenomenon. They can challenge the prevalent under- arching epistemological, theoretical and political orien- standing by revealing the errors and elements of igno- tations of the synthesis. rance underpinning the prevalent understanding of a A number of groups can differentially influence or phenomenon. Rather than forcibly constructing a single be affected by a research synthesis, including the fol- coherent understanding of a phenomenon, critical syn- lowing: learners, families of learners, educators and thesists could illuminate multiple understandings of a educational institutions, primary researchers in the sub- phenomenon to expand possibilities for practitioners stantive area, policy makers, the wider community, (Sholle, 1992). Postmodern synthesists, as a sub-g roup commercial and political groups with an interest in the of critical synthesists, could disrupt and problematize topic. Funding agencies, editorial boards and communi- the metanarratives in a research domain in order to ties, and professional synthesists can also feature in enhance multiple discourses that celebrate diversity and these groups. inclusivity by refusing to provide simplistic explana- Stakeholders in a synthesis are often anticipatory tions for complex phenomena (Lather, 1993). rather than retrospective. Synthesists can purposefully Examples of questions addressed by critical synthe- collaborate with diverse stakeholders to achieve some sists could include the following: What are the gaps in of the following objectives: our understanding of a phenomenon? What methodolo- gies or theoretical perspectives are likely and/or OO encourage syntheses that address the concerns of a unlikely to be employed by primary researchers in the wide range of stakeholders; field? In the published literature, whose questions are OO facilitate syntheses informed by the perspectives of different groups; 435
Methodologies for educational research OO empower members of different groups by facilitat- equally legitimate but tailored to serve different ing their participation in syntheses which may be of purposes. Synthesists must search strategically for the interest to them; relevant evidence to meet the synthesis purpose effi- ciently within the available resources and pragmatic OO enhance the impact of a synthesis by promoting par- constraints. ticipation of the agents of change who are crucial in Several publication biases and search biases can implementing the recommendations made by the influence funding, publishing and visibility of certain synthesis; types of primary research as well as research synthesis (Petticrew and Roberts, 2006). Funding bias, methodo- OO contribute to wider dissemination of research logical bias, outcome bias and confirmatory bias are syntheses; examples of publication bias. Database bias, citation bias, availability bias, language bias, country bias, OO deepen academic synthesists’ understandings of familiarity bias and multiple publication bias are exam- the collaborating stakeholders’ concerns and ples of search bias that must be considered in planning understandings. appropriate search techniques. Through a careful cost– benefit analysis, synthesists make decisions related to Different collaborators have the potential to enrich the inclusion or exclusion of unpublished reports. synthesis by bringing in their own particular expertise. The terms ‘exhaustive’ and ‘expansive’ are some- Each form of collaboration also introduces issues of times used to distinguish between two approaches to power and varied interests that can add complexity to searching for suitable studies in recent literature on the synthesis process (Yu, 2011). In a collaborative qualitative evidence synthesis in health care (Finfgeld- synthesis, synthesists must carefully negotiate issues Connett and Johnson, 2012). Exhaustive searches are arising from different perceptions of roles, responsibil- more suitable for integrative syntheses aimed at pro- ity, collaboration, authority and authorship (Baldwin ducing generalizable findings. They are typically and Austin, 1995). employed by meta-a nalysts and systematic reviewers. When seeking input from stakeholders, synthesists Expansive searches with purposeful sampling strategies should sensitively clarify the nature of input being are more suitable for syntheses with emergent designs, sought and what can or cannot be negotiated; address where the search criteria evolve as the synthesis power imbalances between different stakeholders; rec- progresses, and are aimed at facilitating understanding, ognize heterogeneity within stakeholder groups; and participation, emancipation or deconstruction (Suri, ensure that less powerful groups do not feel further dis- 2011). empowered with the perception that their views are not A clear set of inclusion criteria defines the scope of being paid adequate attention (Petticrew and Roberts, the synthesis. The level of specificity associated with 2006; Rees and Oliver, 2012). the inclusion criteria at different stages of the synthesis Often, research synthesists begin by reading previ- may vary in accordance with the synthesis purpose. ous research reviews in the field. Previous reviews, The synthesist must strategically choose and sequence along with their bibliographic references, can provide the use of appropriate search channels in a way that is useful information for developing a broad overview of aligned with the sampling logic and which yields the primary research and research syntheses reported in the most relevant, trustworthy and comprehensive evidence field. In formulating an appropriate synthesis purpose, within the available resources. synthesists often consider: the topicality of the field; the nature of predominant methodologies employed in Phase four: evaluating, interpreting and the primary research studies; the general relationship distilling evidence from selected studies between individual studies; and the volume and scope of the relevant primary research. Paying careful atten- While some synthesists argue that all research reports tion to ethical issues of representation and non- which meet the substantive selection criteria ought to representation, synthesists should formulate the purpose be included in the synthesis (e.g. Jensen and Allen, for an intended audience and synthesis goal. 1996; Glass, 2000), others insist that studies which have relatively weak study designs ought to be Phase three: searching for relevant excluded (e.g. Eastabrooks et al., 1994; Slavin, 2008). literature All synthesists agree that findings from primary research that have stronger study designs should be Research synthesists draw their evidence from the given more weight in constructing collective under- primary research, secondary research and previous standings. Rather than asking ‘Is this a perfect study?’, research syntheses reported in a field. Research synthe- ses on the same topic conducted for different purposes can have different sampling strategies, each being 436
Meta-analysis, systematic reviews and research syntheses a synthesist ought to ask ‘How do methodological fea- subjective judgements and biases by maximizing tures of this study impact upon the trustworthiness of inter-c oder reliability (Stock, 1994) or by deliberately its findings in ways that are relevant to my synthesis critiquing and contesting each other’s emerging under- purpose?’ Synthesists must examine each report closely standings (Ogawa and Malen, 1991); as a form of trian- for a coherence between its theoretical background, gulation to improve rigour (Eastabrooks et al., 1994); intended purpose, context and/or nature of intervention or for co-constructing collective interpretations through being studied, methods for collecting, analysing and dialogic discussions (Wideen et al., 1998, p. 135; Pater- interpreting evidence, results and conclusions. Equally son et al., 2001, p. 47). important is the relevance of the report for the synthesis purpose (Dixon-W oods et al., 2006; Major and Savin- Phase five: constructing connected Baden, 2010). understandings In a rigorous synthesis, the set of evaluation criteria for individual studies is essentially guided by the over- While some research synthesis methods focus on arching teleological and theoretical orientation of the variable-o riented connections, others focus on study- synthesis. For example, studies with representative oriented connections. In variable-o riented connections, samples are suitable for generating generalizations. the variations in the effects, implementations, manifes- However, issues of how the views and voices of differ- tations, meanings, understandings or conceptions of a ent stakeholders are represented become particularly phenomenon are the prime focus of a synthesis. Each important in an emancipatory synthesis. Useful discus- account is examined to the extent that it contributes to sions have been published for evaluating the quality of explaining the relationships between the target varia- primary research reports stemming from different tradi- bles. When several individual studies examine a partic- tions, such as those for post-positivist research (e.g. ular intervention, concept or phenomenon using similar Centre for Reviews and Dissemination, 2009; Valen- methodologies, the findings may be aggregated to tine, 2009), interpretive research (Hannes et al., 2010), increase size and variations of the overall sample. participatory research (Heron and Reason, 1997) and These findings can then be used to make generaliza- critically oriented research (Lather, 1986). tions, provide plausible explanations and predict pat- Ongoing reflexive engagement with the selected terns of human behaviour as in a meta-analysis or an studies is crucial to reduce unaccounted or unacknowl- aggregated analysis (Eastabrooks et al., 1994). Com- edged biases. While coding information from individ- parative techniques such as content analysis, statistical ual studies, synthesists can assign ‘confidence ratings’ techniques and visual displays are commonly employed to findings or insights that require higher levels of inter- for constructing variable-o riented connections. pretation (Stock, 1994, p. 128). One approach to coding Study-oriented connections are amenable for con- can be to assign each finding one of the following three structing holistic and complex understandings with an codes based on the degree to which the finding is sup- attempt to retain the contextual integrity of individual ported by reported evidence: compelling, credible and accounts, as in Noblit and Hare’s (1988) meta- unsupported (Major and Savin-Baden, 2010). Decisions ethnography. The focus here is on understanding the in relation to dealing with missing data or biased find- dynamics of individual accounts as the synthesist ings should be consistent, substantiated and disclosed. attempts to reconcile the dynamics of each study with To minimize unstated subjective biases, meta- those of the other studies. The synthesist tries to deter- analysts and systematic reviewers try to maintain con- mine the relations and tensions between individual sistency and transparency by adhering to a priori accounts through a dialectical process of comparing protocols. Research synthesists espousing critical ori- key constructs, phrases and themes used in each study entations, on the other hand, stress that synthesists as interpreted by each co-s ynthesist (Noblit and Hare, should reflexively respond to change, rather than rigidly 1988). follow an a priori protocol (Zhao, 1991; Pawson et al., When individual reports are addressing similar 2005). A reflexive stance involves constantly reflecting issues, they are amenable to a reciprocal translational critically on how the synthesist’s own dispositions and synthesis to construct a collective understanding that perspectives are dialectically influencing, and being captures the essence of all the included studies (Jensen influenced by, the synthesis process and findings and Allen, 1996; Paterson et al., 2001). The findings of (MacLure, 2005). each report are tested for their abilities to translate the Several authors recommend that the complexity of a findings of the other reports. Those terms or findings research synthesis process requires collaborative efforts are selected which can more succinctly describe the to ensure a certain level of trustworthiness: to minimize findings of all the reports within the subset. When the synthesist has limited resources, the synthesist can 437
Methodologies for educational research select an exemplary study and examine other studies Phase six: communicating with an audience for the extent to which they demonstrate or add to the description of the phenomenon in the exemplar study It is crucial that synthesists carefully select the content, (Miles and Huberman, 1994). representational style, medium, genre and techniques in When individual reports give conflicting representa- alignment with the impact they wish to make on their tions of the same phenomenon, they lend themselves to target audience. They must share the process and the a ‘refutational synthesis’ (Noblit and Hare, 1988, p. 47) product of their sense-m aking skilfully and in a way where the relationships between individual studies and that is credible, trustworthy and useful to the intended the refutations become the focus of the synthesis audience. A range of interesting, engaging and effec- process. The contradictions between individual reports tive possibilities for communication can arise by may be explicit or implicit. The implicit refutations are embedding various representational tools within diverse made explicit using an interpretive approach. This is structural genres expressed through varied media. As followed by an attempt to explain the refutation. educational reviews are often seen to represent wider If individual reports examine different aspects of the power relations, synthesists should be critically con- same phenomenon, a ‘lines-o f-argument’ synthesis scious of how a review may preferentially normalize could be used to make inferences (Noblit and Hare, certain representations or practices over the others 1988). In this method, findings from individual reports (Baker, 1999). are used as pixels to get a fuller picture of the phenom- Many synthesists, especially meta-a nalysts, employ enon at hand. It involves a grounded theory-like the scientific reporting format with four distinct sec- approach for open-coding and identifying the catego- tions, i.e. Introduction, Methods, Results, Discussion. ries emerging from the data. The key categories that are Many others use coherent thematic narratives to share more powerful in representing the entire data set are their synthetic understandings with an audience. They identified by constant comparisons between individual construct their narrative by organizing into a coherent accounts. These categories are then linked interpre- logical structure the main themes which contextualize tively to create a holistic account of the phenomenon. and describe the synthesis process and product. These When synthesizing methodologically diverse themes, headings and subheadings may vary purpose- reports, a synthesist can begin by constructing collec- fully according to: intervention types; methodological tive understandings from clusters of studies with designs; contextual features (e.g. Engberg, 2004); types similar designs and then synthesize collective under- of descriptive commentaries such as retrospective, pro- standings across clusters (Suri, 1999; Greenhalgh et al., spective or critical (e.g. Bransford and Schwartz, 1999); 2005). or the aspect of the synthesis process being described Study-oriented connections can also be imple- such as introduction, methods, results and discussion. mented, as in Pawson’s realist synthesis, by inferring For each idea, theme or finding, synthesists should theory from each study and examining its transferabil- identify pertinent techniques for a rich, succinct and ity to other cases to refine the initial theory. In a realist audience-friendly representation. Given the vast nature synthesis, this process is repeated with every study of of evidence in research syntheses, descriptive statistics, successful and unsuccessful implementation to develop abridged quotes and visual displays can be particularly a more sophisticated and comprehensive theory that useful tools. Synthesists can also choose from a range can explain many cases (Pawson et al., 2005). of narrative techniques and artistic devices. Strategies for enhancing plausibility, authenticity, Quality research synthesis reports generally share utility, robustness and validity of synthesis findings the following characteristics: conceptually substanti- include: ated and well-b ound coverage of the substantive topic; rigorous critique of previous reviews; identification of OO reflexivity; common assumptions, theories, methods and findings OO collaborative sense-m aking; emerging from extant research; critical analyses of OO eliciting feedback from key stakeholders; extant research; coherent structuring of the report along OO identifying disconfirming cases and exploring rival meaningful themes; a unique conceptual framework or perspective to think about the topic, future research, connections; practice and policy; clear implications for researchers, OO sensitivity analyses and using multiple lenses to practitioners and policy makers; and a discussion of any caveats associated with synthesis findings to clarify identify the dependence of a synthesis finding on its domain of applicability (Suri, 2014). underlying assumptions or the frame of reference. 438
Meta-analysis, systematic reviews and research syntheses 21.5 Conclusion each of six phases of a research synthesis, and these are discussed from a methodologically inclusive perspec- This chapter has discussed many of the key issues in tive. The strengths, complexities, domains of applica- carrying out meta-a nalysis, systematic review and bility and caveats of the different approaches have been methodologically inclusive research syntheses as an discussed so that researchers can make their own educational researcher. The chapter’s central focus informed choices about the kind of research approach settles on the many critical decisions associated with they wish to adopt for a given project. Companion Website The companion website to the book includes PowerPoint slides for this chapter, which list the structure of the chapter and then provide a summary of the key points in each of its sections. These resources can be found online at www.routledge.com/cw/cohen. 439
Action research CHAPTER 22 Action research is a widely used approach to research, research (Munn-G iddings, 2012, p. 71). Action as evidenced in the numerous journals whose titles bear research, as one of its leading proponents, Stephen the phrase ‘action research’. The main aim of action Kemmis (2009) notes, seeks to change and transform research, as one of its key proponents, Elliott (1991), ‘practitioners’ practices, their understandings of their states, is ‘to improve practice’ (p. 49). It is a methodol- practices, and the conditions in which they practice’ ogy for researchers (often teachers) to understand and (p. 463). It is a ‘practice-changing practice’ (p. 464; generate knowledge about educational practices and italics in original; see also Kemmis et al., 2014). their complexity (McAteer, 2013, p. 21). McAteer (2013) adds to this the point that, by being a In action research, typically teachers and other collaborative process, it aims to become both a ‘demo- parties research their own institutions, staff develop- cratic and democratising process’ (p. 17). Cain (2011) ment facilitators bring about change and groups and argues that action research moves beyond positivist, communities undertake research. This chapter intro- interpretive and critical research, being self-reflexive, duces key issues in the planning, conduct and reporting collaborative, political and suitable for dissemination, of action research, including: for example, to other teachers (pp. 13–14). McNiff (2010) advises researchers to OO defining action research OO principles and characteristics of action research take action to improve something; make sure you OO participatory action research understand what you have done; use understanding OO action research as critical praxis (knowledge) to give explanations for how and why OO action research and complexity theory you have improved it (in academic terms, this means OO procedures for action research that you generate new theory – the word ‘theory’ OO reporting action research means explanation). OO reflexivity in action research OO ethical issues in action research (McNiff, 2010, p. 16) OO some practical and theoretical matters Action research, she observes, is ‘practice-b ased’ The chapter draws links between action research and (p. 33), concerns learning and the creation of know critical theory, in particular in respect of participatory ledge, is ‘values laden’, ‘educational’, ‘collaborative’, action research. It also notes the connections between ‘critical and risky’ and ‘always political’ (pp. 33–4), action research and complexity theory. and she provides plentiful advice for researchers working in this mode. 22.1 Introduction Action research can be used in almost any setting where a problem involving people, tasks and proce- Action research is a useful tool for change and dures needs a solution, or where some change of feature improvement at the local level. Indeed Kurt Lewin’s results in a more desirable outcome. It embraces both own work (one of action research’s founding fathers) problem posing and problem solving. It need not focus was deliberately intended to change the life chances of solely on problems; it can also embrace areas of inter- disadvantaged groups in terms of housing, employ- est for development (McNiff, 2010). It can be under- ment, prejudice, socialization and training. Its combina- taken by the individual teacher, a group of teachers tion of action and research has contributed to its working cooperatively within one school or a teacher attraction to researchers, teachers and the academic and or teachers working alongside a researcher or research- educational community, breaking the culture of ‘spec- ers in a sustained relationship, possibly with other tator research’ (Cain, 2011, p. 3) and ‘ivory tower’ interested parties like advisers and university depart- ments (Holly and Whitehead, 1986). Ferrance (2000) 440
Action research identifies different levels of action research: individual form of participatory research (see Chapter 3; Kapoor teacher research, collaborative action research, and Jordan, 2009). ‘Commitment’ is a feature of partic- school‑wide action research and district-wide actions ipatory research (see Chapter 3), and participatory research (p. 6). action research both requires and builds commitment Action research can be used in a variety of areas, for (David, 2002). Participatory research breaks the separa- example: tion of the researcher and the participants; power is equalized and, indeed, they may all be part of the same OO teaching methods: replacing a traditional method by community. The research becomes a collective and a discovery method; shared enterprise in many spheres, including: research interests, agendas and problems; generation and analy- OO learning strategies: adopting an integrated approach sis of data; equalization of power and control over the to learning in preference to a single-subject style of research outcomes, products and uses; development of teaching and learning; participant voice, authorship and ownership; emancipa- tory agendas and political goals; a process-o rientated OO evaluative procedures: improving one’s methods of and problem-s olving approach; and ethical responsibil- continuous assessment; ity and behaviour. We refer the reader to Chapter 3 for a fuller discussion of this. OO attitudes and values: encouraging more positive atti- tudes to work, or modifying pupils’ value systems 22.2 Defining action research with regard to some aspect of life; Action research comes in many forms, including action OO continuing professional development of teachers: research, participatory action research, critical action improving teaching skills, developing new methods research, diagnostic action research, practitioner of learning, increasing powers of analysis of height- research, classroom-b ased action research, empirical ening self-a wareness; action research and many others (e.g. Jefferson, 2014). It is typically a small-s cale intervention in the function- OO management and control: the gradual introduction ing of the ‘real’ world and a systematic, close examina- of different techniques of class management; tion, monitoring and review of the effects of such an intervention, combining action and reflection to improve OO administration: increasing the efficiency of some practice (cf. Ebbutt, 1985, p. 156; Hopkins, 1985, p. 32; aspect of the administrative side of school life. New South Wales Department of Education and Train- ing, 2010). When operating with collaborative groups of It can be used to foster democratic institutions, encour- teachers, it is designed to address collective improve- age change, empower individuals and groups, encour- ment and development. Action research is an investiga- age reflective practice and be a test-b ed for new ideas tion which is intentionally directed towards solving a and practices (Creswell, 2012, p. 578). problem or focusing on an issue raised by, and owned These examples do not mean, however, that action by, an individual or a group (Kemmis and McTaggart, research can be typified straightforwardly; that is to 1988, p. 5; McNiff, 2010). It has a deliberately applied distort its complex and multifaceted nature. Indeed focus (Creswell, 2012, p. 577) and is ‘hands on’ Kemmis (2010) suggests that there are several schools research (Denscombe, 2014, p. 122). of action research. That said, what unites different con- Piggot-Irvine et al. (2015) define action research as ceptions of action research is the desire for improve- ‘a collaborative transformative approach with joint ment to practice, based on a rigorous evidential trail of focus on rigorous data collection, knowledge genera- data and research. tion, reflection and distinctive action/change elements Ferrance (2000, p. 1), echoing Watts (1985), argues that pursue practical solutions’ (p. 548). This draws that a powerful justification for action research is that attention to two key points taken up further in this teachers: chapter: the collaborative, emancipatory claims of action research, and its practical outcomes in terms of OO work best on problems that they have identified for bringing about change. themselves; The rigour of action research is attested by one of its founding fathers, Corey (1953, p. 6), who argues that it OO become more effective when they are encouraged to is a process in which practitioners study areas for examine and assess their own work and then con- development scientifically (our italics) so that they can sider ways of working differently; OO help each other by working collaboratively; OO help each other in their professional development by working together. She suggests that action research builds on, and builds in, these principles. Indeed action research is a potent 441
Methodologies for educational research evaluate, improve and steer decision making and prac- The several strands of action research are drawn tice. (Helskog (2014) discusses the debate as to whether together by Kemmis and McTaggart (1988) in their or not action research is a science and what the impli- view of action research as a type of critical reflective cations of this debate are in regard to justifying action enquiry which participants undertake on and for them- research.) Kemmis and McTaggart (1992, p. 10) argue selves, focusing on problems and practices which they that action research requires systematic planning, identify themselves and which affect them, with the acting, observing and reflecting in a manner that is intention of understanding and improving the educa- more demanding and rigorous than in the everyday tional and social practices in which they are involved course of life. and the circumstances in which they take place, in A philosophical stance on action research that order to promote social justice; it is undertaken collab- echoes the work of Habermas is taken by Carr and oratively, albeit sometimes focusing on individuals in Kemmis (1986, p. 162), who regard it as a form of the group (p. 5). ‘self-reflective enquiry’ by participants which is under- Kemmis and McTaggart (1992, pp. 21–2) distin- taken in order to improve their understanding of their guish action research from the everyday actions of practices with a view to maximizing social justice (see teachers: in four main ways: also Gibbs et al., 2016). Grundy (1987, p. 142) regards action research as concerned with improving the ‘social OO it is thinking in a more systematic and collaborative conditions of existence’. Kemmis and McTaggart way than the customary, everyday ways in which (1992) suggest that action research is concerned with teachers consider their own practices; changing both individuals and the institutions, societies and cultures of which they are members. OO it moves beyond problem solving alone to identify- Action research is designed to bridge the gap between ing and raising problems, regarding problem solving research and practice (Somekh, 1995, p. 340), thereby as opportunities for change, learning and improve- striving to overcome the perceived persistent failure of ment rather than as simply curing ills (a pathological research to impact on, or improve, practice. Stenhouse model); (1979) suggests that action research should contribute not only to practice but to a theory of education and teaching OO it involves participants working on their self- which is accessible to other teachers, making educational identified areas of work, i.e. it is owned by the par- practice more reflective (Elliott, 1991, p. 54). ticipants rather than external researchers; Action research combines diagnosis, action and reflection (McNiff, 2010), focusing on practical issues OO it adopts a heterogeneous rather than unitary concept that have been identified by participants and which may of the science of teaching. somehow be both problematic yet capable of being changed (Elliott, 1978, pp. 355–6). McNiff (2010) Noffke and Zeichner (1987) make several claims for places self-reflection at the heart of action research, action research with teachers, namely that it: brings suggesting that whereas in some forms of research, the about changes in their definitions of their professional researcher ‘does research on other people’, in action skills and roles; increases their feelings of self-w orth research, the researcher does it to herself/himself. Jef- and confidence; increases their awareness of classroom ferson (2014) notes that key assumptions of action issues; improves their dispositions towards reflection; research include that ‘practitioners work best on prob- changes their values and beliefs; improves the congru- lems that they have identified themselves’ and that they ence between practical theories and practices; and increase their effectiveness if they examine and ‘assess broadens their views on teaching, schooling and their own work and then consider ways of working dif- society. ferently’ (pp. 91–2) and work collaboratively, thereby Action research lays claim to the professional devel- developing their professional practices. Ferrance (2000) opment of teachers; action research for professional and McNiff (2010) note that it is not concerned simply development is a frequently heard maxim (e.g. Somekh, with solving problems (‘what is wrong’; p. 2) but is 1995, p. 343, 2006; Winter, 1996; Ferrance, 2000; New more concerned with how to improve. As Zuber- South Wales Department of Education and Training, Skerritt (1996b, p. 83) remarks: ‘the aims of any action 2010). It is ‘situated learning’: learning in the work- research project or program are to bring about practical place, about the workplace and for the workplace (cf. improvement, innovation, change or development of Collins and Duguid, 1989; NSW Department of Educa- social practice, and the practitioners’ better understand- tion and Training, 2010). The claims for action ing of their practices’. research, then, are several. Arising from these claims and definitions are several principles. 442
Action research 22.3 Principles and characteristics 3 collaboration, which is intended to mean that every- of action research one’s view is taken as a contribution to understand- ing the situation; Hult and Lennung (1980, pp. 241–50), McKernan (1991, pp. 32–3), Ferrance (2000), the New South 4 risking disturbance, which is an understanding of Wales Department of Education and Training (2010) our own taken-for-granted processes and willing- and Kemmis et al. (2014) suggest that action research: ness to submit them to critique; OO makes for practical problem posing and problem 5 creating plural structures, which involves develop- solving as well as expanding scientific knowledge; ing various accounts and critiques, rather than a single authoritative interpretation; OO enhances the competencies of participants; OO is collaborative; 6 theory and practice internalized, which is seeing OO is undertaken directly in situ; theory and practice as two inter-d ependent yet com- OO uses feedback from data in an ongoing cyclical plementary phases of the change process. process; Action research has been compared to, and claimed to OO seeks to understand particular complex social differ from, ‘formal research’ in several respects (New South Wales Department of Education and Training, situations; 2010, p. 1), for example: it requires little training OO seeks to understand the processes of change within (whereas formal research often requires extensive training); its intention is to focus on a local, institu- social systems; tional situation or practice with the intention of OO is undertaken within an agreed framework of ethics; improving practice, in contrast to those kinds of OO seeks to improve the quality of human actions; research which seek generalizability; the sample com- OO focuses on those problems that are of immediate prises the relevant participants in the institution or sit- uation rather than representing the wider population concern to practitioners; or being drawn randomly from the wider population; OO is participatory; and it is of practical rather than theoretical OO frequently uses case study; significance. OO tends to avoid the paradigm of research that isolates Action research resists the criticism that it is not ‘proper research’ (Somekh, 2006), as it abides by many and controls variables; of the tenets of research (e.g. hypothesis generation and OO is formative, such that the definition of the problem, testing, new ways of looking at problems, generating knowledge, rigorous investigation) and, indeed, it ben- the aims and methodology may alter during the efits from insider knowledge. Similarly, Casey (2013) process of action research; notes that action research benefits from being con- OO includes evaluation and reflection; ducted by ‘someone native to the field’ and with action OO is methodologically eclectic; taken ‘at the site where practice occurs’ (p. 149). OO contributes to a science of education; Teachers, for example, do it to and for themselves, OO strives to render the research useable and shareable becoming part of their everyday practices (p. 149). by participants; The several features that the definitions at the start OO is dialogical and celebrates discourse; of this chapter have in common suggest that action OO has a critical purpose in some forms; research has key principles. These are summarized by OO strives to be emancipatory. Kemmis and McTaggart (1992, pp. 22–5), when they aver that action research: Zuber-Skerritt (1996b, p. 85) suggests that action research is: critical (and self-critical) collaborative OO seeks to improve education by deliberately and inquiry by reflective practitioners being accountable deliberatively trying to change it, focusing on and making results of their enquiry public, self- improving participants’ own practices collabora- evaluating their practice and engaging in participatory tively and with consequent learning from these problem solving and continuing professional develop- changes and their involvement in the process; ment. This is echoed in Winter’s (1996, pp. 13–14) six key principles of action research: OO involves ongoing, systematic and ongoing develop- mental cycles of planning, implementing, observing 1 reflexive critique, which is the process of becoming and reflecting (self-reflection) on changes. Such aware of our own perceptual biases; actions contribute to collaborative communities of 2 dialectical critique, which is a way of understanding the relationships between the elements that make up various phenomena in our context; 443
Methodologies for educational research practitioners which are characterized by their self- The adherence to action research as a group activity critical reflection on, and insights into, relationships derives from several sources. Pragmatically, Oja and between contexts, actions and outcomes, and which Smulyan (1989, p. 14), in arguing for collaborative promote emancipation, empowerment, legitimation action research, suggest that teachers are more likely to and social justice in participants’, others’ and insti- change their behaviours and attitudes if they have been tutions’ educational values and practices, thereby involved in the research that demonstrates not only the constituting a political process and endeavour; need for such change but that it can be done – the issue OO requires participants to enquire into and theorize of ‘ownership’ and ‘involvement’ that finds its parallel about the conditions of, and circumstances and prac- in management literature which suggests that those tices in, their own lives. In this process, their ideas, closest to the problem are in the best position to iden- values and assumptions are tested against rigorous tify it and work towards its solution (e.g. Morrison, evidence in an open‑minded, evidence-b ased, 1998). critical-a nalytical and reflexive spirit; Ideologically, there is a view that those experienc- OO leads to the development of a clearly justified, justi- ing the issue should be involved in decision making, fiable, reasoned, evidence-based rationale for educa- itself hardly surprising given Lewin’s own work with tional practices; disadvantaged and marginalized groups, i.e. those OO can begin small-s cale, even on an individual or groups with little voice (cf. David, 2002). small-g roup level, and can spiral out to affect and Politics and ideology are brought together in partici- involve others in communities of practice, thereby patory action research and action research as critical engaging issues of power and empowerment in deci- praxis, and it is to this that we turn. sion making. OO can benefit from careful records of changes and 22.4 Participatory action research improvement, the processes involved in making them, ways of perceiving and understanding Some researchers differentiate action research from them, the social relationships involved in them, and participatory action research, the latter being a more raised awareness of constraints on situations and specific subset of action research, whilst other research- participants. ers make no such distinction (Munn-Giddings, 2012, p. 72). Participatory action research has attracted atten- Though these principles find widespread support in the tion across the world in its advocacy of democracy, literature on action research, they require some empowerment and emancipation (cf. David, 2002; comment. For example, there is a strong emphasis in Jones and Stanley, 2010). Kemmis and McTaggart these principles on action research as a cooperative, (2005) comment that participatory action research collaborative activity (rather than an individualistic seeks to create conditions for people to work together activity) (Kemmis and McTaggart, 1992, p. 15). Indeed collaboratively in the search for valid, authentic and Kemmis and McTaggart (1992) locate this in the work morally correct and appropriate ways of understanding of Lewin himself, commenting on his commitment to the world and participating in it (p. 578). Whereas some group decision making (p. 6). They argue, too, that it is action research focuses on individuals, participatory those groups who are involved in, or affected by, action research is communitarian and social, seeking to planned interventions and changes who should bear the bring about social change and improvement to the prime responsibility for taking decisions on actions, quality of people’s lives (Creswell, 2012, pp. 582–3). based on reasoned, interrogated, evidence-based analy- McTaggart (1989), Kemmis and McTaggart (2005), sis and evaluation (p. 15). Locke et al. (2013) and Kemmis et al. (2014) suggest The view of action research solely as a group activ- several tenets of participatory action research, indicat- ity, however, might be too restricting. It is possible for ing that it is a social process that focuses on the action research to be an individualistic matter as well, relationship between individuals and their social envi- relating action research to the ‘teacher-as-researcher’ ronment. It is deliberately practical, seeking to improve movement (Stenhouse, 1975; Pring, 2015). Whitehead social practice and people by having them work on (1985, p. 98) explicitly writes about action research in themselves. In doing this it requires authentic participa- individualistic terms, and we can take this to suggest tion and is collaborative, establishing self-critical, non- that a teacher can ask herself or himself: ‘What do I see hierarchical communities and partnerships. It is also a as my problem?’ ‘What do I see as a possible solution?’ recursive and systematic process of learning, with plan- ‘How can I direct the solution?’ ‘How can I evaluate ning, action, analysis and reflection leading to further the outcomes and take subsequent action?’ planning, action, analysis and reflection. This involves 444
Action research people in theorizing about their own practices and possible consequences of actions, the awareness of values, testing their own assumptions, values, ideas and structural conditions) (Weiskopf and Laske, 1996, practices in real-life practice; in other words it is reflex- pp. 132–3). ive, drawing together theory and practice. Participatory Participatory action research is distinguished not action research requires participants to build and keep only by its methodology (collective participation) and evidential records of practice, theory and reflection and its outcomes (democracy, voice, emancipation), but by to provide a reasoned justification to others for their its areas of focus (inequalities of power, grass-roots work. These, in turn, require participants to look at and agendas for change and development, e.g. educational document their own experiences objectively. Participa- inequality, social exclusion, sexism and racism in edu- tory action research is part of a political process (e.g. cation, powerlessness in decision making, student dis- towards democracy) and involves people in making affection with a socially reproductive curriculum, critical analyses of a situation, research and practice. It elitism in education) and its intention to change society starts small and in small cycles, with small groups and social situations (cf. Fine, 2010; INCITE, 2010; of people, and is critical and emancipatory, with Kemmis, 2010; Kemmis et al., 2014). Importantly here, participants addressing and interrogating unjust social the agendas and areas of focus are identified by the par- structures which limit people’s development and ticipants themselves, so they are rooted in reality, are self‑realization. It also disseminates findings to other authentic and are ‘owned’ by the participants and com- practitioners and networks. munities themselves. For Kemmis et al. (2014), adopting a critical/critical Participatory action research – people acting and theory approach is a sine qua non of participatory researching on, by, with and for themselves – is a dem- action research, the major aim of which is to change ocratic activity (Grundy, 1987, p. 142; David, 2002; from practice to praxis (committed, informed, self- Jones and Stanley, 2010; Kemmis et al., 2014). This realizing action, ‘practical philosophy’ which links form of democracy is participatory (rather than, for theory, thinking and practice (Carr, 2005)), inspired by example, representative): a key feature of critical the language and operation of possibility, solidarity, research (and Lewin – a key early figure in action open communication and freedom from dominatory research – advocated democratic workplaces) (Jeffer- and oppressive social conditions. son, 2014, p. 93). It is not merely a form of change That there is a coupling of the ideological and politi- theory, but addresses fundamental issues of power and cal debate here has been brought into focus with the power relationships, for, in according power to partici- work of Freire (1972) and Torres (1992, p. 56) in Latin pants, action research is an empowering activity America, the latter setting out several principles of par- (David, 2002; Kemmis et al., 2014). Elliott (1991, ticipatory action research: p. 54) argues that such empowerment has to be at a col- lective rather than an individual level, as individuals do OO it commences with explicit social and political not operate in isolation from each other, but are shaped intentions that articulate with the dominated and by organizational and structural forces. poor classes and groups in society; The issue is important, for it begins to separate action research into different camps (Kemmis, 1997, OO it must involve popular participation in the research p. 177). On the one hand are long-time advocates of process, i.e. it must have a social basis; action research such as Elliott (e.g. 1978, 1991) who, in the tradition of Schwab and Schön, emphasize reflec- OO it regards knowledge as an agent of social transfor- tive practice; this is particularly so in curriculum mation as a whole, thereby constituting a powerful research with notions of the ‘teacher-as-researcher’ critique of those views of knowledge (theory) as (Stenhouse, 1975) and the reflective practitioner somehow separate from practice; (Schön, 1983, 1987). On the other hand are advocates of the ‘critical’ action research model, for example, OO its epistemological base is rooted in critical theory Carr and Kemmis (1986), Kemmis et al. (2014). and its critique of the subject/object relations in research; 22.5 Action research as critical praxis OO it must raise the consciousness of individuals, groups and nations; Much of the writing in this type of action research draws on the Frankfurt School of critical theory (see OO it must lead to transformation and emancipation. Chapter 3), in particular the early work of Habermas. Participatory action research does not mean that all par- ticipants need to be doing the same. This recognizes a role for the researcher as facilitator, guide, formulator and summarizer of knowledge, raiser of issues (e.g. the 445
Methodologies for educational research (Kemmis (2006) addresses some of the later work of ical enterprise rather than the accretion of trivial cook- Habermas in his (Habermas’s) concern for the ‘public book remedies (a technical exercise). Indeed Luttenberg sphere’.) Indeed Weiskopf and Laske (1996, p. 123) et al. (2016) regard reflection on action as a moral and Kemmis et al. (2014) locate action research, in the activity, not simply a technical-instrumental matter. German tradition, squarely as a ‘critical social science’ Marshall and Rossman (2016) state very clearly (p. 26) with an emancipatory interest: to challenge and thence that action research eschews the claimed neutrality and to transform unjust and repressive, alienating social objectivity of traditional research in favour of promot- structures. Using Habermas’s early writing on ing values-b ased change in their own institutions. knowledge-c onstitutive interests (1972, 1974), a three- Emancipatory action research has an explicit agenda fold typology of action research can be constructed which is as political as is it educational, promoting which comprises technical, practical and emancipatory social justice (Gibbs et al., 2016). Grundy (1987) (critical) interests; the classification was set out in argues (pp. 146–7) that such emancipatory action Chapter 3, and is the basis for the seminal works of research seeks to develop in participants their under- Carr and Kemmis (1986) and Grundy (1987) in the standings of illegitimate structural and interpersonal field of action research. constraints that are preventing the exercise of their The work of Carr and Kemmis (1986) fuelled a tra- autonomy and freedom. These constraints, she argues, dition of critical action research and its ‘explanatory, are based on illegitimate repression, domination and normative and practical dimensions’ (Hawkins, 2015, control. When participants develop a consciousness of p. 466). Critical theory advocates the understanding these constraints, she suggests, they begin to move and ideological interrogation of social conditions, and from unfreedom and constraint to freedom, autonomy aims to bring about democracy, equality and social and social justice. justice, partly by exposing and working on the under- Kincheloe (2003, pp. 138–9) clarifies emancipatory standings of participants who are seeking such emanci- action research as: pation and societal transformation and also by looking at the ‘conditions of possibility’ (p. 466) for such to OO constructing a system of meaning; occur. OO understanding dominant research methods and their Grundy (1987, p. 154) argues that ‘technical’ action research is designed to render an existing situation effects; more efficient and effective, to improve outcomes of OO selecting what to study; practice (Kemmis, 2009, p. 469). In this respect it is OO acquiring a variety of research strategies; akin to Argyris’s notion of ‘single-loop learning’ OO making sense of information collected; (Argyris, 1990), being functional, often short-term and OO gaining awareness of the tacit theories and assump- technical. It is also akin to Schön’s (1987) notion of ‘reflection-in-action’ (cf. Luttenberg et al., 2016). tions which guide practice; Elliott (1991, p. 55) suggests that this view is limiting OO viewing teaching as an emancipatory, praxis- for action research since it is too individualistic and neglects wider curriculum structures, regarding teach- based act. ers in isolation from wider factors. By contrast, ‘practical’ action research is designed ‘Praxis’ here is defined as action informed through to promote teachers’ professionalism by drawing on reflection, and with emancipation as its goal, a ‘morally their informed judgement to enable them to act more committed action’ (Kemmis, 2009, p. 465), and eman- wisely (Grundy, 1987, p. 154; Kemmis, 2009, p. 470). cipatory/critical action research is part of a collective This underpins the ‘teacher-a s-researcher’ movement, and collaborative enterprise to transform social forma- inspired by Stenhouse (cf. Pring, 2015). It is akin to tions, structures and practices that are built into the Schön’s ‘reflection-o n-action’ and is a hermeneutic architecture of our lives and societies and which are activity of understanding and interpreting social situa- deemed to be unsustainable on moral, social, ecologi- tions with a view to their improvement (Luttenberg et cal, material, rational, ideological, personal, political al., 2016). Echoing this, Kincheloe (2003, p. 42) sug- and economic grounds (cf. Kemmis, 2009, pp. 470–1, gests that action research rejects positivistic views of 2010). rationality, objectivity, truth and methodology, prefer- Action research, here, empowers individuals and ring hermeneutic understanding (phronesis) (Thomas, social groups to take control over their lives within a 2010) and emancipatory practice. As Kincheloe notes framework of the promotion of rather than the ‘sup- (p. 108), the teacher-a s-researcher movement is a polit- pression of generalizable interests’ (Habermas, 1976, p. 113). It commences with a challenge to the illegiti- mate operation of power and requires participants to question and challenge given value systems. For 446
Action research Grundy (1987), praxis fuses theory and practice within Because it is a community of equals, action research is an egalitarian social order, and action research is necessarily democratic and promotes democracy. designed with a political agenda of improvement Indeed the search is for consensus (and consensus towards a more just, egalitarian society. This accords to requires more than one participant), hence it requires some extent with Lewin’s view that action research collaboration and participation. leads to equality and cooperation, an end to exploita- The link between Habermas’s ‘ideal speech situa- tion and the furtherance of democracy (see also Carr tion’ and action research is reinforced by Eady et al. and Kemmis, 1986, p. 163; Jones and Stanley, 2010). (2015) in their argument that action research benefits Zuber-Skerritt (1996a) suggests that: from ‘communicative space’: a physical, emotional and temporal space in which ‘professionals are able to emancipatory action research … is collaborative, engage in meaningful modes of collaboration, demo- critical and self-critical inquiry by practitioners … cratic and non-judgmental dialogue’ (p. 107), i.e. learn- into a major problem or issue or concern in their own ing together through dialogue. practice. They own the problem and feel responsible Emancipatory action research fulfils the require- and accountable for solving it through teamwork and ments of action research set out earlier by Kemmis and through following a cyclical process of: McTaggart (1988, 2005) and Kemmis et al. (2014); indeed it could be argued that only emancipatory action 1 strategic planning; research (in the threefold typology) has the potential to 2 action, i.e. implementing the plan; do this. 3 observation, evaluation and self-e valuation; Kemmis (1997, p. 177) suggests that the distinction 4 critical and self-c ritical reflection on the results between the two camps (the reflective practitioners and the critical theorists) lies in their interpretation of action of points 1–3 and making decisions for the next research. For the former, action research is an improve- cycle of action research. ment to professional practice at the local, perhaps class- room level, within the capacities of individuals and the (Zuber-Skerritt, 1996a, p. 3) situations in which they are working; for the latter, action research is part of a broader agenda of changing Action research, she argues, education, changing schooling and changing society. A key term in action research is ‘empowerment’; for is emancipatory when it aims not only at technical the former camp, empowerment is largely a matter of and practical improvement and the participants’ the professional sphere of operations, achieving profes- better understanding, along with transformation and sional autonomy through professional development. change within the existing boundaries and condi- For the latter, empowerment concerns taking control tions, but also at changing the system itself or those over one’s life within a just, egalitarian, democratic conditions which impede desired improvement in society. Whether the latter is realizable or utopian is a the system/organization.… There is no hierarchy, matter of critique of this view. Where is the evidence but open and ‘symmetrical communication’. that critical action research either empowers groups or alters the macro‑structures of society? Is critical action (Zuber-Skerritt, 1996a, p. 5) research socially transformative? Several concerns have been levelled at emancipa- This form of participatory research forms ‘empa- tory action research (Gibson, 1985; Morrison, 1995a, thetic and compassionate ties’ (Hawkins, 2015, p. 468) 1995b; Somekh, 1995; Melrose, 1996; Grundy, 1996; that hold people together. Weiskopf and Laske, 1996; Webb, 1996; McTaggart, The emancipatory interest takes seriously the notion 1996; Kemmis, 1997; Elliott, 2005; Hadfield, 2012), of action researchers as participants in a community of including the views that: equals. This, in turn, is premised on Habermas’s notion of the ‘ideal speech situation’ (Morrison, 1995a, 1 it is utopian and unrealizable; pp. 99–104; cf. Hawkins, 2015, p. 468). Here action 2 it is too controlling and prescriptive, seeking to research is construed as reflective practice with a politi- cal agenda and in which all participants (and action capture and contain action research within a partic- research is participatory) are equal ‘players’. Action ular mould – it moves towards conformity; research is necessarily dialogical – interpersonal – 3 it adopts a narrow and particularistic view of eman- rather than monological (individual), and communica- cipation and action research, and how to undertake tion is an intrinsic element, with communication being the latter; among the community of equals (Grundy and Kemmis (1988, p. 87) term this ‘symmetrical communication’). 447
Methodologies for educational research 4 it undermines the significance of the individual 22.6 Action research and complexity teacher-a s-researcher in favour of self-critical com- theory munities. (Kemmis and McTaggart (1992, p. 152) pose the question ‘why must action research consist Action research links with participatory research and has of a group process?’); affinities with complexity theory. Phelps and Graham (2010, p. 184) argue that action research ‘can readily 5 the threefold typification of action research is accommodate the key tenets of complexity theory’ and untenable; that there is a ‘deep complementarity’ between them. For example, they note (p. 187) that action research accepts 6 it assumes that rational consensus is achievable, that systems are unpredictable, open and non-linear. It that rational debate will empower all participants resonates with issues of adaptation to environment and (i.e. it understates the issue of power, wherein the can lead to bifurcation, i.e. when a system moves from most informed are already the most powerful. one ‘point of stability to another’ (p. 190). It celebrates Grundy (1996, p. 111) argues that the better argu- the interaction of participants and requires both feedback ment derives from the one with the most evidence and feed forward. It is reflective and shows an interest in and reasons, and that these are more available to ‘exceptions’ or outliers (which can lead to major change) the powerful, thereby rendering the conditions of (p. 194). It is not concerned with controlling variables, equality suspect); and accepts that the systems in which it takes place are complex and dynamic (see also Davis and Sumara, 2005, 7 it overstates the desirability of consensus-oriented p. 455). Action research, like complexity theory, cele- research (which neglects the complexity of power); brates self-o rganization, with new states and situations emerging from tipping points (Morrison, 2008): self- 8 power cannot be dispersed or rearranged simply by organized criticality (Bak, 1996). Similarly, Luttenberg rationality; et al. (2016) note that action research, and the reflection that it involves, produces outcomes and processes that 9 action research as critical theory reduces its practi- are open, dynamic, non-linear, adaptive and co‑adaptive cal impact and confines it to the commodification (between components), and emergent, and that action of knowledge in the academy; research itself can be regarded as a complex system (pp. 6–12). 10 it will promote conformity through slavish adher- ence to its orthodoxies; 22.7 Procedures for action research 11 is naive in its understanding of groups and cele- There are several ways in which steps in action research brates groups over individuals, particularly the ‘in- have been analysed. Lewin (1946, 1948) codified the groups’ rather than the ‘out-g roups’; action research process into four main stages: planning, acting, observing and reflecting (cf. New South Wales 12 privileges its own view of science (rejecting objec- (NSW) Department of Education and Training, 2010). tivity) and lacks modesty; This operates in a cyclical process, with one cycle of this four-s tep approach leading into the subsequent four-s tep 13 privileges the authority and supremacy of critical cycle. The NSW Department of Education and Training action research over other equally positive forms (2010) suggest that: between ‘planning’ and ‘acting’ of action research; come ‘identifying’, ‘informing’ and ‘organising’; between ‘acting’ and ‘observing’ come ‘trialling’, ‘col- 14 critical action research has framed rather than lecting’ and ‘questioning’; between ‘observing’ and changed or shaped social praxis; ‘reflecting’ come ‘analysing’, ‘reporting’ and ‘sharing’; and between ‘reflecting’ and the new cycle comes ‘plan- 15 is elitist whilst purporting to serve egalitarianism; ning’, ‘evaluating’, ‘implementing’ and ‘revising’ (p. 3). 16 assumes an undifferentiated view of action research; Piggot-Irvine et al. (2015) note that, in fact, action 17 is attempting to colonize and redirect action research has no clearly defined ending (p. 549) and the end of one cycle leads into the beginning of the next. research. Lewin (1946, 1948) suggests that action research commences with a general idea and data are sought This critique serves to remind the reader that critical about the presenting situation. The successful outcome action research is problematical. It may be just as con- trolling as those controlling agendas that it seeks to attack (Morrison, 1995b). Indeed Melrose (1996, p. 52) suggests that because critical research is itself value- laden, it abandons neutrality; it has an explicit social agenda that, under the guise of examining values, ethics, morals and politics which operate in a particular situation, is actually aimed at transforming the status quo. This echoes the critique of non-neutral research by Hammersley (2000, 2014) in Chapter 3. For a simple introductory exercise for understanding action research, see the accompanying website. 448
Action research of this examination is the production of a plan of action 1 Diagnosis of a problem, which involves what is to to reach an identified objective, together with a deci- be investigated and the purposes of the action sion on the first steps to be taken. Lewin acknowledges research, for example, to answer a research ques- that this might involve modifying the original plan or tion, to test a hypothesis, to improve practice. idea. The next stage of implementation is accompanied by ongoing fact-finding to monitor and evaluate the 2 Planning an intervention to address the problem, intervention, i.e. to act as a formative evaluation. This which involves considering what the intervention feeds forward into a revised plan and set of procedures will comprise, what data are required and how to for implementation, themselves accompanied by moni- gather them, and what data-c ollection instruments toring and evaluation. Lewin (1948, p. 205) suggests are required. that such ‘rational social management’ can be con- ceived of as a spiral of planning, action and fact-finding 3 Action: putting the intervention into practice, which about the outcomes of the actions taken. involves consideration of timing and duration, par- McKernan (1991, p. 17) suggests that Lewin’s ticipants, contents of the intervention. model of action research is a series of spirals, each of which incorporates a cycle of analysis, reconnaissance, 4 Assessment: how far the intervention has met its reconceptualization of the problem, planning of the objectives of solving the problem, which involves intervention, implementation of the plan and evaluation consideration of how to analyse and interpret of the effectiveness of the intervention. Ebbutt (1985) the data. adds that feedback within and between each cycle is important, facilitating reflection. This is reinforced in 5 Critical reflection and communication of learning: the model of action research by Altricher and Gstettner reflection on the experience and what has been (1993) where, though they have four steps (p. 343) – learned, and sharing this, which involves considera- (a) finding a starting point, (b) clarifying the situation, tion of how to disseminate the findings. (c) developing action strategies and putting them into practice, (d) making teachers’ knowledge public – they McAteer (2013) sets out a five-stage process of action suggest that steps (b) and (c) need not be sequential, research: identifying the research question; finding out thereby avoiding the artificial divide that might exist about the present situation; identifying changes (‘action between data collection, analysis and interpretation. steps’) that can be made; evaluating the effects of such Zuber-S kerritt (1996b, p. 84) sets emancipatory changes; and revising the original question as a conse- (critical) action research into a cyclical process of: ‘(1) quence of the findings of the research (pp. 32–3). strategic planning, (2) implementing the plan (action), An alternative, eight-s tage model is thus: (3) observation, evaluation and self-evaluation, (4) crit- ical and self-c ritical reflection on the results of (1) – (3) Stage 1: Decide and agree one common problem that and making decisions for the next cycle of research.’ you are experiencing or need that must be addressed. Bassey (1998) sets out eight stages in action Stage 2: Identify some causes of the problem (need). research: Stage 3: Brainstorm a range of possible practical solu- tions to the problem, to address the real problem and Stage 1: Defining the inquiry. the real cause(s). Stage 2: Describing the educational context and Stage 4: From the range of possible practical solutions decide one of the solutions to the problems, perhaps situation. what you consider to be the most suitable or best solu- Stage 3: Collecting evaluative data and analysing them. tion to the problem. Plan how to put the solution into Stage 4: Reviewing the data and looking for practice. Stage 5: Identify some ‘success criteria’ by which you contradictions. will be able to judge whether the solution has worked Stage 5: Tackling a contradiction by introducing to solve the problem, i.e. how will you know whether the proposed solution, when it is put into practice, has change. been successful. Identify some practical criteria which Stage 6: Monitoring the change. will tell you how successful the project has been. Stage 7: Analysing evaluative data about the change. Stage 6: Put the plan into action; monitor, adjust and Stage 8: Reviewing the change and deciding what to evaluate what is taking place. Stage 7: Evaluate the outcome to see how well it has do next. addressed and solved the problem or need, using the success criteria identified in Stage 5. Moroni (2011), deliberately echoing Zuber-Skerritt Stage 8: Review and plan what needs to be done in (1996b), sets out a five-step process of action research: light of the evaluation. 449
Methodologies for educational research The key features of action research here are: Step 3 OO it works on, and tries to solve, real, practitioner- The third step, in some circumstances, may involve a identified problems of everyday practice; review of the research literature to find out what can be learned from comparable studies, their objectives, pro- OO it is collaborative and builds in teacher involvement; cedures and problems encountered. OO it seeks causes and tries to work on those causes; OO the solutions are suggested by the practitioners Step 4 involved; The fourth step may involve a modification or redefini- OO it involves a divergent phase and a convergent tion of the initial statement of the problem at Step 1. It may now emerge in the form of a testable hypothesis, phase; or as a set of guiding objectives. Sometimes change OO it plans an intervention by the practitioners agents deliberately decide against the use of objectives on the grounds that they have a constraining effect on themselves; the process itself. It is also at this stage that assump- OO it implements the intervention; tions underlying the project are made explicit (e.g. in OO it evaluates the success of the intervention in solving order to effect curriculum changes, the attitudes, values, skills and objectives of the teachers involved must be the identified problem. changed). We set out below our own eight-step process of action Step 5 research, drawing together the several strands and steps of action research. The fifth step is concerned with the selection of research procedures – sampling, administration, choice Step 1 of materials, methods of teaching and learning, alloca- tion of resources and tasks, deployment of staff and so The first step involves the identification, evaluation and on. Here it must be stated that embedded within the formulation of the problem (widely defined, e.g. to overall scope of the term ‘action research’ might be a include a need for innovation) perceived as critical in number of different research designs that include dif- an everyday teaching situation. McAteer (2013) sug- ferent methods of gathering data. A piece of action gests that the problem should: be related to improving research might include, for example: one’s own practice; enable explanations and hypotheses to be developed (relating them to a broader base of OO an initial and end-o f-intervention survey (a pre- and theory); be within the action researcher’s own power post-s urvey); and control to change; and be professionally and per- sonally important and pertinent (p. 28). OO an experimental or quasi-e xperimental design (e.g. where some students/teachers are involved in the Step 2 intervention and some are not, or where pre- and post-testing of students/teachers is undertaken); The second step involves preliminary discussion and negotiations among the interested parties – teachers, OO a longitudinal study (over the duration of the researchers, advisers, sponsors, possibly – which may intervention); culminate in a draft proposal. This may include a state- ment of the questions to be answered (e.g. ‘Under what OO participant and non-p articipant observation; conditions can curriculum change be best effected?’ OO interviews and field notes; ‘What are the limiting factors in bringing about effec- OO one or more case studies; tive pedagogical change?’ ‘What strong points of action OO documentation from, and about, participants; research can be employed to bring about assessment OO questionnaire data. change?’). The researchers in their capacity as consult- ants (or sometimes as programme initiators) may draw In this respect readers are advised to go to the chapters upon their expertise to bring the problem more into in this book that address these methods, in particular on focus, possibly determining causal factors or recom- case study, experiments and quasi-e xperiments, and mending alternative lines of approach to established observation. Many novice researchers are unsure ones. This is often the crucial stage for the venture as it whether their research is action research or a case is at this point that the seeds of success or failure are study; indeed it may be both, but a distinguishing planted, for, generally speaking, unless the objectives, feature may be whether the research involves an purposes and assumptions are made perfectly clear to intervention on the part of the researcher(s), or whether all concerned, and unless the role of key concepts is the data are largely only collected. If it is the former stressed, the enterprise can easily miscarry. 450
Action research – concerning change, development and intervention – project (cf. Woods, 1989). Discussions of the findings then it may be action research, whereas if it is largely take place in the light of previously agreed evaluative the latter, then it may be more of a case study; one has criteria. Errors, mistakes and problems are considered. to be very cautious in making this distinction because A general summing-u p may follow this in which the there can be gross overlaps between the two. outcomes of the project are reviewed, recommenda- As action research is intended to bring about a change, tions made, and arrangements for dissemination of with an intervention involved, then the researcher may results to interested parties decided. wish to use an experimental or quasi-e xperimental At every stage there is reflection and self-reflection, approach in the action research in an attempt to identify addressing reflexivity (discussed below). This eight- causality through a controlled intervention, with control step process is set out in Figure 22.1. It does not neces- and experimental groups (see Chapter 20). sarily follow a linear sequence, and steps may be recursive and in a different sequence. As Figure 22.1 Step 6 indicates, evaluation and reflection accompany every stage of the process. Reflection can be descriptive (per- The sixth step is concerned with the choice of the eval- sonal, looking back at what has happened), perceptive uation procedures to be used and takes into considera- (e.g. emotional), receptive (relating views of others to tion that evaluation in this context will be continuous. one’s own views), interactive (lining the past and present to future action) and critical (interrogating the Step 7 context in which the teacher operates) (McAteer, 2013, p. 26). It can engage a retrospective analysis of critical The seventh step is the implementation of the interven- incidents: those which make a significant difference or tion itself (over varying periods of time). It includes: sudden solution to a situation (p. 72). the conditions and methods of data collection (e.g. fort- This is a basic framework; much activity of an inci- nightly meetings, the keeping of records, interim dental and possibly ad hoc nature will take place in and reports, final reports, the submission of self-evaluation around it. This may comprise discussions among teach- and group-e valuation reports, etc.); the monitoring of ers, researchers and students; regular meetings among tasks and the transmission of feedback to the research teachers or schools to discuss progress and problems team; and the classification and analysis of data. and to exchange information; possibly regional confer- ences; and related activities, all enhanced by current Step 8 hardware and software. The eighth step involves the interpretation of the data; inferences to be drawn; and overall evaluation of the Problem identification How well intervention Possible interventions solved the problem to address problem Review and evaluate REFLECT AND Decision on particular intervention EVALUATE intervention Monitor and record Plan intervention, with implementation/effects success criteria Implement the intervention FIGURE 22.1 A framework for action research 451
Methodologies for educational research Hopkins (1985), McNiff et al. (1996), McNiff and the group of action researchers, indicating clearly Whitehead (2009) and McNiff (2010) offer much practi- the progress that has been made; cal advice on the conduct of action research, including OO bring in outsiders (e.g. external consultants) where ‘getting started’, operationalization, planning, monitor- appropriate, for example, to provide legitimation for ing and documenting the intervention, collecting data the action research; and making sense of them, using case studies, evaluating OO ensuring that the action research enables participants the action research, ethical issues and reporting. We urge to put their educational values into practice. readers to go to these helpful sources. These are useful introductory sources and guides for practice, in particular It is clear from this list that action research is a blend of McNiff (2010). Indeed McNiff (2010) takes the reader, practical and theoretical concerns; it is both action and novice or experienced, through key features of action research. research, including: what it is, how it differs ‘traditional In conducting action research the participants can be research’ and how it fits western research traditions; why both methodologically eclectic and can use a variety of people should do it; how to do it; who can do it; where instruments for data collection: questionnaires, diaries, to do it; what it involves; how to start and how to iden- interviews, case studies, observational data, experimen- tify a concern; matters of values; kinds of action; how to tal design, field notes, photography, audio and video reflect on its different aspects and elements; action plan- recording, sociometry, rating scales, biographies and ning, data collection and analysis; how to ensure that accounts, documents and records, in short the full conclusions are fair, valid and reliable; how to judge its gamut of techniques (for a discussion of these see Parts significance; implications of the action research for dif- 3 and 4 of the present volume). ferent parties; how to write up, report and disseminate action research, and how to use it in the development of 22.8 Reporting action research a professional portfolio. Along the way, she raises a range of clearly expressed questions, points for reflec- McNiff and Whitehead (2009, p. 15) suggest that, in tion, including action perspectives and research perspec- reporting action research, it is important to note not tives (p. 22), and ‘difficult questions’ such as ‘Whose only the action but the research element, including the practice?’ ‘Whose research?’ ‘Whose voice?’ ‘Whose rigorous methodology and interpretation of data, and theory?’ and ‘Who speaks?’ (p. 57). She stresses the that it is important to state clearly: importance of setting evaluative criteria. Without success criteria it is impossible for the researcher to know OO the research issue and how it came to become a whether, or how far, the action research has been suc- research issue in the improvement of practice; cessful. Action researchers could ask themselves ‘How will we know whether we have been successful?’ Her OO the methodology of, and justification for, the inter- volume is a tour de force for action researchers. vention, and how it was selected from among other Kemmis and McTaggart (1992, pp. 25–7) offer a possible interventions; useful series of observations for beginning action research which involves: OO how the intervention derived from an understanding of the situation; OO convening and organizing a group of action researchers as participants, even if the group is OO what data were collected, when and from whom; small; OO how data were collected, processed and analysed; OO how the ongoing intervention was monitored and OO being prepared to start small and expanding over time, keeping a focus on the longer term and larger reviewed; issues (e.g. whole-s chool issues) as well as the OO how reflexivity was addressed; shorter term and immediate issues; OO what were the standards and criteria for success, and OO setting time frames and actions in them, including how these criteria were derived; support and development activity; OO how conclusions were reached and how these were OO building in tolerance, involvement of all participants validated; and support as participants learn by doing, reflecting OO what and how the researcher learnt as a consequence on what is happening and taking responsibility for actions and consequences; of the action research; OO how practice was changed as a consequence of the OO scrupulously recording developments and progress in a timely fashion, and disseminating these beyond findings. The authors note that validity is a key concern in reporting, that warrants have to be justified for the conclusions drawn, that these warrants reside in the 452
Action research evidential trail provided in the research (p. 23) and that OO what possible interventions were considered, and reflection and reflexivity must be demonstrated (p. 28). why some of these were rejected/accepted (e.g. on It was noted at the start of this chapter that the goal of what criteria); action research is improvement; therefore the report must indicate not only what the improvement was, but that it OO how the intervention was planned and implemented; was attributable to the intervention and not to other OO how ongoing data were gathered, processed and factors, i.e. that causality is demonstrated. This requires a level of rigour that is indicated in the ‘research’ part of used during the action research; the ‘action research’. More than this, given that action OO what were the roles of the action researcher; research concerns research, the report should indicate not OO what was discovered during, and as a consequence only how the research led to improvement in practice, but how the action research in question contributes to the of, the action research; expansion of knowledge, scholarship and scholarly OO what conclusions were drawn, and how they were enquiry, i.e. what significance the research has for both the academic and professional communities. The report, valid (their warrants); then, serves a dual set of criteria: (a) criteria for the plan- OO an indication of the significance of the action ning, conduct, reporting and evaluation of the research; and (b) criteria for the planning, conducting, reporting research – for action and for research; and evaluation of practice/action. OO an indication of how practice was modified and Given that the intervention is into a ‘real-life’ situa- tion, it is important to include in the report some infor- improved as a consequence of the action research; mation about the ‘real-life’ context of the intervention, OO an indication of, and justification for, the success so that the reader has a clear picture of this. This means that the report must include necessary descriptive data criteria used to evaluate the action research; (McNiff and Whitehead, 2009, p. 37), together with OO the reflections of the action researcher, together with scholarly enquiry (e.g. a literature review), explana- tions, reflections, research methodology, data collec- evidence of growth in reflective ability (and the cri- tion, analysis and interpretation, consideration of teria used to evaluate this). alternative explanations, and, of course, evidence that there has been an improvement in practice and in the The action researcher has to adopt a potentially schizo- development of the researcher (e.g. in terms of peda- phrenic stance to the action and the research, being gogy, subject knowledge, researcher ability and skills, both in it and of it, but also having to stand back from reflective capacity). the situation and viewing it with as much objectivity as Action research may be reported in narrative form possible; subjectivity and objectivity (or, perhaps (e.g. McNiff and Whitehead, 2009, p. 49; McNiff, better, relative subjectivity and objectivity) are com- 2010) and must be written with the reader in mind. An bined in a single action researcher. action research report should address (cf. McNiff and Whitehead, 2009, p. 56; McNiff, 2010): 22.9 Reflexivity in action research OO the action researcher’s concern and the reason for The analysis so far has made much of the issue of that concern; reflection, be it reflection-in-action, reflection-o n-action or critical reflection. Reflection, it has been argued, OO an indication of the presenting situation at the start occurs at every stage of action research. Beyond reflec- of the action research; tion, reflexivity is central to action research, because the researchers are also the participants and practitioners in OO a review of what, how and why the action researcher the action research – they are part of the social world moved into action and reflection; that they are studying (Hammersley and Atkinson, 1983, p. 14). Hall (1996, p. 29) suggests that reflexivity OO what methodology, design and data were used in the is an integral element and epistemological basis of action research (e.g. it was suggested earlier that emancipatory action research because it takes as its embedded in action research might be a case study, basis the view of the construction of knowledge in an experimental or quasi-experimental approach, a which: (a) data are authentic and reflect the experiences survey, an ethnography); of all participants; and (b) democratic relations exist between all participants in the research; the researcher’s OO what were the research questions; views (which may be theory-laden) do not hold prece- OO what were the problems that the action research was dence over the views of participants. Reflexivity requires a self-c onscious awareness of intended to address/solve; the effects that the participants-a s-practitioners-a nd- researchers are having on the research process, how their values, attitudes, perceptions, opinions, actions, feelings etc. are feeding into the situation being studied 453
Methodologies for educational research (akin, perhaps, to the notion of counter-transference in OO how to address ‘bad news’, i.e. reporting negative counselling). The participants-as-practitioners-a nd- results and presenting results in a bad light; researchers need to apply to themselves the same criti- cal scrutiny that they are applying to others and to the OO informed consent; research, as discussed in Chapter 14. OO the increased workload on participants that is likely Reflexivity also links to awareness of possible bias, in that the practitioner is also the researcher and may to come with action research; not be entirely disinterested. For example, in an attempt OO protection of vulnerable people; to impress a senior manager, a teacher who is an action OO recognition that protection from harm trumps per- researcher may present a rosier picture of the outcome of the action research than is really the case, or, by con- sonal beneficence or benefit; trast, a teacher who may be pressing for increased OO equitable selection and inclusion of participants. resources may present the outcome more negatively than it is. Here ethics, validity and political agendas Locke et al. (2013) set out key ethical principles for coincide. action research (pp. 113–14): 22.10 Ethical issues in action OO respect for all participants as stakeholders who genu- research inely share decisions (the ‘principle of inclusivity’); Action research is not exempted from the ethical issues OO respect for all participants, in whatever roles, as ‘full that were identified in Chapter 7. It requires the informed members’ of the action research group (the ‘princi- consent of participants, options for teachers/students not ple of maximal participant recognition’); to take part, and with no penalty (Nolen and Vander Putten, 2007), and confidentiality and autonomy of par- OO aims, content and operation of the research and ticipants to be respected. Whilst referring the reader to ownership of the data and report agreed and decided Chapter 7, we also note that there is a blurred dividing in consultation by all participants (‘the principle of line between the teacher qua teacher and qua researcher, negotiation and consensus’); and that effective teaching also concerns effective researching. Perhaps, also, the fact that minors attend OO rights of withdrawal and renegotiation of grounds school on a compulsory basis already gives the teacher for participation (‘the principle of communicative automatic right to research them as part of her everyday freedom’); teaching. Where is the dividing line? Gibbs et al. (2016) draw attention to the ethical OO use of plain, comprehensible language by all parties challenges faced by insider and outsider action (‘the principle of plain speaking’); researchers, for example: ‘fiduciary responsibilities’ to whom and for what; the principle of ‘do no harm’ OO ensuring that all ‘members’ collaboratively adjudi- (p. 7); ownership of intellectual property (pp. 11–12). cate the moral rightness of the aims, processes and Locke et al. (2013, pp. 109, 119–20) and Dens- understandings of the research (‘the principle of combe (2014, p. 127) identify a range of ethical issues right action’); that action research should address: OO ensuring questioning of, and transparency in, the ‘dis- OO how to maintain confidentiality whilst acknowledg- cursive assumptions’ that participants bring to the ing others’ contributions, and how to address the research (‘the principle of critical self-reflexivity’); balance between confidentiality and disclosure; OO ensuring that the feelings of all participants are OO the potential knock-o n effects of the action research respected and count (‘the affective principle’). on participants and other relevant parties; 22.11 Some practical and theoretical OO how to avoid doing harm to participants (e.g. from matters disclosure); Much has been made in this chapter of the democratic OO how to corroborate the data and interpretation; principles that underpin some types of action research. OO the need to seek approval and clearance for the The ramifications of this are several. For example, there must be a free flow of information between participants research (i.e. simply because it is action research and communication must be extensive (Elliott, 1978, does not exempt the researcher from seeking ethical p. 356). Further, communication must be open, uncon- approval); strained and unconstraining – the force of the better argument in Habermas’s ‘ideal speech situation’. That this might be problematic in some organizations has been noted by Holly (1984, p. 100), as action research and schools are often structured differently, with schools being hierarchical, formal and bureaucratic whilst action research is collegial, informal, open, 454
Action research collaborative and crosses formal boundaries. In turn this OO How can we formulate a method of work which is suggests that, for action research to be successful, the sufficiently economical as regards the amount of conditions of collegiality have to be present, echoing data gathering and data processing for a practitioner Habermas’s ‘ideal speech situation’, for example (Mor- to undertake it alongside a normal workload, over a rison, 1995a, 1998, pp. 157–8, 2011, p. 153): limited timescale? OO participatory approaches to decision making; OO How can action research techniques be sufficiently OO democratic and consensual decision making; specific to enable a small‑scale investigation by a OO shared values, beliefs and goals; practitioner to lead to genuinely new insights, and OO equal rights to determine policy; avoid being accused of being either too minimal to OO equal voting rights on decisions; be valid, or too elaborate to be feasible? OO the deployment of sub-groups who are accountable OO How can these methods, given the above, be readily to the whole group; available and accessible to anyone who wishes to OO shared responsibility and open accountability; practise them, building on the competencies which OO an extended view of expertise; practitioners already possess? OO judgements and decisions based on the power of the OO How can these methods contribute a genuine argument rather than the positional power of the improvement of understanding and skill, beyond advocates; prior competence, in return for the time and energy OO orientation to a common interest ascertained without expended – that is, a more rigorous process than that deception; which characterizes positivist research? OO everyone’s freedom to enter a discourse, to check questionable claims, to evaluate explanations, to Another issue of some consequence concerns head modify a given conceptual framework and to reflect teachers’ and teachers’ attitudes to the possibility of on the nature of both knowledge and political will; change as a result of action research. Hutchinson and OO everyone’s freedom to assess justifications and to Whitehouse (1986), for example, note possible resist- alter norms; ance from headteachers and teachers themselves. OO mutual understanding between participants; Further, Jones and Stanley (2010) comment that OO recognition of the legitimacy of each subject to par- action research involving university researchers and ticipate in the dialogue as an autonomous and equal public stakeholders seriously challenges ‘the demo- partner, with equal opportunity for discussion; cratic principles commonly associated with this genre OO discussion to be free from domination and distorting of critical enquiry’ (p. 161), as micro‑politics can frus- or deforming influences; trate the endeavour to be truly democratic. Kemmis OO all motives except for the cooperative search for (2006) and Gibbs et al. (2016) question whether indi- truth are excluded; vidual or even collaborative action research can really OO the speech act validity claims of truth, legitimacy, live up to its claim to radically challenge and change sincerity and comprehensibility are all embodied; injustice, i.e. whether its putative emancipatory poten- OO illocutions (where the outcome is open or negotia- tial is realizable in practice, and that it is more descrip- ble) replace perlocutions (achieving a given, prede- tive of the reflective process than being evaluative or termined, non-n egotiable outcome by saying emancipatory (p. 7). something); Piggot-Irvine et al. (2015) comment that more needs OO shared ownership of decisions and practices. to be done to evaluate the outcomes and impact of action research rather than its predominant focus on Zuber-Skerritt (1996b, p. 90) suggests that the main process, and they indicate criteria, foci and indicators barriers to emancipatory action research are: (a) single- for evaluating precursors/foundations, processes/activi- loop learning (rather than double-loop learning) ties, outcomes and impacts. Similarly, Heikkinen et al. (Argyris, 1990); (b) overdependence on experts or (2012) suggest five principles for the validation of seniors to the extent that independent thought and action research: historical continuity (locating the expression are stifled; (c) an orientation to efficiency research in its antecedents and unfolding course of rather than to research and development (one could add action); reflexivity (awareness and disclosure of the here ‘rather than to reflection and problem posing’); (d) impact of personal experience on the research); dialec- a preoccupation with operational rather than strategic tics (inclusion of multiple voices in the research and its thinking and practice. She suggests (1996a, p. 17) four interpretation); workability and ethics (whether the practical problems that action researchers might face: research has led to changes in practice and addresses ethical issues transparently); and evocativeness (how 455
Methodologies for educational research effectively the research evokes images and emotions in Action research is a flexible, situationally responsive the reader). Pring (2015, p. 160) adds that including methodology that offers rigour, authenticity and voice. externality in action research can increase the public This chapter has tried to expose both the attractions perception of its validity and rigour, though how far and problematic areas of action research. In its thrust this addresses ‘objectivity’ is a moot point. towards integrating action and research one has to question whether this is an optimistic way of ensuring 22.12 Conclusion that research impacts on practice for improvement, or whether it is a recessive hybrid. There are Action research is a potential means of empowering several journals that focus on action research in educa- teachers, though this chapter has questioned the extent tion, for example: Educational Action Research and of this. As a research device it combines six notions: Action Research Reflective Practice, and some key websites: 1 A straightforward cycle of identifying a problem, planning an intervention, implementing the inter- http://cadres.pepperdine.edu/ccar/about.html (the Centre vention, evaluating the outcome. for Collaborative Action Research) 2 Reflective practice. www.jeanmcniff.com (the website of Jean McNiff ) 3 Political emancipation. www.actionresearch.net (the website of Jack Whitehead) 4 Critical theory. http://aral.com.au (the website of Bob Dick) 5 Professional development. 6 Participatory practitioner research. Companion Website The companion website to the book includes PowerPoint slides for this chapter, which list the structure of the chapter and then provide a summary of the key points in each of its sections. These resources can be found online at www.routledge.com/cw/cohen. 456
Virtual worlds, social CHAPTER 23 network software and netography in educational research Stewart Martin Virtual worlds, social network software and netography In a virtual world, individuals can project their are key features of contemporary concern in educa- actions, views or values through their avatar and tional research. This chapter introduces researchers to receive feedback from others in the system.1 Projection this field, addressing: techniques can encourage a sense of authenticity to externalize the self and create an impression of pres- OO key features of virtual worlds ence in an environment that we are not physically part OO social network software of. This relies on the ‘willing suspension of disbelief ’ OO using virtual worlds and social media in educational (Coleridge, 1817) to create an ‘illusory shift in point of view’ (Dennett, 1978, p. 312) as well as on the use of research our own knowledge, imagination and enthusiasm OO netography, virtual worlds and social media network (Zhao, 2003). Our sense of being present in the physi- cal world appears to be an essential component of con- software sciousness but is not something we normally think OO opportunities for research with virtual worlds, social about unless prompted by a displacement of our self- perception, for example, through a dream, literature, network software and netography film or theatrical experience. Any relative lack of OO ethics realism is not therefore an obstacle to user acceptance OO guidelines for practice of a virtual world and many early examples were effec- OO data tive and engaging, despite having poor graphics; some included no pictorial images at all (Nelson and Erland- 23.1 Introduction son, 2012; Martin, S., 2014). Virtual worlds are computer-b ased multi-user simula- 23.2 Key features of virtual worlds tions that are shared online by individuals, who appear as a graphical character: an interactive icon or avatar. Virtual worlds are often three-d imensional, visually Although creative inventions, these environments have realistic and attractively designed; they can be useful in persistence; they continue to exist and change with pos- stimulating participants’ imagination where the sense sible consequences for an avatar, even when a user is of being in a real place may be important, for example, not present online, because of the ongoing actions of to encourage engagement in exploring moral dilemmas other online users. Current computer technology (Martin, 2015). Creating a sense of life as a virtual enables virtual worlds to offer increasingly convincing experience may also be valuable for investigating three-d imensional imagined environments for individu- socially sensitive issues by invoking a feeling of a ‘safe als to explore alone or as a member of a community distance’ between the individual and the thing being and, because of their great versatility and realistic explored, as the avatar becomes the presenter of a par- appearance, they are commonly dedicated to entertain- ticular perspective or behaviour. Such ‘projection’ may ment or social interaction or are found in military or help to displace any associated difficulty or stress away commercial applications. However, they are increas- from the individual and ease the exploration of sensi- ingly used in educational and research settings for tive or highly charged issues (Freud, 1936). Virtual exploring languages, ethics, management, history, psy- worlds can also be valuable for developing otherwise chology, design, pedagogy and a rapidly widening inaccessible or impossible environments to explore range of other areas (see Peachey et al., 2010; Hinrichs and Wankel, 2011; Hunsinger and Krotoski, 2012; Gregory et al., 2015). 457
Methodologies for educational research human interaction, agency, values and perceptions, 23.3 Social network software such as historical locations and experiences, future imagined scenarios or uninhabitable settings. Social network software is constantly developing, These features make virtual worlds particularly increasingly multimodal and dynamic, and is used for a effective training environments, where the depiction of wide range of purposes including communication, self- a hostile environment (e.g. fire-fighter training for casu- expression, maintaining friendships, sharing informa- alty location) can be enhanced to offer more realism tion and for enjoyment. Boyd and Ellison (2007) define without physical risk, or otherwise impossible experi- them as ‘web-b ased services that allow individuals to ences can be created, such as that of being immobile or construct a public or semi-p ublic profile within a disabled, or being a member of a different social or bounded system’ (p. 211); Kaplan and Haenlein (2010) ethnic group. add that they are also ‘a group of Internet-based appli- The low risk and ‘repeatability’ of experiences in cations that build on the ideological and technological virtual worlds have advantages not just because of the foundations of Web 2.0, and that allow the creation and safety in otherwise dangerous or unpredictable environ- exchange of User Generated Content’ (p. 61). All ments, but also because, despite their heightened sense media foster communication and so to some degree are of realism compared to other approaches, they afford social by definition (Papacharissi, 2011) and it may discardable experiences at relatively lower personal, often be useful to use one or more of them as part of experiential or emotional cost, for example: when train- data gathering and communication in research. ing armed forces in decision making in pressured situ Social media vary and include networking services ations; or training medics to treat ‘real’ casualties; or for sharing personal information (e.g. Facebook); sites advancing views, proposals or identity depictions in for content sharing or discussion (e.g. Flickr, YouTube, hostile situations (Waller et al., 1998; Martin, S., 2013). Instagram, WhatsApp, Digg or Reddit); forums (e.g. Virtual worlds therefore have advantages of economy wikis such as wiktionary.org); professional networking (being cheaper to create and use than real-life settings); sites (e.g. LinkedIn); blogs (e.g. Twitter, Blogspot); of visibility (important things can be made clearer and virtual social worlds (e.g. Habbo Hotel, Club Penguin, more accessible); of control (much more control of the Second Life); and massive multiplayer online role- setting is possible than in real life); and of safety, where playing games (MMORPGs) such as World of situations can be used that in real life would be too dan- Warcraft, Final Fantasy and Skyrim. Although used for gerous, difficult, sensitive or ethically questionable different purposes, these technologies can have features (Bailey, 2007; Nelson and Erlandson, 2012). in common; platforms such as Twitter or Facebook Together with a configurable avatar, these features have shared content that is generated, blended or re- make virtual worlds effective technologies for studying posted by users. and affording rehabilitation experiences for individuals who have undergone traumatic experiences such as 23.4 Using virtual worlds and social domestic violence, or by offering opportunities for media in educational research people to make otherwise unavailable choices regarding actions, gender or personality. Their potential for exploit- The popularity of social network sites reflects the sig- ing role/real playing and the blending of physical real- nificance of communication in contemporary society world activity with virtual activity offers scope for and influences values, ideas and behaviours that are research using innumerable scenarios. These are envi- often of interest to researchers. Digital spaces have ronments in which participants could project, share and architectures that affect the way people work and live reflect on their own and others’ actions and views, which in them, and social media offer opportunities to circum- may lead to the surfacing and exploration of further, vent traditional power relationships and create a greater often sensitive, issues for investigation and possible res- likelihood that individuals may feel free to express olution, the reconciliation of potential conflict and disa- themselves. Virtual worlds and social media can there- greement, and hence to the development and enlargement fore help individuals to communicate more freely and of understanding. Although the Internet and its increas- engage with a kind of ‘ideal speech’ (Habermas, 1979, ingly varied digital spaces offer some distortions in the 1982, 1984, 1987b; see Chapter 14 of the present portrayal of everyday life, through the creation of virtual volume). These ideal speech features can support spaces and communication within them, participants can research by encouraging participants to more openly: be encouraged to be more open, honest and authentic in disclosing their views, their values and their beliefs about OO enter a discourse and check questionable claims; real or created situations and issues. OO evaluate explanations and assess justifications; 458
Virtual worlds, social network software and netography OO modify a given conceptual framework and influence fuse authorship and readership and create what some norms; see as new expressions and forms of literacy that also offer research opportunities (Greenhow and Gleason, OO reflect on the nature of political views and action; 2012; Stevens et al., 2015). OO develop mutual understanding between participants, Social media offer ways of exploring the space between institutionally managed systems and non- accept that consensus is derived from the better institutional personal usage; such liminal (boundary) argument and not from the influence or authority of areas are seen as fascinating ‘third spaces’ (Bhabha, individual participants, and accept that the coopera- 2004; Turkle, 2007; Aaen and Dalsgaard, 2016). tive search for truth should be the only motive; However, participants may prefer to keep their social OO recognize the legitimacy of others to participate in and academic presences separate, so it may be better to the dialogue as an autonomous and equal partner, explore informal learning using media such as Face- and to promote equal opportunity for discussion. book or Moodle and use other methods for traditional, more formal activity. The division between first (after Morrison, 1995a, p. 102) (formal) learning spaces and second (informal) spaces can be conscious and functional rather than uncon- These features may influence who participates and what scious, accidental or disruptive; a number of studies kinds of things are shared, as not everyone may feel have found that students and their teachers often dislike comfortable in such contexts; this should be borne in blending study and their social life (Manca and Ranieri, mind when considering the topic and desired partici- 2013). Research may therefore encounter resistance to pants for a research project. Cultural or individual dif- using social network technologies for educational uses, ferences may also predispose some individuals to as participants may prefer more traditional approaches accept or decline an invitation to participate and so to learning; in the interests of promoting high-q uality affect the sample, with implications for the representa- research, any such preferences should be incorporated tiveness of a study and for the nature and robustness of and not overlooked. Additionally, while many Face- its possible conclusions. book groups are ‘open’ (public), many are ‘closed’, and Since its founding in 2006 as a simple way to help participation may require an invitation (consent) from friends stay in touch, the microblogging and social group members. network site Twitter has become a popular communica- tion medium alongside its earlier counterpart Facebook. 23.5 Netography, virtual worlds and The use of Twitter and Facebook in many western high social media network software schools is common as young people create content and tag and communicate this to interact socially in school Developed by Kozinets (2002, 2010), netography and in informal spaces beyond school. Users typically (network ethnography) relies heavily on observation set Twitter accounts to ‘public’ to share their social and, like traditional ethnography, is an immersive and lives widely and this information can be used to study interpretive exploration of a particular space – in this connections between people, their self-e xpression and case of embedded technology. The traditional pro- engagement with resources and learning communities. longed immersion of ethnographic research is equally Both Facebook and Twitter, as ‘always on’ technolo- effective in digitally mediated settings, but engagement gies, are thought to particularly encourage social par- may be more about following connections than contin- ticipation and interaction by facilitating communication uous physical presence in one ‘space’. As a general and collaboration among students and with teachers, rule, researchers should try to engage as fully as possi- peer assessment and learning. ble with the digital environment being studied, but this Despite the ubiquity and high consumption of such does not mean that they must participate in every activ- media, surprisingly little research has been done to ity in order to conduct meaningful observation. Some- explore why and how they are used to make meaning times full participation is not possible or desirable and and this offers considerable opportunity for new studies total immersion is not essential in ethnography. None- (Gleason, 2016). The use of such technologies in theless, such factors may significantly affect how other formal settings such as educational institutions also participants perceive and engage with the researcher, offers rich opportunities to study changes to the histori- and sufficient expertise with the environment will be a cal ‘imbalance of power within most educational uses prerequisite for conducting any study, and the persona of technology’ (Selwyn, 2010, p. 71). Twitter and Face- being portrayed may need to be consistent and coherent book, like all digital technologies, have particular his- across all digital platforms employed. tories, constraints and affordances that bear on the ways people use them to communicate. Some, like Twitter, have potential to blend private and public spaces and to 459
Methodologies for educational research Netography often uses multifaceted engagement granted by participants in a given setting (so ‘normal’) with a setting for studying online hyperlinked resources that they are not remarked or reflected upon by them such as websites, Facebook pages or blogs which have (Malinowski, 1922). Not all data are therefore accessi- been created by individuals to project their ‘fame’ ble through interviews or other traditional means of (compare with Malinowski, 1922; Munn, 1986). Such data recording (indispensable though these are) and so studies may include a number of sites and different participant observation plays an especially important forms of data collection such as: formal interviews; role. Participant observation within virtual worlds and observing individuals both online and in the physical social network platforms can offer significant opportu- world (e.g. in cybercafes or game conventions, or nities, as the extended timeframe allows the researcher whilst physically present but online); watching and to reflect, revise classifications, assumptions and analy- chatting to members of the online community being sis and to discuss with participants any emerging studied, or during casual encounters online and offline. themes and interpretations. These promote reflection Big data may also be important in this field and Chapter and the revision of assumptions, classifications and 8 contains an exploration of the implications of its use. analysis. The ‘participant’ role of the researcher entails A sense of presence (‘being there’) in these spaces searching for unremarked-u pon things proactively and is likely to be highly individual and conditional upon being aware that what is observed, even if only a par- the user’s level of control over the environment tially glimpsed pattern or behaviour, may be valuable (Sheridan, 1992; Ijsselsteijn et al., 2000; Sadowski and and should be recorded and followed up, even if Stanney, 2002). An immersive experience results when (perhaps especially if ) the rest of the community presence in an environment is augmented by its appar- appears to pay it little attention. These processes make ent overall fidelity to physical reality (Slater and Steed, short studies more difficult as they require substantial 2007) and appears when being there is augmented by a and sustained time and commitment from the total response of ‘making sense there’ (Schuemie et al., researcher, so when planning a project it is important to 2001; Riva et al., 2003). Presence in an online commu- allow for regular engagement with the community of nity is often a prerequisite for engagement; new partici- interest over extended periods (e.g. weeks and months). pants often have to be invited or accepted and there Before 1910, there was surprisingly little in the lit- may be official gatekeepers or a formal arrangement for erature of the social sciences about the now common- gaining access – a rite of passage that existing commu- place notion of ‘community’ and the first clear nity members will have gone through so as to be able definition of it focused on defining rural communities to participate comfortably in the digital environment. around villages (Galpin, 1915); in the forty years fol- Sometimes it may instead just be a case of building an lowing, over 100 other definitions appeared (Hillery, identity and presence in whatever ways the platform 1955). The notion of ‘community’ in the social sciences allows, and creating an array of connections over time. grew to include the idea of community as a value (see Facebook involves signing up for an account, decid- Frazer, 2002) and increasingly embraced overlapping ing what photographs and information to upload, which descriptions or ideas that mingled together and became individuals to ‘friend’ (connect to) and how often to difficult to separate, as, for example, when applied to update information; to use Twitter the researcher must the study of ideas of solidarity, trust, mutuality, fellow- decide on a name and what personal information to ship or conflict. The more recent additions of commu- reveal in the profile and the nature of any ‘tweets’ or nities that are ‘imagined’ or ‘virtual’ have extended this ‘retweets’ (messages) to be made. These processes are range and complexity even further (e.g. Anderson, relatively straightforward; but platforms such as World 1983; Rheingold, 2000). of Warcraft and Final Fantasy are likely to require con- The first task of a researcher studying a community siderable familiarity and skill with navigation, raids, is therefore to clarify what is to be meant in their study questing and other features of the environment to facili- by ‘community’, as communities within digital envi- tate extended contact with other players and acquiring ronments are often only very loosely bounded and these might take significant time (Sveinsdottir, 2008). dynamic by nature; hence it is important to be able to The purpose of both physical world and digital eth- distinguish who and what is ‘in’ and ‘out’ of the study nography is not just empirical description but also and how this is to be decided. This is what ethnogra- about developing a theoretically rich description that phers would usually think of as identifying the ‘field- relates to the particular issue or area of study and con- site’, which may traditionally have been one or more nects with wider discussions (Hine, 2015). Many geographical areas with relatively clear boundaries (e.g. important norms, beliefs, attitudes and behaviours that a village), perhaps with more fluid boundaries for par- could be of interest to the researcher can be so taken for ticular social groups. Group ‘types’ are sometimes also 460
Virtual worlds, social network software and netography used this way in researching the digital world and to minimize this effect by adopting only passive obser- might include professional organizations (communities vation strategies may prove unworkable, as online com- of practice), or communities that are workgroups, fami- munities may detect and censure those online who are lies, friendship groups or diasporic groups or guilds of not participating (lurking) and using covert observation people who join together for some other activity (e.g. raises further ethical issues. Visibility is therefore in online games). Researchers may also study online important but does not need to be in every communica- groups, not so much by community membership but by tion medium or to be constant, and it is possible and activity, so they may focus on novice participants, or often acceptable to other participants to observe an those with longer engagement who may act as manag- online discussion group without posting messages. ers or leaders, or those who are present but do not oth- However, avoiding lurking by posting repeatedly erwise engage (‘lurkers’). So a fieldsite can be thought without contributing to the aims and goals of the group of as a collection of places, individuals or practices that would often breach group etiquette and is best avoided. might be physical or virtual or some combination of these; and this kind of study is sometimes called ‘con- 23.6 Opportunities for research with nective ethnography’ where researchers seek to virtual worlds, social network describe the use of these related sites and explore the software and netography connections between them (Leander and McKim, 2003; Hine, 2007; Taylor, 2009). Studies of online communities may involve: study but In netographic research it can be difficult to separate not participation, which some may regard as somewhat the offline and online lives of individuals, as their contradictory and ethically questionable; or study with online engagement may be deeply enmeshed with their some participation; or study plus offline/online inter- life in the physical world. The researcher cannot then views; or study which also includes offline research follow the traditional ethnographic methodology of vis- methods. Deciding on data collection and other appro- iting a physical organization or place because online priate methodology for exploring virtual worlds or and virtual communities exist in placeless spaces. The digital social networks in educational research, there- online/offline distinction also seems increasingly artifi- fore, often involves considering a range of offline and cial, as many new digital technologies claim to have a online tools (Fielding et al., 2008; Markham and Baym, ‘social’ element. The use of social media is participa- 2008). Existing community members may have several tory and collaborative by nature, but it is also often means of communicating, such as email, face-to-face complex and intermingled with issues of identity and meetings or through blogging, so a study may not be social relationships (Selwyn and Stirling, 2016). Some confined to a single ‘site’ or software but could include suggest that these new media challenge and destabilize and draw upon a range of online and offline settings. the very concept of self (Baudrillard, 2012). One con- As a result, mixed methods/mixed worlds/mixed media sequence of this is that maintaining boundaries between approaches are increasingly common (Johnson et al., the research and other aspects of the researcher’s life 2007; Martin et al., 2010), although these can exacer- may become difficult; they may have a digital presence bate some research problems, especially of maintaining that reaches far beyond their project that participants user engagement (see the case study in Livingstone and may access. These complexities arise largely because Bloomfield, 2010). the rise of social network software has shifted Internet Virtual world and social network platforms can use from passive consumption towards active participa- provide environments that reflect and include features tion, from ‘pull’ to ‘push’ behaviours. Conducting par- of ‘real life’ and are valuable for studying interactions ticipant observation in such settings may be difficult to between individuals, especially when we wish to characterize, as when deciding what ‘participation’ explore contexts where sensitive issues may be the means and what exactly is being ‘observed’; the focus. They can be useful for the study of interaction, researcher may be observing interactions which they especially in dynamic, fluid, uncertain or contested have in part created, so are in some senses observing contexts, for exploring complex behaviour and for themselves (Law, 2004). monitoring developments over time. By their nature, However, ethnography never provides a neutral or virtual worlds, especially the perceptually realistic ones objective account of what is studied, as the researcher that are increasingly common, offer the researcher an always plays some part in constructing the object of opportunity to exploit a sense of immersion in the their study, although it is important for them to main- created world (a sense of being there) and also the tain an awareness of this phenomenon and its likely sense of a shared experience with others (a sense of consequences (see Clifford and Marcus, 1986). Seeking being there). Both immersion and co-p resence have 461
Methodologies for educational research been recognized as important facilitators of user presents for fair assessment (Manca and Ranieri, 2016). engagement in a time when media consumers demand The effects of social media use in education tend to be more and deeper experiences (Turkle, 2000; Riva et al., quite diverse and sometimes negative; so whilst their 2003; Boellstorff, 2008). use can be correlated with increased student involve- Behaviours and attitudes towards others can be ment (Junco, 2012) and time on Facebook has been explored in such contexts to study contested opinions associated with academic success (Labus et al., 2015), or beliefs, or individual and self-perceptions in relation it has also been found to be inversely correlated with to others and how these might change over time. This academic progression (Paul et al., 2012). Results from makes virtual worlds useful places for researching the different studies sometimes conflict and so can be diffi- development of understanding, of perception, of proc- cult to integrate, perhaps because different features of esses where negotiated meaning is important, and of particular social media can support diverse forms of the dynamics that generate consensus and discord. involvement and different activities may produce dif- Research should always be focused and highly contex- ferent effects (Matzat and Vrieling, 2016). tualized but also relate to wider issues of interest; Social media can also be used to study a range of virtual worlds and other social network software are demographic and other variables in relation to topics well placed to provide insights into embodiment, self- such as teacher networking, student expectations, peer hood, globalization or learning and many other topics feedback, identity presentation and development, that may be important for those with little interest in support, maintaining relationships and different phases digital environments. of transition, in addition to their possible academic uses, Another advantage of virtual worlds lies in exploring which have received relatively little attention. Social contexts where the experimenter neither desires nor is media can also present some conundrums for educa- able to exert control over every aspect of the situation tional institutions; the lack of accountability and other and where behaviour is shaped by the agency of partici- effects of anonymity can lead to cyber-b ullying, racial pants. Virtual worlds are also highly suited to mixed hostility or the promotion of damaging lifestyles. This methods approaches, the development of research meth- disinhibition effect can occur when anonymity encour- odology and activity design and the exploitation of ages feelings of a ‘safe barrier’ between perpetrator and game theory (Broadribb et al., 2009). Technologies such victim. Control of content and interaction can also be a as Second Life, Club Penguin, Facebook or Twitter are ‘sharing’ that looks like intimacy but is actually a kind collaborative environments suited to ‘inclusive’ research of distancing (Burbules, 2016). However, more positive practices where researchers and subjects engage on outcomes for classroom engagement and study can also equal terms and therefore offer opportunities to develop arise from online disinhibition: for example, students scenarios of ‘ideal speech’ (Rybas and Gajjala, 2007; may more freely share their academic work, or feelings Sheehy, 2010), discussed earlier. and problems that they might not reveal offline. Both Researchers have explored the influence of social positive and negative scenarios highlight the need for media on campus life, identity, sexuality, relationships great care with ethics, where informed consent, data and attitudes towards ‘others’; some feel that such tech- security and participant anonymity may present particu- nologies relax compliance with the social norms that lar challenges (Rowan-K enyon et al., 2016; see also affect inter-p erson spaces, and that this allows observa- Chapter 8 of the present volume). tion of communication with fewer constraints from the Exposure to conflicting values is likely to be an social conventions that may appear in traditional face- unavoidable feature of many social media and research to-face interactions such as focus groups or interviews into or with them is therefore likely to encounter ten- (Pitcher, 2016). Social media also have significant sions. In education, this raises questions for the kinds potential to disrupt established modes of interaction or of educational spaces being fostered, as short and hierarchies of authority and power within institutions, perhaps superficial critique from anonymous others to challenge and sometimes remake them, from settings may expose students to points of view or ideas that may of political activism to those between students and challenge and disturb them. The ‘messy democracy’ of teachers (Selwyn and Stirling, 2016). social media affords particular kinds of robust interac- However, the open, social and collaborative nature tion for which the researcher and participants need to of participatory networks are often seen as also posing be prepared; they can be productive and creative but challenges for education, implying possible shifts in the also can be hypercritical and forceful (Burbules, 2016). roles of learner and teacher, making institutions more Some commentators suggest that: the benefits of porous and raising concerns about untangling collabo- social media have been exaggerated and that using such rative from individual learning and the challenges this technology actually minimizes opportunities for collab- 462
Virtual worlds, social network software and netography oration, as people may work on their own on separate say that participants should know the terms of service parts of a project; such media therefore lead to greater and functionality of platforms such as Twitter or Face- misunderstanding, less knowledge sharing and less cre- book when they open an account. It is likely that many ative or higher-o rder cognitive processes; using social experienced users of Facebook remain unaware that media such as Facebook is time-wasting and that both people who have not ‘friended’ them can nonetheless students and academics dislike the blurring of their access their photo albums, or that images uploaded to social and professional identities and raise concerns social media often include embedded metadata with about how they become represented (Manca and details including author, location, time and much more Ranieri, 2013; Salmon et al., 2015; Stirling, 2016). (see Raynes-G oldie, 2010). Twitter is argued by some to have a zero or negative Social media and virtual worlds present particular effect on learning (Kucuk and Sahin, 2013; Arabaci- problems for traditional frames of reference for ethical oglu and Ajar-Vural, 2014), whilst others find its influ- research, especially when children or young people are ence positive (Evans, 2014; Ricoy and Feliz, 2016). involved. It may be unclear who the participants actually Kirschner (2015) challenges the idea that Facebook can are, to whom the data belongs, what data can legitimately be useful for formal learning outcomes, citing its inade- be regarded as ‘public’ or ‘private’; and the possible con- quacy for academic discussion, knowledge construction sequences for participants and their networks now and in and argument; and others find that it encourages narcis- the future may be unknowable. The main ethical issues sism, tribalism and superficiality by connecting indi- here concern informed consent, the vulnerability and viduals with similar views and thus discouraging individual risk with online identities, the public/private openness to differing views and the reflective and nature of communication, security and confidentiality. objective analysis of evidence and extended argument There is no single template for ethical research and (Manca and Ranieri, 2016). Facebook has also been Internet-mediated research is no exception; the context found to reinforce culturally embedded relations of and nature of the study will influence ethical considera- power distance, as more successful or able students, or tions. For example, covert observation is often frowned students wishing to manifest gratitude or respect, tend upon because it violates the principle of informed consent to benefit from exchanges with teachers whilst others do but it may be ethically acceptable and individuals’ not (Tananuraksakul, 2014; Manca and Ranieri, 2016). informed consent may not be necessary in online or other Many studies using self-report methods focus on pos- contexts where data exist within the public domain and itive implications for learner attitudes, engagement or where the risk of harm to users is low (Steven et al., attendance, whereas studies finding benefits for academic 2015). Other covert or ‘unobtrusive’ methods may rely knowledge, understanding or attainment are less on publicly available data, as one kind of unstructured common, and those finding negative effects tend to have observation of things that the researcher might not be used objective data (Tess, 2013). Opportunities exist for able to obtain or ask about directly. Such methods may more empirical studies of direct benefits for ‘hard’ out- be helpful and ethical where participants could find it dif- comes such as academic progression, knowledge, under- ficult to give authentic or honest answers, perhaps standing or attainment when comparing settings with and because these may be socially undesirable, or sensitive, without the use of social media integration (Ricoy and or where not employing unobtrusive approaches would Feliz, 2016). Other potential areas of study include encourage only ‘diplomatic’ responses. implicit and explicit institutional policies and traditional Unobtrusive methods may not engage participants, pedagogic and role expectations, which may be espe- or solicit comments from them and therefore use the cially interesting in cultural settings where the mainte- role of lurker or voyeur. Users may not have agreed to nance of social harmony is important. participate (and may not have been asked) and so ano- nymizing the data becomes especially important. 23.7 Ethics Examples of unobtrusive ways of collecting social media data may include: online discussion forums; How should we treat information posted online, such as Google Trends data (see www.google.com/trends); images from a mobile phone or information perhaps of Facebook and Twitter postings; YouTube videos; and a personal nature intended for family and friends but downloaded ‘chats’ from message rooms, community not for others? This could be important when individu- sites or user forums. However, such ‘found’ or als may not have a strong understanding of the possible ‘non-reactive’ data may easily become ‘reactive’ if long‑term ramifications of such posting, even though reinserted into the digital setting, such as, for example, the data are potentially discoverable and therefore when presented to participants to ask them what they already ‘public’. It is no excuse for the researcher to made of it. 463
Methodologies for educational research It may be ethically problematic to analyse found OO Fieldsites are rarely online or offline but are fluid, data in the absence of participant consent, unless inter- and researchers may need to engage with both; do pretations are depersonalized and carefully justified. not assume the existence of boundaries (Johnson et However, depersonalizing is no longer a strong guaran- al., 2007; Martin et al., 2010). tee of anonymity, as ‘[t]oday, the private man is a public entity, even a public display, that he controls OO Explore all stages and forms of engagement and use only partly’ (International Council on Human Rights a range of tools for recording and interpretation Policy, 2011) because (Broadribb et al., 2009). in an international and pervasive network (e.g. the OO Virtual worlds and social media sites have customs Internet) that is persistent in its records, and increas- and norms that mediate communication, but to their ingly searchable across indexes, databases and other users may be simply ‘the way things are’; adopting taxonomies, ultimately every interaction online has the the perspective of the stranger can help to under- potential to be traceable, either now or in the future. stand why this may be so or could be otherwise (Malinowski, 1922). (Henderson et al., 2013, p. 551) OO Expect a variety of different media and experiences One solution might be to adopt the principle of ‘non- across multiple platforms. alienation’ (Bakardjieva and Feenberg, 2000), where data may not be removed from someone’s control or OO Expect uncertainty: digital spaces often do not used for things they were not aware of without their provide a single, verifiable or ‘objective’ reality. explicit permission, which employs the principle of consent being ongoing, where participants are consulted OO Allow sufficient time: familiarity and expertise may at all stages of the research so as to have more control be needed to engage with the community and gain over data collection, analysis and reporting of research acceptance. (Henderson et al., 2013; Ramírez and Palu-a y, 2015). The notion of informed consent is therefore poten- OO Ensure anonymity of data, whether participants tially complex with all social digital media, even when request it or not; no one can know what may happen the risk to participants may be outweighed by the in future, and identifying an individual may also potential benefits to knowledge about the field of study identify their social networks once data and analysis (Pitcher, 2016). Nonetheless covert or unobtrusive data are in the public sphere (Henderson et al., 2013). collection is common (arguably necessary) in some dis- ciplines even when it ‘potentially poses a substantial OO When participating within a particular group, avoid threat to those who are involved or have been involved taking sides; avoid conflict. in it’ (Lee, 1993, p. 4), or when those studied may view the research as somehow undesirable (Van Meter, OO Remain alert to the way the researcher’s own agency 2000). This is because the research may be deemed to intrudes in the process of creating an authentic, rich have overriding benefits for the good of wider society and thick account of the ‘messy reality’ of the digital (e.g. when studying illegal activity). Chapter 8 of the cultural space as perceived and understood by par- present volume discusses further a range of ethical ticipants (Law, 2004). issues in Internet research. OO Be aware that study participants may have access to 23.8 Guidelines for practice the researcher’s own online identity; consider whether and how this may influence the study and A range of issues have to be considered when conduct- plan accordingly. ing research using virtual worlds and social networks (Moschini, 2010; Hine, 2015) and each may be more or OO Ideally all forms of misrepresentation or deception less prominent depending on the setting and nature of are best avoided; they are antithetical to openness the research: and trust. Covert observation is a form of deception that cannot be guaranteed to be free of potential OO The holistic approach of netography means remain- harm because we do not know what may be ing alert for unanticipated acts of meaning-making observed. However, some kinds of research may be and for how activities make sense to the individuals less reliable or even impossible without some form engaged in them (Lewis and Allan, 2005). of deception, and the balance and degree of any potential harm/benefit requires careful consideration and justification throughout (Steven et al., 2015). In addition to the usual decisions to be made when designing a research project, using a virtual world or social network will require it to be set up and managed and may need someone with technical expertise to help decide whether to make use of an existing commercial product or to create a purpose-built environment. The 464
Virtual worlds, social network software and netography former is more straightforward but the options available apply different procurement constraints, use dissimilar for customization may be more limited, so it is important and sometimes incompatible IT infrastructures, or have to be clear in advance which features are required for the policies with very different embedded institutional and study, what data need to be collected, and how this will cultural assumptions about the nature of academic roles be done. A customized platform may be the ideal, partic- and responsibilities (e.g. for academics and instructors ularly in the case of a virtual world, but the resource or for students and research participants). When using implications may be significant and, in the interests of virtual worlds, having computers with appropriate viability, it may instead be worth adjusting the research specifications will also be important (check the soft- methodology to allow the use of a commercial product. ware’s website), along with sufficient bandwidth and A prerequisite for research in any online environ- safe passage through institutional network firewalls, so ment is sufficiently high-quality online availability, and discussions with institutional IT managers are impor- a prudent researcher will check this and software/ tant when planning a study. system requirements at an early stage. The researcher Whilst participant enthusiasm is common, it should must also fully understand the protocols, etiquette and not be assumed and some studies have found to their common practices of the fieldsite(s) sufficiently for cost that participants who are initially enthusiastic find them to engage and blend in with the community they that time constraints, the relative complexity of virtual wish to study. This may require technical and personal worlds and bandwidth demands (which can create oper- preparation before beginning the study, and the time ational slowness) can prove a disincentive for continu- and energy that will need to be dedicated to such ing engagement (see Jarmon et al., 2009). Despite the ‘acculturation’ should not be underestimated. visual and conceptual allure of virtual worlds and the As with all contexts for research, enquiry into digital popularity of social media, successful participant artefacts such as virtual worlds, digital social networks, recruitment may require careful preparation (Fetscherin online forums, blogs or wikis should be driven prima- and Lattemann, 2007). Consider who might facilitate rily by research questions, not by decisions about which recruitment beyond known contacts, to include local methodology to adopt, and no particular mode of study authorities, professional organizations, universities or should be automatically privileged above others. employers. It may be useful to set up a website to Instead, what matters is a clear and careful link between explain the research and provide information and docu- the research question and the methodological design of mentation, and this will also facilitate engaging with, the research. This is often an iterative process and an and recruiting, potential participants. effective researcher will always be open to the possibil- ity of adjusting (or even sometimes discarding) his/her 23.9 Data original research question as exploration proceeds and new insights appear. Exploration and insights offer val- Because virtual world and social media fieldsites may uable opportunities to redirect the research to exploit be large, diffuse and engage with different media, it can them, and research should always respond to the pres- be hard to establish the target population or get a sense sure of evidence (Malinoswki, 1922). of how representative a given sample may be (Hine, By their nature, virtual worlds and social networks 2015). In many virtual environments, events and lend themselves to projects which do not require partici- objects may also be connected to other online techno pants or researchers to be physically located near each logies such as blogs, wikis, questionnaires, rating other. However, much existing research with these tech- systems, databases, etc. and increasingly to in-w orld nologies focuses on single groups of educators or stu- tools. In such circumstances, careful planning of data- dents or members of a specific university, so a project collection strategies is essential. with more breadth than this would have increased value. Communication in virtual worlds via text, chat, voice Collaborative affordances may then become important if and signing can all be captured for later analysis via using a research team, and researcher collaboration can recording and transcription and, because data can be be facilitated by technologies such as Google docs or time-s tamped, it is possible to compare the outcomes Dropbox to share documents, spreadsheets and data- from analysis from these different communication chan- bases, and work on them to develop understandings or nels for data triangulation (Martin and Vallance, 2008). dynamic trend analysis using notes, ‘mind-maps’, By providing a scaffolded vocabulary, researchers may graphs, charts or tools such as Gapminder. also more easily collect data from individuals with Challenges can emerge when creating common pro- communication difficulties, where the slower pace of tocols for a research team, especially if individuals are interaction and reduced amount of data involved in com- based in different institutions or countries, as these may munication in these environments (such as lack of subtle 465
Methodologies for educational research facial or body-language communication) may be advan- collected. Leaving this task until later may be unwise, tageous (Ravenscroft and McAllister, 2006). These data as it may then be difficult to remember which data goes can be converted to numbers for quantitative analysis or where and to what other data it is related. can be used to develop a richer understanding through Before choosing software to be used for recording interpretative, phenomenological analysis, which is a data it is important to consider: useful form of qualitative analysis when we are inter- ested in describing how people negotiate, understand and OO whether it will meet the research needs; make sense of the world. Both quantitative and qualita- OO whether it can be supported by and run effectively tive data need to be used in light of the perspective adopted, the kind of information collected and the on the computer system to be used (e.g. PC, Mac, assumptions and objectives of the research. smartphone); Field notes are an important element of data collection OO whether enough storage is available, and how long in netography but they should be more than just descrip- recordings will usually be and how many of these tive accounts; they can be early interpretations of what there might be. How much detail is important: do happens, notes about feelings, ideas about what these video recordings need to be full-screen or high- might mean, why things happened or did not happen, ‘to definition? These considerations will affect the size do’ lists, as well as notes of frustrations or puzzles. They of recorded files, so using a large-capacity external might include sketches to suggest meaning, proximity or drive may be necessary; relationships, or be loosely organized reflections and OO whether advanced editing features will be needed or ideas for later use and reflection. The researcher may just a basic package; wish also to blog or tweet about the research or post OO testing all software thoroughly with the system on things on Facebook or Instagram to encourage further which it will be used before adopting or purchasing interaction and debate, although the consequences may it, and running extensive tests before the research be some potentially awkward decisions about what to begins, to minimize the risk of technical problems. reveal and concerns about ‘over-sharing’ that may pre- empt later and more thoughtful analysis (Boellstorff, Visual representations such as screenshots can be 2008; Boellstorff et al., 2012; Hine, 2015). useful when combined with other data, either to illus- Screen shots, field notes and other data should be trate an observation in the field notes or to remind the numbered, dated, time-stamped and written up immedi- researcher of some important event or pattern of behav- ately after an intervention. Making notes with a word iour. Using visual records from online environments, processor can make it easier to search for keywords whether single images or video, requires careful ethical later, and audio or video recording of sessions can be consideration and participants need to be aware that useful for future analysis and allow the researcher to such records will be collected and to have given prior concentrate on making observations during sessions. and informed consent. Individuals must also not be Virtual worlds and social media allow researchers to identifiable from visual data; care must be taken to ano- collect more participant data than are otherwise possi- nymize any that are used by removing identifying ele- ble, by capturing text, chat and user presentations from ments (see also Chapter 31 of the present volume). the in-w orld environment and by recording activity. It Visual data are not always available, as some online is important to establish whether the chosen online environments can be entirely composed of text, or have setting will automatically capture data from features only text chat as their means of communication. For like chatlogs and, if so, this should be tested to make many others, voice chat is increasingly an inbuilt sure the date and time are always recorded. feature, but where not it is common for online commu- It may not be immediately clear what valuable nities to employ additional software, effectively a information is within such data, as it will often contain phone service over the Internet known as Voice over intermingled issues, but it cannot be collected Internet Protocol (VoIP). The researcher may therefore retrospectively, so capturing it as work progresses is need to make enquiries within the community to be important. Each specific event should be recorded with studied to determine whether VoIP is used, and then a unique filename so that data can be correctly adopt it to augment the research data by capturing sequenced and matched together later, as it might be important communications, ambient sounds or music. extremely difficult to do this retrospectively. Collecting Participants in an online environment may also make data from online environments is relatively easy and use of associated separate communication media, but the can quickly generate large amounts of material, so all individuals in any of these other communications may relevant information needs to be kept together as it is not necessarily be frequent, or the same users as found in the main study site, and careful notes will help with 466
Virtual worlds, social network software and netography mapping these different audiences and sub-groups to to rethink, refine and improve existing pedagogy and ensure clarity and avoid unhelpful confusion during research methodology and instrumentation. analysis. As is the case for all data, keeping notes of The companion website to the book provides filenames and associated other data will be essential, so worked examples of virtual world, social media and as to know which participants were present, the time, the netography research, together with website references event(s) and location and any other relevant details. for the topics addressed in this chapter, virtual world Finally, offline gatherings such as refreshment places, research methods and PowerPoint slides for this conventions and workshops are sometimes associated chapter, which list the structure of the chapter and then with, and used by, particular online communities and can provide a summary of the key points in each of its sec- be valuable sites for gathering additional data and tions. This resource can be found online at: www. meeting known or new members. routledge.com/cw/cohen. 23.10 Conclusion Notes This chapter has argued that a range of emergent, con- 1 The term ‘virtual world’, as used here, presumes the pres- tentious or sensitive topics can be usefully explored ence of human users, although strictly speaking even when using virtual environments and networked social media empty of participants such an environment may remain a and that these present both opportunities and challenges. virtual world. Part of the reason for the ongoing debate Virtual worlds and increasingly other social media can about what is and what is not a virtual world, and whether offer high levels of interaction and embodiment and these things should be given a different name, is that the heightened immersion via technologies such as Oculus use of virtual world features is becoming more common in Rift, PlayStation VR or HTC Vive. This combination of the technologies of the World Wide Web, in which one technologies offers rich communication affordances and increasingly sees visually realistic three-d imensional some unique opportunities for data capture,2 including virtual environments on websites, discussion forums, the ability to record sound, chat and text as well as con- blogs, chat rooms and social network sites where user temporaneous images and video in real time. involvement is mediated by avatars. In such research settings there may be a need to scru- tinize what is understood by ethnographic or qualitative 2 Many commercial companies invest a great deal of time approaches such as participant observation, focus groups and money acquiring and analysing data from social and interviewing. The traditional assumptions that are media. These companies often use expensive specialist made when deploying methodological tools may need to companies to do this work but are sometimes prepared to be revisited to verify whether they are still applicable in offer substantially lower prices to students or academic these digitally blended communities, in which alternative institutions, so check to see if your institution has or can values and different articulations of reality are to be secure this. There are also some relatively straightforward found, where participants may alter their displayed methods of collecting social media data that can be used embodiment at will, and in which we may simultane- by researchers for free or at minimal cost. ously conduct our research with individuals in different Twitter: Entrepreneur (www.entrepreneur.com/article/ locations around the world. By exposing the researcher 242830); The Chorus Project (http://chorusanalytics.co. and practitioner to new constructions, expressions and uk) (see also Tweetcatcher). transformations of identity, reality and community, Facebook: Graph (www.facebook.com/graphsearcher); virtual worlds and other social network software offer Statista (www.statista.com/statistics/264810/number-o f- many unique opportunities to re-e xamine the nature of monthly-a ctive-facebook-u sers-worldwide) (check to see community and self across the virtual/physical world and if your institution provides access). VennMaker (a mapping tool for collecting and analys- ing data in social network analysis) (www.vennmaker. com/?lang=en). Some straightforward guidance is also available on obtaining data from most social media software at WikiHow (www.wikihow.com/Main-Page). Companion Website The companion website to the book includes PowerPoint slides for this chapter, which list the structure of the chapter and then provide a summary of the key points in each of its sections. These resources can be found online at www.routledge.com/cw/cohen. 467
Search
Read the Text Version
- 1
- 2
- 3
- 4
- 5
- 6
- 7
- 8
- 9
- 10
- 11
- 12
- 13
- 14
- 15
- 16
- 17
- 18
- 19
- 20
- 21
- 22
- 23
- 24
- 25
- 26
- 27
- 28
- 29
- 30
- 31
- 32
- 33
- 34
- 35
- 36
- 37
- 38
- 39
- 40
- 41
- 42
- 43
- 44
- 45
- 46
- 47
- 48
- 49
- 50
- 51
- 52
- 53
- 54
- 55
- 56
- 57
- 58
- 59
- 60
- 61
- 62
- 63
- 64
- 65
- 66
- 67
- 68
- 69
- 70
- 71
- 72
- 73
- 74
- 75
- 76
- 77
- 78
- 79
- 80
- 81
- 82
- 83
- 84
- 85
- 86
- 87
- 88
- 89
- 90
- 91
- 92
- 93
- 94
- 95
- 96
- 97
- 98
- 99
- 100
- 101
- 102
- 103
- 104
- 105
- 106
- 107
- 108
- 109
- 110
- 111
- 112
- 113
- 114
- 115
- 116
- 117
- 118
- 119
- 120
- 121
- 122
- 123
- 124
- 125
- 126
- 127
- 128
- 129
- 130
- 131
- 132
- 133
- 134
- 135
- 136
- 137
- 138
- 139
- 140
- 141
- 142
- 143
- 144
- 145
- 146
- 147
- 148
- 149
- 150
- 151
- 152
- 153
- 154
- 155
- 156
- 157
- 158
- 159
- 160
- 161
- 162
- 163
- 164
- 165
- 166
- 167
- 168
- 169
- 170
- 171
- 172
- 173
- 174
- 175
- 176
- 177
- 178
- 179
- 180
- 181
- 182
- 183
- 184
- 185
- 186
- 187
- 188
- 189
- 190
- 191
- 192
- 193
- 194
- 195
- 196
- 197
- 198
- 199
- 200
- 201
- 202
- 203
- 204
- 205
- 206
- 207
- 208
- 209
- 210
- 211
- 212
- 213
- 214
- 215
- 216
- 217
- 218
- 219
- 220
- 221
- 222
- 223
- 224
- 225
- 226
- 227
- 228
- 229
- 230
- 231
- 232
- 233
- 234
- 235
- 236
- 237
- 238
- 239
- 240
- 241
- 242
- 243
- 244
- 245
- 246
- 247
- 248
- 249
- 250
- 251
- 252
- 253
- 254
- 255
- 256
- 257
- 258
- 259
- 260
- 261
- 262
- 263
- 264
- 265
- 266
- 267
- 268
- 269
- 270
- 271
- 272
- 273
- 274
- 275
- 276
- 277
- 278
- 279
- 280
- 281
- 282
- 283
- 284
- 285
- 286
- 287
- 288
- 289
- 290
- 291
- 292
- 293
- 294
- 295
- 296
- 297
- 298
- 299
- 300
- 301
- 302
- 303
- 304
- 305
- 306
- 307
- 308
- 309
- 310
- 311
- 312
- 313
- 314
- 315
- 316
- 317
- 318
- 319
- 320
- 321
- 322
- 323
- 324
- 325
- 326
- 327
- 328
- 329
- 330
- 331
- 332
- 333
- 334
- 335
- 336
- 337
- 338
- 339
- 340
- 341
- 342
- 343
- 344
- 345
- 346
- 347
- 348
- 349
- 350
- 351
- 352
- 353
- 354
- 355
- 356
- 357
- 358
- 359
- 360
- 361
- 362
- 363
- 364
- 365
- 366
- 367
- 368
- 369
- 370
- 371
- 372
- 373
- 374
- 375
- 376
- 377
- 378
- 379
- 380
- 381
- 382
- 383
- 384
- 385
- 386
- 387
- 388
- 389
- 390
- 391
- 392
- 393
- 394
- 395
- 396
- 397
- 398
- 399
- 400
- 401
- 402
- 403
- 404
- 405
- 406
- 407
- 408
- 409
- 410
- 411
- 412
- 413
- 414
- 415
- 416
- 417
- 418
- 419
- 420
- 421
- 422
- 423
- 424
- 425
- 426
- 427
- 428
- 429
- 430
- 431
- 432
- 433
- 434
- 435
- 436
- 437
- 438
- 439
- 440
- 441
- 442
- 443
- 444
- 445
- 446
- 447
- 448
- 449
- 450
- 451
- 452
- 453
- 454
- 455
- 456
- 457
- 458
- 459
- 460
- 461
- 462
- 463
- 464
- 465
- 466
- 467
- 468
- 469
- 470
- 471
- 472
- 473
- 474
- 475
- 476
- 477
- 478
- 479
- 480
- 481
- 482
- 483
- 484
- 485
- 486
- 487
- 488
- 489
- 490
- 491
- 492
- 493
- 494
- 495
- 496
- 497
- 1 - 50
- 51 - 100
- 101 - 150
- 151 - 200
- 201 - 250
- 251 - 300
- 301 - 350
- 351 - 400
- 401 - 450
- 451 - 497
Pages:
