Methodologies for educational research transcription, to the extent that a ‘radical critique’ of Despite these challenges, this chapter has argued interviews would reject them out of hand (Hammersley, that qualitative research in its many forms is a very val- 2013, pp. 69–72), for example, for being unreliable and uable approach to educational enquiry. invalid. Useful websites for those commencing qualitative Hammersley (2006) also notes that the ‘radical cri- research are: tique’ of interviews raises questions of how far what is said in an interview really represents what is happening http://nsuworks.nova.edu/tqr (which gives the websites outside the interview (p. 9). Interviews, he contends, of several hundred other sites providing materials on are their own context, and they shape what is said or qualitative research); not said (p. 9). Further, Atkinson and Delamont (2006) argue that short-stay, ‘quick-fix’ activities like inter- www.ukdataservice.ac.uk (the UK’s Data Service, views risk betraying the complexity of the social situa- which includes qualitative data); tion that qualitative research seeks to portray and understand, which can only be achieved by sustained www.data-archive.ac.uk (the UK Data Archive); research in the field. http://gsociology.icaap.org/methods/qual.htm (a source for accessing other websites for online materials and support). Companion Website The companion website to the book includes PowerPoint slides for this chapter, which list the structure of the chapter and then provide a summary of the key points in each of its sections. These resources can be found online at www.routledge.com/cw/cohen. 322
Historical and CHAPTER 16 documentary research Jane Martin 16.1 Introduction 16.2 Some preliminary considerations: theory and method The focus of this chapter is on research using historical and documentary evidence. Its aim is to convey the American historian Bonnie G. Smith looked at what value of historical study, to consider the resources that happened when the practice of scientific history took may be available for the researcher and how relevant root in nineteenth-c entury universities in Western sources should be handled. The chapter goes beyond Europe. In The Gender of History (1998) she argued command of institutionalized, archival sources to con- that Enlightenment thinking influenced the making of a sider the sheer diversity of sources and the ways these professional discipline in which empirically minded are affected by ideas of progress and loss. Debate and men defined themselves and their intellectual products discussion will include: recent empirical trends and in opposition to an older, more popular amateur history methodological arguments concerning the diverse read for moral instruction and entertainment, which material from which history is made; the availability they deemed trivial. In such a context, history was no and representativeness of ‘historical data’; and the rela- longer regarded as a branch of literature and a tradition tionship between theory and ‘facts’. In turn, the chapter of women’s scholarship (as authors of textbooks, biog- points out some of the problems involved in doing his- raphies or memoirs, translators and editors of original torical and documentary research in educational set- documents) disappeared from view. Consequently, tings. We finish by looking at a detailed case study of when we imagine a great historian we automatically research that has made use of archival materials in an think of a man, we accept as natural titles like Malcolm account that takes the figure of a woman educator activ- Bradbury’s The History Man, published in 1975, ist as its central focus. The discussion will focus prima- because professionalization and historical science rily on methodological approaches and strategies and developed at a time of separate spheres, when it was their influence on the final outcome of the study. assumed a woman’s place was in the home. Thus The aspiration to discover what happened in the past the making of a professional discourse involved Other- and what it was like to live in the past is traditional to ing the scholarship, style and preferences of those the nature of history. The challenge of accessing the without the ideological means to achieve disciplinary voice of the past through time and space is something ascendancy. that excites historians. ‘The past is a foreign country’, Foundational claims over scientific, empirical sighs old Leo as he looks backs on his childhood with history were intrinsic to the formation of this new, nostalgia in L. P. Hartley’s influential novel The Go- university-led discipline constructed upon a theory of Between, adapted as a film in 2015: ‘they do things dif- knowledge that had its origins in the belief that knowl- ferently there’. Historians make history through the edge derived from observation of the material world, production of knowledge, explanations and interpreta- the core tenets being: rigorous examination and knowl- tions of what has gone before. In discussing questions edge of historical evidence, verified by references; of sources and interpretations I shall endeavour to pass impartial research, devoid of a priori beliefs and preju- on the pleasures of working as an historian. It will be dices; an inductive method of reasoning, arising out of argued that the uses and limitations of historical sources the sources themselves and moving from the particular can only be fully appreciated when they are understood to the universal (see also Chapter 1 of the present in their social context as historical products. There is volume). By 1900 the assumptions about historical no straightforward sense in which history simply practice implicit within the study of history encouraged ‘speaks for itself ’. A very brief history of the study of an approach consisting of the collecting and reading of history since the mid-nineteenth century will help the papers which official authorities drew up for the show why. purposes of the conduct of their affairs, or which they 323
Methodologies for educational research used in conducting them. To be steeped in primary or The ‘take-o ff ’ of cultural history from the late 1980s original sources, which were generated at the time of onwards ruled out a fundamentally positivistic concern the event under consideration, helped lend intellectual with getting at the truth, giving readers ‘the facts’. authority among the historical profession. Scholarly While I shall not explore the minutiae of these reputations were built on the principles of the search approaches here, a post-s tructural approach to history for objective truth: the production of irrefutable, factual entails remaking ourselves as readers and writers, information located at the heart of the historical enquiry giving us new methodological tools with which to (Tosh, 2008, pp. 2–5). approach the task of assessing and interpreting sources. Within this approach ‘the facts’ speak for them- Here meaning is looked for in a culture’s language and selves independently of a particular point of view. In systems of representation. Perception of empirical New Perspectives on Historical Writing, British cul- reality was constituted through multiple refracting per- tural historian Peter Burke (2001, pp. 3–6) stressed that spectives: one that is constantly changing, subject to the range of documents considered by traditional variation over time as well as in space. Thus, rather empiricist historians tended to be remarkably narrow. than a historical practice based on straightforward read- In the 1960s and 1970s many aspects of this account ings of state papers and official data, the same material were energetically challenged (e.g. see Thompson, artefacts or texts may be used, but read against the 1963; Hobsbawm, 1964; Rudé, 1964; Jones, 1971; grain, looking for contested meanings and omissions. Samuel, 1975). Social history, sometimes described as The linguistic turn opened up new ways of interpreting the ‘history of the people’ or ‘history from below’, texts, which may contain different rhetorical strategies emerged as an alternative to conventional political and voices as opposed to being written as supposedly history, both in terms of its objects of interest and its objective, and especially the relationship between them belief in deep-rooted economic and social factors as – ‘intertextuality’. agents of historical change. ‘History from below’ as For historians, many post-s tructuralist topics and practised by social historians used the language of methods are a legacy of the work of Michel Foucault. class, including a Marxian approach to society that Foucault studied what he termed ‘the history of systems could be applied to a wide range of historical cases, as of thought’ (e.g. Foucault, 1970). He argued that docu- indeed Marx applied them. Revisionist studies in the ments are not of interest because of what they tell us history of education tried to question the Whiggish about the author, but because they inform us about the assumption that the course of historical development mechanisms through which power is exercised. That is, was an unbroken chain of ascent from a benighted past documents are a medium through which power is to an enlightened present. In their introduction to Chil- expressed. In adopting an interpretivist or discourse dren, School and Society, published in 1981, Anne analysis approach, one would believe that the critical Digby and Peter Searby expressed their dissatisfaction analysis of a document involves questioning why the with a view of education from Westminster or White- document was produced, what is being said (overtly hall rather than the home or the schoolroom and their and covertly) and what is not being said (see also aspiration to provide a corrective to the dreary sequence Chapter 3 of the present volume). Post-s tructuralism of institutional growth that characterized former treats texts of all kinds as systems of signification (col- narrow, excessively bureaucratic histories of education. lections of signs which conform to some internal A broadening of attention to other documentary system), whose meanings can be ascertained in part by sources was an important feature of the broad body of deconstruction, acknowledging that meaning can be social history, allied with a willingness to supplement self-referential and not entirely taken from the context documents with other sources of evidence. For in which it was produced or from authorial intent. example, the revival of oral history derived from a new The collection and presentation of historical mate- generation of historians steeped in the politics of the rial is inevitably selective, and, to some extent at least, New Left, civil rights and feminism (e.g. Rowbotham, this selection and interpretation relies on the questions 1975; Thompson, 1978). Recovering lost voices asked of the material, or the theoretical perspective provided a means to empower women, the working- which is brought to bear on any particular piece of class and minority communities, allowing them to research. No general consensual version of history is speak for themselves, thereby contesting the national possible and monolithic accounts are unlikely to be consensus, enlarging the explanatory concepts available either adequate or satisfactory. Different groups do to historians, generating new perspectives and radical have different interests, experiences and cultural forms, critiques of education, youth and structural racism and and do provide alternative definitions and accounts of sexism. these. Much of the historian’s skill lies in the creative 324
Historical and documentary research and self-a ware use of the sources from which history is an official document did not believe what he or she made. Let us now turn to the task of assessing and recorded, while personal documents may be produced interpreting those raw materials. It is important to for a host of reasons, depending on the mood of the appreciate the richness and the limitations of each type. moment. Third, we need to ask whether the document is rep- 16.3 The requirements and process resentative. Are we looking at a unique view or does it of documentary analysis represent a ‘general mood of the time’? Assessing the typicality, or otherwise, of evidence centres on the two When discussing documentary sources it is conven- aspects of ‘survival’ and ‘availability’. Not every docu- tional to differentiate between ‘primary’ and ‘second- ment will make its way into an archive: documents ary’. The former encompass every kind of evidence have differential survival rates and those which do which people have left of their past activities, produced survive do not always provide all the information during the period being studied. The latter discuss the required. The answers to a great many questions are period studied but are created sometime after it or in simply not available, since the necessary records either some way removed from the actual events that are the never existed or failed to survive. With respect to the focus. Simply put, primary analysis is an interpretation UK national archive, selection of public records takes of raw materials, whereas secondary analysis involves place in two stages. At the outset, records which are an examination of the interpretations of others. Obvi- considered worthless are destroyed, and those which ously, it is possible for some sources to act as both have been identified as valuable for future administra- primary and secondary sources, depending upon the tive need or future research are kept for further review exact context of the information we are interested in. when the record is fifteen to twenty-five years old. The We can perceive this in my study of the ‘career’ UK’s Freedom of Information Act (2000) governs chances for twentieth-century women historians, which access to information held by most public authorities, employed writings conventionally designated second- with two forms of exemption: ‘absolute’ and ‘quali- ary sources, as primary sources. Thus, whereas Eileen fied’. In the case of the latter, a public interest test must Power’s Medieval Women is a secondary text for a his- be made, balancing the public interest in maintaining torian of the period, for me, as an example of Power’s the exemption against the public interest in disclosing oeuvre, it became a primary source (Martin, J., 2014). the information. A classic example is data on school When we decide to use historical documents, their exclusion, which would be subject to a 100-year rule validity and reliability must always be held up for scru- because disclosure could cause distress to living tiny. Scott (1990) identifies four potential challenges. individuals. To start we have to consider the issue of authenticity; Finally, we need to pay attention to the meaning of who a document was written for and by (authorship); the document. There are three aspects to this. These whether it constitutes a first-hand, second-hand or even are: the intended content of a text, the received content more remote account; whether confidential or not, of a text and the internal meaning of a text. Acknow public or private, forced or voluntary and so on. At a ledging the complications of text comprehension, Scott simple descriptive level, whether the document is (1990) describes the process of understanding a text ‘sound’ or authentic may be challenged on several hermeneutically according to his four criteria (authen- grounds – if it contains many errors, is one of many ticity, credibility, typicality and meaning). To some versions, is inconsistent in relation to other similar doc- extent a heuristic tool, the hermeneutic circle requires uments and in terms of ‘ownership’. It is not always the closing the loop whereby knowledge and understanding case that the identity of the author is apparent. of the text as a whole is achieved through ‘dialogue’ Credibility is a second potential challenge. In other within boundaries set by the frames of reference of the words, is the document we are analysing reliable? Is it researcher and those who produced the text. undistorted, ‘sincere’ and ‘accurate’? For a document to be credible, we need to be aware of the purpose of 16.4 Some problems surrounding the document. Was it produced to describe events, to the use of documentary sources persuade (such as a school brochure) or to self-p rotect (such as a ministerial memorandum)? The purpose of a Documents are selective in terms of the information communication, then, can provide an important context presented. Some documents are produced with research for understanding its content. This is not to say that you in mind (most institutions and some individuals have should be suspicious of every document you encounter. deliberately sought, with an eye on the future, to gener- However, there is always a possibility that the author of ate accounts of their activities); others are produced for 325
Methodologies for educational research personal use and are less self-conscious. Either way, cumstances as being more generally applicable. It the act of recording will also be informed by the social, means we cannot explain change over time, or under- cultural, economic and political landscape of which stand apparent continuities and apparent breaks in they are a part. activity. Alternately, one could adopt a more random It is important to ask who created the source and approach, for example, the selection of, say, one news- how it was understood by contemporaries. When politi- paper a month, or more systematic sampling (see cal texts are employed, intellectual historian Quentin Chapter 12 of the present volume) of every tenth news- Skinner (2002) takes this process a step further. First, paper. An obvious drawback to this method is the diffi- by insisting that the works of political thinkers be culties it presents in following up stories. The best understood within the context out of which their works solution to the problem may be the use of a variety of were produced, and second, that they be understood as sources simultaneously so that an overall pattern begins acts of rhetorical communication – to consider the to emerge which suggests directions for future research. intentions of the author and how they were received. There can be no ‘formula’ for decisions of this kind. A A less obvious problem is that those who work with ‘feel’ for the period, the relevant questions and the historical documentary evidence can have both too little sources is ultimately what guides the methodology of and too much available to them. You might never find any historical research project, and this can only prop- exactly what you need; on the other hand, where, in the erly be gained as part of the research process itself. large amount of twentieth-c entury material for example, Ideally it may be that the research should be considered do you start? In reality, our understanding of educa- finished when all the classes of document relevant to it tional history is informed by a selective reading of doc- have been exhausted, but this is rarely a practical prop- uments. It is highly unlikely, particularly for osition. There is a sense, inevitably, in which the undergraduate students, that you will have had the time research is over whenever it is time to stop. or opportunity to read everything about a person (for Quality appraisal is a never-e nding process. This example, an education minister) or an event (for does not mean that only error-free, typical documents example, the imposition of a legally enforced national (from written to oral) which have a common interpreta- curriculum). Similarly, the types of document we read tion can be used. What it does mean is that we must be vary in terms of information and accuracy: some are at least aware of any challenges to reliability that may very opinionated or subjective whilst others may be exist, or that others may interpret the information held highly factual and descriptive. within differently. The categorization of historical documents largely follows the dominant definitions of a particular period 16.5 The voice of the past: whose and if we are to read the silences we need also to look account counts? at material which gives little hint that evidence we need may be lurking inside it. There are several conse- A move away from structuralism in all the social sci- quences of this approach. First of all, for much of the ences is important in accounting for the increasing time one is working in the dark. Decisions about what numbers of historians who have turned to biography in to look at and what to ignore may be a hit-a nd-miss the last thirty years. In a widely read textbook on the affair. Second, this kind of research is time-consuming, study of history, Ludmilla Jordanova (2000, p. 41) frustrating, often unrewarding and frequently leads to a likens biography to ‘holistic history’. Barbara Finkel- feeling of wasted time and effort. Third, the amount of stein (1998), Diana Jones (1998) and Peter Figueroa material which historical researchers often have to (1998) all indicate clearly the considerable strength of handle makes some kind of selective reading or sam- the biographical approach for an understanding, and pling, whether deliberate or otherwise, inevitable. But bringing to life, of the history of education. Educational on what basis should choices be made? The collection biography offers a frame of reference within which to and presentation of raw materials is inevitably selec- assess the relative power of material and ideological tive, and to some extent at least this selection and inter- circumstances, the meaning of policy and practice, pretation relies on the questions asked of the material, the utility of formal and informal schooling and or the theoretical perspective which is brought to bear the relationship between learning and teaching. on any particular piece of research. Researchers utilize any form of writing that includes a One method of selection is to focus on ‘significant’ construction of the self (diaries, memoirs, letters, auto- events or periods, guided by information from second- biography and biography, travel writing), oral testi- ary sources. One drawback of this approach is the mony, photographs and material objects (see also possible risk of interpreting a set of extraordinary cir- Plummer, 2001). 326
Historical and documentary research Kathleen Weiler’s (1998) study of women teachers and presuppositions of moral citizenship. But the notion in two rural California counties from 1850 to 1950 of ‘teacher resilience’, whereby a set of values under- relies heavily on oral testimony. Texts and testimonies stood as a structure of feeling – following Raymond explore the social contexts of teaching, employed to Williams (1961) – emerged as something that endures understand what teaching meant to women teachers, across generations and this was partly explained by the what it provided them and how it shaped their catego- importance of ‘emotional labour’ in teaching (Kirk and ries of experience. Building on the work of gender his- Wall, 2011). torian Joan Scott (1986) and influenced by Foucault, Another example of what historical and archival three key concepts inform Weiler’s approach – knowl- resources have to offer educational researchers was edge, language and subjectivity. Used to convey the provided in 2001 by Jonathan Rose, whose Intellectual constructed quality of memory and experience, subjec- Life of the British Working Classes was one of the first tivity includes the struggle and contest over identity, examine the reading practices of past generations. the process of identification and an unstable, shifting There is some evidence for select members of the well- subject constructed both through dominant conceptions connected, articulate, document-p reserving classes, but and resistance to those conceptions. what of the little recorded majority? Rose employs Peter Cunningham and Peter Gardner (2004) also various kinds of autobiography and memoir written by used interviews as a source to help reconstruct what those from working-class or other modest backgrounds, being a student teacher meant to various groups in people who had usually received very little formal early-twentieth-century England. They wanted to write schooling, at least until the middle years of the twenti- a different kind of history concerned with the day-to- eth century. He also makes good use of library records, day experience of ordinary teachers as opposed to the educational archives, oral histories and Mass Observa- administration of education. For them this was not tion, and early social surveys to create a detailed history solely a matter of restoring ‘lost voices’ or interlocu- of working-class reading. tors; it was also about recording and interpreting events Using image as historical evidence is a relatively previously excluded both for contemporaries and for new addition in the history of education pioneered in future generations. It was about the creation of a more Silences and Images (1999), edited by Ian Grosvenor, accurate historical record and therefore a more Martin Lawn and Kate Rousmaniere. Taking represen- ‘useable’ past that includes possibilities we might not tations of teachers as an example, António Nóvoa even have considered because the record of the road (2000) analyses his personal archive of over 600 not followed was less likely to survive (possibly images from published sources that spans two centuries destroyed by people with different priorities). Opening and a range of continents. Nóvoa notes a consistency of themselves to this recognition produced a history- themes equating to what he terms a ‘grammar of writing that is wider in scope and does not just reflect schooling’, consisting of secondary teachers undertak- the standpoint of authority. ing pupil assessment and male primary teachers repre- Oral testimonies also formed the core of a project sented as disciplining figures. In so doing, he combines exploring work and identity in three main occupational quantitative analysis and iconography (understood to sectors, including teachers, in twentieth-century mean a range of, or system of, types of image used to England. Overall the team conducted forty interviews convey particular meanings) used as a way of ‘reading’ with three generations of men and women teachers: visual sources (see also Chapter 36 of the present retired, mid/late working life and younger teachers or volume). people who were new entrants, with interviews taking Christine Wall (2008) drew on careers literature place in London and the south-east and the north-west available in University College London’s Institute of of England. Many of the interviews with older people Education library on open access and material pub- were conducted in people’s homes, allowing, in these lished by the largest teaching union, the National Union cases, for a more reflexive account of working life, of Teachers (NUT), held in the Trades Union Congress whereas those spoken to in the workplace gave shorter (TUC) Library Collections at London Metropolitan answers to the interviewers’ prompts: the pressure of a University, to study the formation of gendered teacher working day pre-empting any long conversations, but identities between 1940 and 2000 in Britain. In so not precluding articulate insights into the narration of doing, Wall paid particular attention to visual represen- the ‘teacherly self ’. Whilst they all talked about tations of teachers on the front covers of the house ‘making a difference’, ‘postmodern’ teachers expressed journal of the NUT, published as The Schoolmaster and themselves in terms of a kind of entrepreneurial culture, Woman Teachers’ Journal until 1962, when it became whereas earlier generations articulated the vocabulary The Teacher. Her analysis involved, first, a quantitative 327
Methodologies for educational research ‘sorting’ of images and from this a closer reading of Labourers because, she said, she was a gas worker on selected images employed to depict the occupation of the platform and a general labourer at home. How influ- teaching ensued. The smaller number of images was ential was she? selected on the basis of a particular iconography: a set By 1900, Mary was well-known as a participant of noticeably recurring, thematic compositions conflat- within the broader labour movement and as a cam- ing the role of woman teacher with motherhood. One paigner for improvements in working-c lass education. series of compositional similarities that stood out, for During the First World War, she was in close touch example, was the standard Christian iconography of the with the European anti-w ar movement and threw Madonna and Child. herself into Russian émigré politics. Guiding cam- paigns in defence of the right of asylum, she had a 16.6 A worked example: a range of contacts among suffragettes, trade unionists biographical approach to the history and international socialists. She urged people to fight of education the abandonment of industrial rights and guarantees, such as the right to strike and restrictions on the use of This section reports a worked example of my own prac- child labour, to back the unofficial rank-a nd-file indus- tice as a researcher in the history of education. It indi- trial movement on Clydeside and the educational work cates several key points in planning and ‘doing’ such of the Scottish Marxist John Maclean (1879–1923). research, the personal motivations and commitments of Foes thought her an awful woman; friends like George the researcher-a s-historian, and the processes involved, Bernard Shaw (1856–1950) remembered the power of focusing on my biographical project on Mary Bridges her oratory. The aim of this project was to research and Adams (née Daltry, 1855–1939), one of twenty-nine write a significant, original and debate-changing biog- women members of the London School Board (LSB). raphy to consider the main project of ‘making social- Set up under the 1870 Education Act, school boards ists’ from the standpoint of gender. Such an assessment were the most advanced democratic bodies of their day. has its difficulties but this case outlines the reasons why Ratepayers elected them every three years by secret a study of Mary Bridges Adams is important. ballot, and women could both vote and stand for office. Multiple voting and the possibility of giving your vote The research ‘problem’ to one candidate favoured the representation of elec- toral minorities, especially working people and women. Turning to the past means much more than focusing For example, the nine women elected in 1879 consti- solely upon bureaucrats and politicians who wielded tuted 18 per cent of all LSB members. Women’s enormous influence in the official central state. It can numerical representation in the House of Commons did also involve historical detective work into those places not match this until the 1997 general election, follow- where British women were most influential in the late ing the Labour Party’s adoption of all-w omen shortlists nineteenth century: local education policy and practice. (1993–6). My narrative exposes some of the trials, trib- Mary Bridges Adams was excluded from high politics ulations and benefits of historical and documentary for the whole of her political career but she did not wil- research, with the intention of providing guidance and fully hide herself from history. In her lifetime she pre- understanding to researchers in the field (see Martin, J., served myriad press cuttings about her activism, and 2013). her public utterances stressed her contribution to educa- tion and politics as part of the story of British social- The background to the research ism. A truer picture of the past requires an appreciation that British women, like British working men, played From the 1890s, Mary Bridges Adams (she did not an active role in politics in the years before they hyphenate her surname but others, including her son, obtained the national vote. Researchers have so far built did) played an active public role in the British labour up only a partial picture. movement, even though women did not achieve the In her day, Mary was a national figure in British vote in general elections until 1918. At a local level, socialism. Nonetheless, her voice is absent from the she spent seven years as a member of the LSB, then the established canons of political history, despite inclusion largest and most powerful organ of local government in in the Dictionary of Labour Biography. The exception the world. Mary’s voice rose out in particular. She was to this is Patricia Hollis’s Ladies Elect, published in in a minority of one as a socialist woman of working- 1989. However, her activities do grace the footnotes of class social origin. A unionized worker who called some educational histories (Simon, 1965; Kean, 1990; herself the ‘representative of organized labour’, she Manton, 2001). Building on this, the objectives of the joined the National Union of Gas Workers and General Bridges Adams project were to: (1) offer a timely and 328
Historical and documentary research wide-ranging reappraisal of an era of radical social even if she did not always influence decision making. reform seen through the lens of a pivotal figure; (2) Ahead of her time, she supported the extension of piece together the various parts of Mary’s life: public opportunities for all working-c lass children and specifi- and personal, open and hidden; (3) provide context and cally attacked the idea of a meritocratic ‘ladder of a conceptual tool for understanding developments at a opportunity’ for the few. crucial turning point in English education, 1890–1910; One of ‘my women’ at the heart of what started life and (4) challenge political narratives and promote dif- as a collective investigation of women and educational ferent ways of thinking about the place of the educa- policy making in late Victorian and Edwardian England tional question in the study of British socialism. (Martin, 1999), Mary was little more than a footnote in Mary’s life, her attitudes and actions, her role in the history of education when I began my learning respect of campaigns for improvements in working- journey. Education politics provided early opportuni- class education, are accessible only through the docu- ties for Victorian women and, initially, my main mentation that has survived. We do not know where research questions focused on the workings of gender she was educated, the schools in which she taught or and power on the London School Board, investigating what kind of classroom teacher she was. Yet she was a the ‘success’ factors that facilitated women’s careers in prolific writer of articles and her national reputation as public life and the impact of their presence. Mary was a speaker meant that her political activities were regu- one of these pioneer political women, serving from larly reported, and some of her spoken words were 1897 to 1904 (when the Board was abolished under the relayed in local newspapers. Mindful of the risks of 1903 Education Act). When it came to interpreting the formlessness, the objective was to explore and assess sources, meaning could be imputed, not always demon- the life’s work of Mary Bridges Adams using a range strated. It is hard to apply key sociological concepts of sources – textual, visual, oral – in ways that allow such as power, authority and control to second-h and the reader to understand complexity rather than force accounts of a given historical situation. Ambitious Mary’s experiences into an over-s implified pattern. plans to employ all sources quickly evaporated given the constraints of time and space. A more realistic deci- Initiating the process of researching the sion was taken to focus on the official material on the past: the importance of archives LSB and the Board’s official organ The School Board Chronicle that included details of debate at the weekly It was a hot July day in 1990 when I first read Mary’s meetings of school boards throughout England and passionate speech in support of free school meals in a Wales with extensive coverage of events in London. dusty copy of the School Board Chronicle in the Other published commentary consulted included the London Metropolitan Archives. Just after her election teachers’ press, contemporary periodicals and local as a member for Greenwich in the autumn of 1897, newspapers, supported by reference to the archives of there she was at a public Board meeting effectively other relevant institutions, persons and parliamentary telling the upper-m iddle-class membership of the LSB papers. that they could not possibly imagine what it was like to As my gaze turned to the search for Mary’s story, I be poor. Writing this woman’s life had value for me tracked down the references that followed Mary’s entry because of her lifelong concern for class justice. If she in the Dictionary of Labour Biography. Texts were had emerged victorious from the political battles that sampled to address specific aspects of my main ques- she fought at a crucial turning point in English educa- tions about gender and power: What was the way in tion, my mother’s family would not have felt the sting which Mary’s life unfolded within, and was shaped by, that came from having to leave school at thirteen or and helped to shape, a particular political, economic, fourteen. She hooked me then and she hooks me now. social and cultural context? What were Mary’s distinc- Fairly quickly, I found myself feeling possessive of my tive qualities – personal and political? What was the subject rather like the woman researcher A. S. Byatt impact of her contribution to developments in working- describes in her brilliant 1990 novel Possession. class education, as a committed democrat, socialist and My starting point was to go to sources: institutional internationalist? What were the dilemmas and contesta- records, personal papers, previously unknown and tions that she encountered? What were the criticisms under-u tilized contemporary material, autobiographies, of her? biographies and biographical dictionaries, notably the I constructed my account of Mary’s public life from Dictionary of Labour Biography. The School Board a wider variety of sources, including her own published Chronicle proved crucial because it offered a blow-by- letters and articles culled from the newspapers of blow account of weekly debates at the meeting of the the day, reports in the socialist and educational press, whole Board. Mary’s rhetorical skills were evident, 329
Methodologies for educational research official records, memoirs and other autobiographical at risk, estimated as 10 per cent of the school popu writings, present-d ay books and articles. I benefited lation. In this context, a new category of ‘pre- enormously from an oral history interview with Mary’s tuberculous’ child became the focus of work on the grandson, Nicholas, and the support of his widow, health benefits of fresh air, sunshine, healthy food and Jenifer. Both offered rare vignettes of a personal kind. daily exercise. Mary wanted to establish a national The LSB Minutes contain much informative detail, system of open-a ir recovery schools and asked the such as data on attendance at meetings, voting records RACS to donate some of its land at Bostall Wood, and motions of policy. A search of the index took me to Plumstead, for an experiment. The cooperators agreed Mary’s first appearance via the correspondence pages to her request, the London County Council accepted for 3 May 1895. On that day, six working-class politi- their offer, and the mixed school opened on 22 July cal organizations wrote to request that she inherit the 1907, with 113 weak children. A close reading of press Greenwich seat of a recently deceased Board member. cuttings filed by the cooperators that survive within the ‘Mrs. Bridges Adams, from her learning, great scholas- RACS Papers in Woolwich, London, shows that 143 tic experience, lucidity of thought and expression, newspapers discussed the venture, but only Clarion, aptness of resource and charm of presence would add Justice, the Lancashire Post and the Morning Leader distinction to the Board’, Woolwich Gas Workers mention Mary’s contribution. Her supporters noticed wrote. ‘She had the full support, during the last Elec- this. Under the headline ‘Credit where it is not due’, tion of all the Labour Bodies and polled the largest Justice, the weekly paper of the Marxist Social Demo- number of votes of any defeated candidate.’ In contra- cratic Federation, accused the Conservative councillor vention of established protocol, Board members voted Ernest Gray (1857–1932) of trying to gain recognition to co-o pt a male lawyer. Keir Hardie (1856–1915), for a scheme which was not of his making. leader of the UK’s Independent Labour Party, angrily protested how the established parties closed ranks to Widening the search: uncovering public and keep a socialist out, with the one dissentient hastily private resources and some practical explaining his error (Labour Leader, 18 May 1895, considerations p. 8). Besides the published Minutes, background infor- Much of Mary’s evidence is missing because her son mation on the achievements of the LSB was gleaned destroyed the letters written by her to him, and all from the annual report and address of the chairman photographs of her. As a consequence, her personal (available on open access). I was able to order relevant papers are only fragments and her surviving letters are records produced by the sub-c ommittees on which mostly short. Widening the search, I was able to track Mary served, together with those on the Board schools down a mysterious suitcase of papers left with Mary by for which she became a manager. The amount of infor- a leading Russian émigré deported after the Revolution mation I gathered from the committee records varied. in 1917, now the Mary Bridges-Adams Collection on Sometimes I would spend what seemed an eternity the British Labour Movement and Russian Socialists, hunting down documents that turned out to have 1905–39, held in the Rare Book and Manuscript nothing much of interest. Sometimes I would use files Library at Columbia University. Besides Russian that were a gold mine of information, such as the 1899 letters, they included illuminating correspondence with Report on a Special Committee of the General Purposes Mary’s patron, the Countess of Warwick (1861–1938). Committee, particularly helpful on the question of In association with the study I have compiled a Mary’s campaign for school meals, or the 1907 Report modest database of biographical information to investi- on the Bostall Wood Open-air School that ran for three gate Mary’s networking through the use of prosopogra- months in the summer of 1907. The experimental open- phy, a historical method involving the examination of a air school was Mary’s initiative, conducted on land number of lives in a given place, to show that socialism owned by the Royal Arsenal Co-operative Society was both a lifestyle and a form of organized political (RACS). The support of the RACS, the largest of activism. This approach facilitated an assessment of its kind in Greater London, was critical to Mary’s suc- social and intellectual backgrounds from a grounded cessful candidature for the LSB. Information about and qualitative perspective, showing the sequencing of their association can be found in the Half-Yearly connections located in time and space, social history Reports and the Bulletin of the Society, known as and social geography. All political movements are as Comradeship. much the history of social and intellectual networks as Tuberculosis was the prime cause of child deaths at they are of campaigns and lobbying. Making sense of the time and doctors claimed they could identify a child Mary’s political career, her circle of influence connected to leftist counter-c ultures in Glasgow, 330
Historical and documentary research Lancashire, London, the Rhondda and West Yorkshire. speculate where she trained to teach. No school records Biographical approaches helped generate what Clifford for the locality have survived (before 1910) and the Geertz (1973) called ‘thick descriptions’ in ethnogra- names of pupil teachers do not appear in the Newcastle phy (see Chapter 15 of the present volume). Mary’s life School Board Minutes until the late 1870s. No clues is historically placed. The history-w riting is a graphic were found in the Newcastle directories. encapsulation of the cultural milieu of which she was a Visual remains have been hard to come by, apart part, to convey a sense of how it felt to be there, the from the passport-s ized studio photographs of the motivations of the struggle, from the point of view of newly elected London School Board which take up participants who shared her vision of the future. several pages of the Illustrated London News in Accordingly, the works of other historians, political December 1897. Mary is shown in profile. Luxuriant, writers and journalists of different hues, biographical almost pre-R aphaelite-styled hair frames a strong, writings of fellow travellers and contemporaries were attractive face and she fixes the camera with a com- used to contextualize Mary’s experience of maturing in manding gaze. This is in stark contrast to her female the 1870s and 1880s. It is not too difficult to reconstruct colleagues, their hair scraped off their foreheads and some kind of picture of the social structure and life of pulled behind their ears. Ten years later, there is a Daily the nineteenth-c entury South Wales mining village in Mirror photograph of two ladies in hats – Mary and the which she was born or the Elswick district of aristocratic Daisy Warwick – resplendent in an open Newcastle-u pon-Tyne in which she grew up, with the carriage, alongside a report on the 1907 Trades Union aid of secondary sources, contemporary histories, trade Congress. In a staged photo shoot they look directly at directories, business records, government reports, the camera with panache. There may be gaps in the maps, census material, newspapers, autobiographies account but the real problem in this research centred on and so on. Local history collections were mined to rein- how to handle those questions dealing with Mary’s atti- sert Mary’s presence into the social and political land- tudes, values, beliefs and aspirations. Writing history scape of the period. The best starting point for ‘from the bottom up’ meant finding creative ways to information relating to local government and politics is scrape that bottom for any smidgen of information. the local and regional newspapers for the place in ques- tion. In time, knowledge and understanding based on Building up the political profile reading the School Board Chronicle were supplemented by reports in local newspapers. Bridges Adams was among the thousands of women My own experience of intermittently researching recruited to teach in the nineteenth-c entury elementary and writing Mary’s biography speaks directly to how school system. She objected to the unequal social order the Internet age is affecting scholarship. The expanded that she saw strengthened by the educational process, capabilities of the web are changing our methods of and a vision of a better society carried her out of the historical research. Now, digitized guides serve as time- classroom and into political action. In her late twenties, saving devices for the kind of historical detective work she decided to ‘cross the river of fire’ and enter the described here. Proper integration of a longue durée socialist movement. This was how William Morris perspective requires me to say that if I were starting the (1834–96) characterized the life-changing experience research now it would be far easier to track down the that becoming a socialist represented in the 1880s letters produced by Bedford College, for example, that and 1890s. Eager idealists, borne along by an almost showed how Mary obstinately fought her way into millennial fervour, the men and women (like Mary) tuition in the Classics. who made up the ‘pioneering generation’ of British Up to 2002, I had used the Internet to prepare to socialists spent much time and energy spreading visit traditional archives, but never to retrieve actual the word. material. So, when I started ‘looking for Mary’, a local Creating the National Labour Education League in archives service found her in the 1871 Census listed as 1901 represented a transition in Mary’s political a fifteen-y ear-old pupil-teacher (an apprenticeship journey, putting her on a trajectory of moving between scheme for training teachers in school settings, common local and national activities at several levels. Against a at the time) living with her family at the Robin Adair background of mounting tensions in the educational public house in Elswick. More recently, I was able to world, the League heralded a new step forward – the use subscriber databases to supplement the family closest thing to a Labour education policy then in exist- history and the history for the streets in which she lived ence. For her, the advance of education and political and those of her political supporters. But it remains the progress were part of a single programme. The case that the paucity of sources means that I can only abolition of the School Board in 1904 did not diminish an extra‑parliamentary activism which focused on 331
Methodologies for educational research education but couched the question within wider ques- only part of this. There was also her preparedness to tions of social justice and other interests. With the challenge orthodoxy, demonstrated by her writings and financial support of her aristocratic patron, Mary used action, all of which suggests a deep commitment to fur- her training in street-c orner meetings and outdoor agi- thering the cause of socialism in a time of intense con- tation to fight for open-a ir schools for malnourished flict over the shape and purpose of education. We know and tubercular children, free school meals and medical she troubled the establishment elite, but in assessing inspection. her contributions to the conflicts of the years 1890 to After 1908, Mary was closely involved with the 1910, we need to question the conventional wisdoms Marxist educators of the Plebs League and the labour since, more often than not, they vindicated the wisdom colleges. Like them, her conception of education for the of the powers that be. In so doing we need to go back workers was rooted in the Marxist studies promoted by to the margins, to listen to the testimony of those to the earlier Socialist League slogan: ‘Educate, agitate, whom she was closest. Mary’s specific utility as a his- organise!’ She advocated direct action mounted by torical subject is to represent the common, unnamed rank-and-file agitators and Marxist pedagogues. This socialist woman who fought to bring class conscious- included support for a strongly political conception of ness into being at a specific conjuncture. For Mary, adult education, identifying a distinctively socialist or education was the path to a new social order. working-c lass curriculum. Therefore, she opposed the liberal education philosophy espoused by the Workers 16.7 Conclusion Educational Association (WEA, founded in 1903). She was tireless in her efforts to win trade union backing To those who would do the spade work, different for the principle of working‑class self-organization and approaches are possible even with the existing sources. even secured funding to provide an educational space It is possible to recreate ‘lost lives’ and events by going for politically active working-class women akin to the beyond the official record and digging for the raw School of Social Science maintained by the German material of history. There is a remarkable amount of Social Democratic Party. In 1912, she opened Bebel unexploited personal and ordinary information out House in Lexham Gardens, round the corner from the there and digitization is opening up all kinds of possi- Central Labour College in London’s Earls Court, and bilities. Ultimately, there is great joy and sense of satis- installed herself as resident principal. In these years she faction to be had when a historical ‘puzzle’ seemingly built alliances with radical suffragists, attested to by her falls into place. Despite the attrition of memories and attempts to reach working-c lass Lancashire women as histories I did manage to tell Mary’s story and thereby political editor of the Cotton Factory Times. Almost reveal a valuable history long overlooked. I was every week for many years, the Cotton Factory Times uplifted when, in closing the hermeneutic circle, a included an editorial by her. colleague commended my journey through social Recently opened Home Office files on some of the histories, biography and politics (Martin, 2010) as elu- Russian émigrés with whom she worked helped recon- cidating the fictional past contrived in A. S. Byatt’s The struct Mary’s activities in the First World War, shed- Children’s Book published in 2009: embedded in the ding light on her character and personality (though thoughts, beliefs and feelings of late-V ictorian and individual documents remained closed). Press cuttings Edwardian England. kept by her opponents in the WEA were deeply reveal- But whatever the category of evidence, the past does ing and the librarian at the TUC Library Collections not in any automatic way ‘speak for itself ’. It can prove kindly brought them to my attention when they were impossible to locate information we ‘expect’ to find, so deposited there. Some of Mary’s surviving letters were be prepared for frustrations and disappointments and excavated from the Sheehy Skeffington Papers held in allow your research questions to be guided by the mate- the National Library of Ireland, which include Mary’s rial available, rather than establishing an experimental letter of sympathy concerning the murder of Francis hypothesis approach. Outstanding history-writing Sheehy Skeffington in Dublin at Easter in 1916 and involves conveying to readers something of the proc- active support for his widow Hanna’s campaign for a esses by which the raw materials of history have been public inquiry. produced. To return to the notion of a ‘foreign country’ with Closing the loop which our narrative began: they did do things differ- ently there, but we owe a duty to past, present and Mary’s activities are useful for plotting women’s roles future generations to represent all those pasts and all in British leftist ‘oppositional networks’. The experi- those interlocutors as accurately as possible (see also ence was a contradictory one, but being a woman was 332
Historical and documentary research Lowenthal, 2015). When it comes to our educational underclasses, the unprivileged and the defeated, are past, historical and documentary research needs to told with integrity and not quietly forgotten. In this include not just the leaders but the people who inhabit way, a theoretically informed history of education that the classrooms and the forms, dimensions and meaning gives recognition to alternative trajectories and the road of their experience, to bring history into and out of the not followed can make a fine contribution to the work community and ensure that the unknown ideas from the of creating a better future. Companion Website The companion website to the book includes PowerPoint slides for this chapter, which list the structure of the chapter and then provide a summary of the key points in each of its sections. These resources can be found online at www.routledge.com/cw/cohen. 333
Surveys, longitudinal, CHAPTER 17 cross-sectional and trend studies There are many different kinds of survey; each has its 17.2 What is a survey? own characteristics and key issues. We set these out in this chapter, addressing such matters as: Many educational research methods are descriptive; that is, they set out to describe and to interpret what is. Such OO what is a survey? studies look at individuals, groups, institutions, methods OO some preliminary considerations and materials in order to describe, compare, contrast, OO planning a survey classify, analyse and interpret the entities and the events OO low response and non-response, and how to that constitute their various fields of enquiry. We deal here with several types of survey research, including lon- reduce them gitudinal, cross‑sectional and trend or prediction studies. OO survey sampling Typically, surveys gather data at a particular point OO longitudinal, cross-s ectional and trend studies in time with the intention of describing the nature of OO strengths and weaknesses of longitudinal, cohort existing conditions, or identifying standards against which existing conditions can be compared, or deter- and cross-s ectional studies mining the relationships that exist between specific OO postal, interview and telephone surveys events. They may vary in their levels of complexity, OO comparing methods of data collection in surveys from those which provide simple frequency counts to those which present relational analysis. 17.1 Introduction Surveys may be further differentiated in terms of their scope and complexity. A study of contemporary We advise readers to take this chapter in conjunction developments in post-secondary education, for example, with Chapters 12 (sampling), 24 (questionnaires), 25 might encompass the whole of Europe; a study of (interviews) and data-a nalysis techniques (Part 5). subject choice, on the other hand, might be confined to Many researchers reading this book will probably be one secondary school. studying for higher degrees within a fixed and maybe short time frame; that may render longitudinal study 17.3 Advantages of surveys out of the question for them. Nevertheless longitudinal study is an important type of research, and we intro- A survey has several characteristics and several claimed duce it here. More likely, researchers for higher degrees attractions; typically it is used to scan a wide field of will find cross‑sectional survey research appropriate, issues, populations, programmes, people etc. in order to and it is widely used in higher degree research. measure or describe any generalized features. It is useful In many quarters, Internet surveys are becoming the (OECD, 2012; Dillman et al., 2014) in that it often: predominant method of surveys, through email (with a questionnaire as an attachment, or embedded in the OO gathers data on a one-s hot basis and hence is eco- email, or with a hyperlink link to a website, social net- nomical and efficient; working site, special interest group, listserv, discussion group etc.), with companies providing free or low-cost OO represents a wide target population (hence there is a software to design questionnaires and, indeed, to need for careful sampling, see Chapter 12); conduct the survey and collect data for researchers. Given the rise and widespread usage of Internet OO generates numerical data; surveys, we devote an entire, separate chapter (Chapter OO provides descriptive, inferential and explanatory 18) to this. However, we include reference to Internet surveys in Table 17.3 in this chapter, for purposes of information; comparison with other means of survey design and OO manipulates key factors and variables to derive fre- conduct. quencies (e.g. the numbers registering a particular opinion or test score); 334
S u r v e y s , l o n g i t u d i n a l , c r o ss - s e c t i o n a l a n d t r e n d s t u d i e s OO gathers standardized information (i.e. using the same the discussion of exploratory and confirmatory analysis instruments and questions for all participants); in Chapter 43). Surveys can be descriptive or analytic (e.g. to examine relationships). Descriptive surveys OO ascertains correlations (e.g. to find out if there is any simply describe data on variables of interest, whilst relationship between gender and test scores); analytic surveys operate with hypothesized predictor or explanatory variables that are tested for their influence OO presents material which is uncluttered by specific on dependent variables or relationships between contextual factors; variables. Many surveys combine nominal data on partici- OO captures data from multiple-c hoice, closed ques- pants’ backgrounds and relevant personal details with tions, test scores or observation schedules; other data (e.g. attitude scales, data from ordinal, inter- val and ratio measures) (see Chapter 38). Surveys are OO supports or refutes hypotheses about the target useful for gathering factual information, data on atti- population; tudes and preferences, beliefs and predictions, opinions, behaviour and experiences – both past and present OO generates accurate instruments through piloting and (Weisberg et al., 1996; Aldridge and Levine, 2001; revision; Dillman et al., 2014). Their attraction lies in their appeal to generalizability or universality within given OO makes generalizations about, and observes patterns parameters, their ability to make statements which are of response in, the targets of focus; supported by large data and their ability to establish the degree of confidence which can be placed in a set of OO gathers data which can be processed statistically; findings. OO uses large-s cale data gathered from a wide popula- On the other hand, if a researcher is concerned to catch local, institutional or small-s cale factors and vari- tion in order to enable generalizations to be made ables – to portray the specificity of a situation, its about given factors or variables. uniqueness and particular complexity, its interpersonal dynamics, and to provide explanations of why a situa- Examples of surveys are: tion occurred or why a person or group of people returned a particular set of results or behaved in a par- OO test scores (e.g. from students nationally, interna- ticular way in a situation, or how a programme changes tionally, locally); and develops over time – then a survey approach may be unsuitable. Its explanatory potential or fine detail is OO students’ preferences for particular courses, for limited; it is lost to broad-b rush, often descriptive gen- example, humanities, sciences; eralizations which are free of temporal, spatial or local contexts. Williams et al. (2016) note that having a two- OO attitudes to, and opinions of, quality of teaching; phase process of a postal survey – an initial screening OO surveys of groups of people’s values over time; survey followed by the topic-b ased survey sent to eligi- OO surveys of factors (e.g. income levels, social class ble people – is also a useful device for obtaining more in-depth data. In a survey the individual instance is sac- membership, inequality) over time; rificed to the aggregated response (which has the attrac- OO opinion polls; tion of anonymity, non‑traceability and confidentiality OO reading and mathematics performance surveys. for respondents and opportunity for trends and patterns to be discovered). Surveys in education often use test results, self- Surveys typically, though by no means exclusively, completion questionnaires and attitude scales. Here a rely on large-scale data, for example, from question- researcher may be seeking to gather large-scale data naires, test scores, attendance rates, results of public from as representative a sample as possible in order to examinations etc., all of which enable comparisons to say with a measure of statistical confidence that certain be made over time and between groups. This is not to observed characteristics occur with a degree of regular- say that surveys cannot be undertaken on a small-s cale ity, or that certain factors cluster together (see Chapter basis, as indeed they can; rather it is to say that the gen- 43) or that they correlate with each other (correlation eralizability of such small-s cale data will be slight. In and covariance), or that they change over time and surveys the researcher is usually an outsider; indeed location (e.g. results of test scores used to ascertain the questions of reliability and possible bias can attach ‘value-a dded’ dimension of education, maybe using regression analysis and analysis of residuals to deter- mine the difference between a predicted and an observed score), or regression analysis to use data from one variable to predict an outcome on another variable. Surveys can be exploratory, in which no assump- tions or models are postulated, and in which relation- ships and patterns are explored (e.g. through correlation, regression, stepwise regression and factor analysis). They can also be confirmatory, in which a model, causal relationship or hypothesis is tested (see 335
Methodologies for educational research themselves to researchers conducting survey research involving lectures, readings, etc., or practically oriented on their own subjects, for example, participants in a involving workshops and the production of curriculum course that they have been running. Further, it is criti- materials). cal that attention is paid to rigorous sampling, other- As these details unfold, consideration has to be wise the basis of the survey’s applicability to wider given to the most appropriate ways of collecting items contexts is seriously undermined. Non-probability of information (interviews with selected teachers, samples tend to be avoided in surveys if generalizabil- postal questionnaires to selected schools, online ques- ity is sought; probability sampling will tend to lead to tionnaires etc.). generalizability of the data collected. The population upon which the survey is 17.4 Some preliminary focused considerations The second prerequisite to survey design, the specifica- A fundamental decision by the researcher is whether a tion of the population (e.g. people, issues) to which the survey is the appropriate means of answering the enquiry is addressed, affects decisions that researchers research purposes and research questions (Magee et al., must make both about sampling and resources. In our 2013). Assuming that it is, three prerequisites to the hypothetical survey of in‑service requirements, for design of any survey are: the specification of the exact example, we might specify the population as ‘those purpose of the enquiry; the population and issues on primary and secondary teachers employed in schools which it is to focus; and the resources that are avail within a thirty-mile radius of Loughborough Univer- able. Hoinville and Jowell’s (1978) consideration of sity’. In this case, the population is readily identifiable each of these key factors in survey planning can be and, given sufficient resources to contact every member illustrated in relation to the design of an educational of the designated group, sampling decisions do not enquiry. arise. Things are rarely so straightforward, however. Often The purpose of the enquiry the criteria by which populations are specified (‘severely challenged’, ‘under-a chievers’, ‘intending First, a survey’s general purpose must be translated into teachers’ or ‘highly anxious’) are difficult to operation- a specific central aim. Thus, ‘to explore teachers’ views alize. Populations, moreover, vary considerably in about in-s ervice work’ is somewhat nebulous, whereas their accessibility; students and student teachers are ‘to obtain a detailed description of primary and second- relatively easy to survey, travellers’ children and ary teachers’ priorities in the provision of in-service headteachers are more elusive. More importantly, in a education courses’ is reasonably specific. large survey, researchers usually draw a sample from Having decided upon and specified the primary the population to be studied; rarely do they attempt to objective of the survey, the second phase of the plan- contact every member. We deal with the question of ning involves the identification and itemizing of sampling shortly. research questions which will enable the objective to be addressed. The third phase, usually driven by the The resources available research questions, is to identify subsidiary topics that relate to its central purpose. In our example, subsidiary Resources are not simply financial. For example, issues might well include: the types of courses required; survey design can be costly in terms of time, and con- the content of courses; the location of courses; the sideration of resources has to include human, material, timing of courses; the design of courses; and the financ- financial, administrative, temporal, geographical, tech- ing of courses. nical (e.g. computer-related) costs. An important factor The fourth phase follows the identification and item- in designing and planning a survey is financial cost. ization of subsidiary topics and involves formulating Sample surveys are labour-intensive, the largest single specific information requirements relating to each of expenditure being fieldwork, where costs arise out of these issues. For example, with respect to the type of interviewing time, travel time and transport costs of the courses required, detailed information would be needed interviewers themselves. There are additional demands about the duration of courses (one meeting, several on the survey budget. Training and supervising the meetings, a week, a month, a term or a year), the status panel of interviewers can often be as expensive as the of courses (non-a ward bearing, award bearing, with cer- costs incurred during the time that they actually spend tificate, diploma, degree granted by college or univer- in the field. Questionnaire construction, piloting, print- sity), the orientation of courses (theoretically oriented ing, posting, coding, together with computer program- ming and processing all eat into financial resources. 336
S u r v e y s , l o n g i t u d i n a l , c r o ss - s e c t i o n a l a n d t r e n d s t u d i e s Mode of data collection the consent is being given for, as, for example, to ask for consent before the questions have been asked is There are two main issues to be addressed here: asking participants to take a leap of faith. We also note in Chapters 7 and 28 that informed consent is complex, 1 Will the researcher be completing the survey by as it is unclear what is being consented to, and for how entering data, or will the participants be self- long, and for what purposes and uses, and that these administering the survey? problems are exacerbated when data are archived for future use as secondary data sets. Informed consent 2 How will the survey be administered, for example, a should also include the right not to participate or to postal survey, a telephone survey, an Internet survey, withdraw at any time. by face-to-face interviews, group-administered Ethical issues here also concern attention to confi- surveys, self-administered surveys, drop-off surveys, dentiality, anonymity, privacy and non-traceability. In email? A full account of the interview as a research paper-based surveys this may be easy to guarantee, but, technique is given in Chapter 25. as we indicate in Chapter 18, for electronic and Internet- b ased (e.g. website and email surveys), no such absolute Dillman et al. (2014) advise researchers to use multiple guarantees are available. Such computer-related prob- and mixed modes of delivery/administration, as this lems raise the matter of data security and identity pro- helps response rates. tection. In electronic and paper surveys, telephone interviewing and face-to-face surveys, the researcher Self-reporting might not ask for, or require, identifying features, or might remove these when storing and archiving data. There can be a large difference in the responses gained However, in group interviews these may not be so from self-reporting and those obtained from face-to- easy to protect (e.g. members of the group may talk to face survey interviews or telephone interviews (Dale, others), and in electronic/Internet-based surveys, the 2006, p. 145; Dillman et al., 2014). Many surveys ask service provider can log and track participants, and data respondents not only to administer the questionnaires miners and hackers can break into data, particularly themselves but also to report on themselves. This may email, even when security steps have been taken. We introduce bias, as respondents may under-report (e.g. to discuss this in Chapter 18. avoid socially undesirable responses) or over-report (to The researcher can, and should, take all reasonable give socially desirable answers). Self-reporting also steps to protect confidentiality, anonymity, privacy and requires the researcher to ensure that: respondents all non-traceability and indicate to respondents what those understand the question, understand it in the same way steps are, recognizing that where there are limits (e.g. and understand it in the way intended by the researcher in electronic surveys), this may lead to some respond- (Kenett, 2006, p. 406). The difficulty here is that words ents not taking part. are inherently ambiguous (see Chapter 24 on question- Underpinning ethical issues in surveys is the naire design), so the researcher should be as specific as requirement of primum non nocere: primarily, do no possible. The researcher should also indicate how much harm. The researcher must take every step necessary to contextual information the respondent should provide, address this. This concerns access to, collection, what kind of answer is being sought (so that the storage, use, dissemination and reporting of data, and respondent knows how to respond appropriately), how subsequent archiving of data or locating the data in the much factual detail is required and what constitutes rel- public domain, with immense care being taken with evant and irrelevant data (e.g. the level of detail or regard to identification and sensitive information. This focus on priority issues required) (pp. 407–8). Further, raises issues not only of removing identifying features, surveys that rely on respondents’ memory may be removing certain data, aggregating or anonymizing prone to the bias of forgetting or selective recall. data, but who owns the data and what rights the owner has, once the data have been given to the researcher. Ethics The researcher has a duty of care and of trust here. Ethical issues are discussed in Chapters 7, 8 and 18, 17.5 Planning and designing a and we refer readers to these; here we note the impor- survey tance of gaining the informed consent of respondents. Whilst completion of the survey might be taken as Whether the survey is large scale and undertaken by giving consent, this may not always be the case, and some governmental bureau, or small scale and carried the completion of a consent form may be needed (though some participants may be suspicious of this), and indeed Dillman et al. (2014) note that asking for consent requires the researcher to make it clear what 337
Methodologies for educational research out by the lone researcher, the collection of information there are to be tested; what variables there are to typically involves one or more of the following data- explore); gathering techniques: structured or semi-structured OO sample selection (e.g. what is the target population; interviews, self-c ompletion (e.g. postal and Internet how can access and representativeness be assured; questionnaires), telephone interviews, Internet surveys, what other samples will need to be drawn for the standardized tests of attainment or performance, and purpose of comparison); attitude scales. OO design of measurements (e.g. what will be meas- Planning a survey involves knowing: (a) what ured, and how (i.e. what metrics will be used – see exactly you wish to find out, and why; (b) what data Chapter 24 on questionnaires); what variables will you need to be able to answer (a); (c) what questions be required; how reliability and validity will be you will ask to acquire the data. Researchers must also assured); consider: sample selection and access to the sample; OO concern for participants (e.g. protection of confi- distribution/data collection and return of surveys; meas- dentiality and anonymity; avoidance of pain to the urement design and data types; ethical issues; piloting; respondents; avoiding harm to those who might be analysis and reporting. affected by the results; avoiding over-intrusive ques- Sapsford (1999, pp. 34–40) suggests that there are tions; avoiding coercion; informed consent; see four main considerations in planning a survey: Chapters 7 and 8). OO problem definition (e.g. deciding what kinds and Typically surveys proceed through well-defined stages, contents of answers are required; what hypotheses outlined in Figure 17.1. Though these are set in a DefineDoebfjiencetiovbejsecotfivthees souf rtvheysurveyChronological sequenceLongitudinal, Formulate research questions/hypotheses cross-section Define the target population Constraints: finance, Decide the sampling frame and sampling time, people, Decide the kind of survey required administration, Decide the issue/content for focus location, software Decide the information needed to address the issues/content Quantitative/ Decide the instrumentation and metrics qualitative Generate, design, draft and format the data-collection instrument Question types Decide how the data will be delivered and collected Pilot and refine the instrument Postal service, Train the interviewers (if appropriate) interviews, Collect the data telephone, Send reminders Analyse the data drop-off, email, Report the results Internet FIGURE 17.1 Stages in planning a survey 338
S u r v e y s , l o n g i t u d i n a l , c r o ss - s e c t i o n a l a n d t r e n d s t u d i e s sequence, the sequence may alter and the process is OO sampling; iterative and recursive. The process moves from the OO response rates, non-responses and attrition; general to the specific. A general research topic is oper- OO the medium of delivery, completion and return of ationalized into component issues and questions, and, for each component, questions are set. As with ques- the survey; tionnaires (Chapter 24), it is important, in the interests OO data entry and data cleaning; of reliability and validity, to have several items or ques- OO data analysis and reporting; tions for each component issue, as this does justice to OO missing data; the all-round nature of the topic. OO data archiving. Rosier (1997, pp. 154–62) suggests that the plan- ning of a survey must include clarification of: It is important to pilot and pre-p ilot a survey. The dif- ference between the pre-pilot and the pilot is this: the OO the research questions to which answers need to be pre-p ilot is usually a series of open-e nded questions provided; that are used to generate items and categories for closed, typically multiple-c hoice questions, whilst the OO the conceptual framework of the survey, specifying pilot is used to test the draft of the actual survey instru- in precise terms the concepts that will be used and ment itself (see Chapter 24). explored; A rigorous survey formulates clear, specific objec- tives and research questions; ensures that the instru- OO operationalizing the research questions (e.g. into mentation, sampling and data types are appropriate to hypotheses); yield answers to the research questions; and ensures that as high a level of sophistication of data analysis OO the instruments to be used for data collection, for required can be done (i.e. as the data will sustain). example, to chart or measure background character- Attention must be given to: the mode of data collec- istics of the sample (often nominal data), academic tion; respondent effort (too much and this can lead to achievements (e.g. examination results, degrees non-response); question wording, sequence and format. awarded), attitudes and opinions (often using ordinal data from rating scales) and behaviour (using obser- Some challenges in planning surveys vational techniques); A survey is no stronger than its weakest point, and we OO sampling strategies and sub-groups within the consider a range of issues here in order to strengthen sample (unless the whole population is being sur- each aspect of a survey (e.g. OECD, 2012). Surveys veyed, e.g. through census returns or nationally must minimize errors caused by: aggregated test scores etc.); OO poor sampling (e.g. failure to represent or include OO pre-piloting the survey (to generate items for the sufficiently the target population); survey); OO poor question design and wording (e.g. failure to OO piloting the survey; catch accurately the views of, or meanings from, the OO data-c ollection practicalities and conduct (e.g. per- respondents or to measure the factors of interest); missions, funding, ethical considerations, response OO incorrect or biased responses; rates); OO low response or non-response. OO data preparation (e.g. coding, data entry for compu- ter analysis, checking and verification); The first of these – a sampling matter – may be caused OO data analysis (e.g. statistical processes, construction by a failure correctly to identify the population and its of variables and factor analysis, inferential characteristics, or a failure to use the correct sampling statistics); strategy, or systematically to bias the sample (e.g. using OO reporting the findings (answering the research a telephone survey based on telephone directory entries, questions). when key people in the population – the poor – may not have a telephone, or may have a cellphone rather than a Ruel et al. (2015) comment that researchers need to fixed line (the young, the middle aged but not the consider: elderly), or using an Internet- or email-b ased survey when many respondents do not have access). We OO the kind of survey to be used; address sampling issues in Chapter 12 and below. OO ethical issues; The second of these is a failure to operationalize the OO questionnaire and instrument design and appearance; variables fairly (i.e. a validity issue) or a failure in the OO question construction (measures, responses and measurement error); OO validity and reliability; 339
Methodologies for educational research wording or meanings used or inferred, such that incor- OO measurement error: inaccurate and unreliable rect responses are collected (a reliability issue) (e.g. response because of (a) the metrics, scales and people may not understand a question, or may misinter- units of measurement used; (b) socially desirable pret it, or interpret it differently). We address this in responses and respondent acquiescence (the ten- Chapter 14 and below. dency to agree with an interviewer rather than The third problem is that some participants disagree) in face-to-face survey interviews; (c) ques- may deliberately over-report or under‑report the real tionnaire features, for example, length, difficulty, situation in – often sensitive – matters. For example, questions asked, complexity, order effects, inter- teenage alcohol, smoking or drug use, underage viewer effects, survey mode (post, telephone, email, sexual relations, bullying, domestic violence, petty interview, Internet etc.). criminality may be systematically under-reported (i.e. be biased), whereas the popularity of a teacher or stu- 17.6 Survey questions dents might be over-reported (i.e. biased). Bias obtains where there is a systematic skewing or distortion in the Though we go into detail about questions and question- responses. naires in Chapter 24, here we give advice on some Further, some questions may rely on memory, and important issues in writing and asking questions in memory can be selective and deceptive (e.g. people surveys (Creswell, 2012; OECD, 2012; Abascal and may not remember accurately). Also, some responses Diaz de Rada, 2014; Champagne, 2014; Dillman et al., will depend on a person’s state of mind at the time of 2014; Colorado State University, 2016): completing the survey – asking a teacher about teacher stress and tiredness late on a Friday afternoon in school OO Ensure that the questions cover the topics and with a difficult class could well elicit a completely dif- research questions comprehensively and with the ferent response from asking her directly after a week’s appropriate scales of measurement and scales (e.g. holiday. Some questions may be so general as to be 1–5, –4 to +4, ‘strongly disagree’ to ‘agree’). unhelpful (e.g. ‘how stressed do you feel?’), whereas others might be so specific as to prevent accurate recall OO Keep the survey simple and short, and use whole, (e.g. ‘how many times have you shouted at a class of short sentences. children in the past week?’) (one solution to the latter might be to ask participants to keep a diary of OO Consider respondent effort: avoid overloading the instances). respondent with thinking, recalling, reading and Fowler (2009, p. 15) suggests that a respondent’s responding. answer is a combination of the true response plus an error in the answer given, with errors coming from OO Ensure that the questions apply to all the many sources. respondents. The fourth of these – low response or non-response – is a problem that besets researchers, and is so signifi- OO Consider the order of the questions (questions are cant that we devote a separate section to it below. not independent of each other, and the answer to one Dillman et al. (2014) identify four key errors to be question may affect the answer to another in the avoided in surveys which seek to represent a wider respondent’s mind, e.g. the primacy effect, ‘carry population: over’ and ‘anchoring effect’ (Dillman et al., 2014, p. 235), i.e. what comes first affects what comes OO coverage error (poor and incomplete representation later and respondents use the early questions as a of the population in the sample). For example, a standard against which they compare the later coverage error might be made if telephone or Inter- questions). net surveys are used, as not everyone has a tele- phone (particularly a landline) or access to, and OO Arrange the order and organization of the survey in familiarity with, the Internet; a way that is easy for the respondent to understand (subheadings in a written survey are important OO sampling error (including inaccurate estimates of the here). population); OO Group together questions that cover similar topics, OO non-response error (the difference between a repre- with subheadings in written surveys, to parallel what sentative result and that obtained from non-response would naturally happen in a conversation (NB if of different individual or groups, i.e. a skewed respondents see two questions as similar then, for response); and consistency, they will give answers which are similar). OO Start the survey with questions that respondents will find meaningful and interesting, and will be able to answer. 340
S u r v e y s , l o n g i t u d i n a l , c r o ss - s e c t i o n a l a n d t r e n d s t u d i e s OO If you are using branching questions, ask all the the survey, and who enters the responses (the branching questions before you ask the follow-u p respondent or the interviewer). questions. OO Consider whether it is advisable to have an inter- viewer present or absent, as the interviewer’s pres- OO Ensure that the wording is comprehensible to the ence may bias the respondent, raising issues of the respondent (use easy words) and judge how the respondent’s concern for (a) social desirability and respondent will regard and feel about the question (b) acquiescence (defined above); acquiescence is a asked. particular problem in questions which include ‘agree’, as there is a tendency to agree. OO Keep sensitive questions until later in the survey. OO Avoid putting the important questions right at the Magee et al. (2013) advise researchers to consider: end of the survey. OO how others have addressed the constructs in ques- OO Consider the willingness of the respondent to answer tion; developing and writing relevant survey items clearly; the questions correctly and honestly, and whether the respondent will actually know the answer (e.g. OO the mode of the item, for example, a statement or a to factual questions or to questions which require question (a question is preferable); long-term memory), i.e. whether the question really applies to the respondent. OO the response (number and type, with no smaller than OO Consider what the question is asking for – for a five-point scale; odd numbers or even numbers in example, factual answers; attitudes, perceptions and scaling; inclusion of positive and negative options opinions; behaviours; events – and how to make or only positive options: avoid agreement- or these clear to the respondent. Some factual informa- positive-o nly responses; label each point in an tion is easy (e.g. gender, age) but other data (e.g. ordinal scale); attitudes, behaviours, those which rely on memory) may be less accurate. OO reliability and validity of items; OO Use concrete, specific and precise terms (define OO ensuring that the question is interpreted by respond- terms concretely) so that the respondent understands exactly what is being asked for in the survey. ents in the way intended. OO Consider the suitability of question types and formats: (a) for nominal variables: dichotomous, Given these points, it is essential that a survey be multiple choice (single choice, restricted number of piloted, and we give guidelines to piloting in Chapter choices, free number of choices); (b) for ordinal 24, for example, for content, coverage, ease of under- variables: rating scales, ranking scales; (c) for inter- standing, timing, redundancy, sensitivity, question val, ratio and continuous variables: constant sum, types, question order, mode of delivery, ease of com- percentages/marks out of ten, open number (e.g. pletion, answerability. number of hours of study in a week); (d) for non- numerical answers: open questions. Decide whether 17.7 Low response, non-response to have a mid-point in scale items; use large-range and missing data scales if subsequent factor analysis is intended; and ensure that response categories are exhaustive, to fit Response and non-response are related to contact, the choices that participants will really want, i.e. that cooperation and ease of conduct, completion and return they enable respondents to say what they want to of the survey (Dillman et al., 2014). Non-response to a say (and this underlines the importance of running a whole questionnaire (‘unit non-response’; Durrant, pilot). 2009, p. 293) or to a specific item (‘item non-response’; OO Avoid: double-barrelled questions (asking more than p. 293) is a serious problem for much survey research, one thing in a single question); long and complex though Denscombe (2009b, p. 282) notes that online questions and vocabulary; technical language; nega- surveys tend to have lower item non‑response than tively worded items; ambiguous questions; leading paper-based surveys, though there may be more drop- questions (those which influence the response and outs before reaching the end of an online survey than in indicate a desired response); questions which may a paper-b ased survey. cause embarrassment. Dale (2006, p. 148) suggests that ‘non-respondents OO Consider the medium of the administration/ almost invariably differ from respondents’, and that this conduct/‘delivery’ of the survey, for example, postal affects the validity and reliability of the responses service, email, face-to-face interview, website, tele- obtained, and their analysis. If non-response is received phone, i.e. visual, oral and aural administration of from a very homogeneous sample then this might be 341
Methodologies for educational research less of a problem than if the sample is very varied. Centre for Research Methods (2016) also suggests that Further, if non-response is received randomly across a using means of groups and sub-groups responding to a sample then this might be less of a problem than if the particular item can be used for imputation. Here one non-response was from a particular sub-sector of the looks for patterns of missing data (any groups of units/ respondents (e.g. a very low or a very high socio- cases or items) and calculates an average value (e.g. on economic group), as this would bias the results (cf. a scaled item) for groups/sub-groups of cases (individu- Dale, 2006, p. 148). A subset of non-response to a als), and reporting standard error. whole questionnaire is item non-response, and here Ary et al. (2002) note that non-respondents may be missing data should not be ignored (Dale, 2006, p. 15). similar to late responders, so it might be possible to use Rubin (1987), Little and Rubin (1989), Allison data from late responders to indicate the possible (2001), Dale (2006, pp. 149–50) and Durrant (2006, responses from non-respondents. This requires the 2009) review a range of different ‘imputation methods’ researcher to identify late responders. for handling and weighting non-response, i.e. methods Missing data within a survey can have many causes. for filling in missing data with ‘plausible values’ in For example, people may not be present on the day of order to render a set of data complete and yet to reduce its administration, or they may not understand the ques- bias in the non‑responses, i.e. that bias which might be tion, or they may take exception to the question or caused by the non-responses having different values overlook it by mistake. Pampaka et al. (2016) give the from the non-missing responses (Durrant, 2009, example of the administration of a school survey on p. 295). These depend on whether the non-response is bullying, where students may be absent without pre- largely confined to a single variable or many variables. dictable reasons, or they are representing their school The researcher has to determine whether there are pat- in a competition (e.g. high-p erforming and highly moti- terns of non‑response, as these affect the method for vated students), or they may be more likely to be handling non‑response. For example, if the non- bullied (p. 19). All these, the authors note, lead to response is randomly distributed across several vari biased data. They note that missing data are a particular ables, with no clear patterns of non-response, then this problem in longitudinal surveys and surveys across may be less problematic than if there is a systematic phase transitions. They note that statistical analysis non-response to one or more variables in a survey (e.g. stepwise regression, which ignores missing data) (Durrant, 2009, p. 295; Dillman et al., 2014). Durrant is dangerous if there are missing data, and they argue (2009) sets out several ways of calculating missing for multiple imputation methods. However, they also values, including: note that multiple imputation methods are essentially speculative, based on simulations (p. 21). OO calculating missing values from regression tech- Pampaka et al. (2016) distinguish between missing niques using auxiliary variables (p. 296); data from units (individuals) and items, but both can lead to a biased response. There are many ways to OO ‘hot deck’ methods, in which sub-g roups of partici- address this, for example, by simply analysing incom- pants (based on their scores on auxiliary variables) plete data, or by weighting, and by imputation. Weight- are constructed and the researcher compares their ing is designed to ensure a better representation of the results to the non-missing results of the respondent population, and it can be used to adjust data for non- who had omitted a particular response (p. 297); response, to bring the data into the correct matching of the population. If the incomplete data are random, i.e. OO ‘nearest neighbour’ techniques, in which the results all cases have equal probability of being missing (as in from a person whose data diverge as little as possi- their example of those students who are absent for ble from those of the missing person are used to unpredictable reasons), then the analysis may be unbi- replace the missing values. ased (the claim of randomness for equality of distribu- tions, see Chapter 20 on experiments). Durrant (2006, 2009) and Dillman et al. (2014) identify For further guidance on weighting, standard error further, statistical methods of calculating missing and imputation, we refer the reader to the sources indi- scores, such as multiple and fractional imputation and cated above and to the guidance from the National propensity score weighting. Durrant makes the point Centre for Research Methods (www.restore.ac.uk). that how one calculates the values of missing data In some cases (e.g. when all the students in a class depends on a range of factors such as the purpose of the complete a questionnaire during a lesson) the response analysis, the variable(s) in question, the kinds of data, rate may be very high, but in other circumstances the any patterns of missing data, and the characteristics and response rate may be very low or zero, either for the fittingness of the assumptions on which the particular intended imputation method is based. The National 342
S u r v e y s , l o n g i t u d i n a l , c r o ss - s e c t i o n a l a n d t r e n d s t u d i e s whole survey or for individual items within it, for type of question, for example, for the question ‘Do you several reasons, for example: agree with public examinations at age 11?’, and let us say that 50 per cent (90 people) indicate ‘yes’ and 50 OO the survey never reaches the intended people; per cent indicate ‘no’. If the 10 per cent who did not OO people refuse to answer; reply (20 people) had said ‘yes’ then this would clearly OO people may not be available (e.g. for a survey swing the results as 110 people say ‘yes’ (55 per cent) and 90 people say ‘no’ (45 per cent). However, if the administered by interview), for example, they may response rates vary, then the maximum variation could be out at work when a telephone survey administra- be very different, as in Table 17.1 (cf. Fowler, 2009, tor calls; p. 55). Table 17.1 assumes that, if 100 per cent had OO people may not be able to answer the questions (e.g. replied, 50 per cent said ‘yes’ and 50 per cent said ‘no’; language, reading, speaking or writing difficulties); the rest of the table indicates the possible variation OO people may not actually have the information depending on response rate. requested; Table 17.1 indicates the possible variation in a OO people may overlook some items in error; simple ‘yes/no’ type of question. If a rating scale is OO the survey was completed and posted but failed to chosen, for example a five-point rating scale, the return; number of options increases from two to five, and, cor- OO the pressure of competing activities on the time of respondingly, the possibility for variation increases the respondent; even further. OO potential embarrassment at their own ignorance if respondents feel unable to answer a question; Improving response rates in a survey OO ignorance of the topic/no background in the topic; OO dislike of the contents or subject matter of the A major difficult in survey research is securing a suffi- interview; ciently high response rate to give credibility and reliabil- OO fear of possible consequences of the survey to ity to the data. In some surveys, response rates can be as himself/herself or others; low as 20–30 per cent, and this compromises the reliabil- OO lack of clarity in the instructions; ity of the data very considerably. There is a difference OO fear or dislike of being interviewed (or of the between the intended and the achieved sample (Fogel- interviewer); man, 2002, p. 105). Punch (2003, p. 43) suggests that it OO sensitivity of the topic, or potentially insulting or is important to plan for poor response rates (e.g. by threatening topic; increasing the sample size) rather than trying to adjust OO betrayal of confidences; sampling post hoc. He also suggests that access to OO losing the return envelope or return address; the sample needs to be researched before the survey OO the wrong person may open the mail, and fail to pass it on to the most appropriate person. TABLE 17.1 MAXIMUM VARIATION FOR LOW RESPONSE RATES IN A Non-response can lead to responses that are systemati- YES/NO QUESTION FOR A 50/50 cally different (i.e. biased) than those from the whole DISTRIBUTION sample or population, as the responses from those who did not respond might be distinctively different from Response rate Variation in the true value of ‘yes’ those who actually responded. (%) and ‘no’ votes (lowest % to highest Later in this chapter we discuss ways of improving % in each category) response rates. However, here we wish to insert a note of caution: some researchers suggest that, for non- 100 50−50 responders to an item, an average score for that item 90 45−55 can be inserted. This might be acceptable if it can be 80 40−60 shown that the sample or the population is fairly homo- 70 35−65 geneous, but, for heterogeneous populations or samples, 60 30−70 or those where the variation in the sample or population 50 25−75 is not known, it may be dangerous to assume homoge- 40 20−80 neity and hence to infer what the missing replies might 30 15−85 have been, as this could distort the results. 20 10−90 Let us suppose that, out of a sample of 200 partici- 10 5−95 pants, 90 per cent reply (180 participants) to a ‘yes/no’ 343
Methodologies for educational research commences, maybe pre-notifying potential participants OO provide information about the research through a if deemed desirable. He argues that a poor response level covering letter and/or advance notification; may also be due to the careless omission of details of how and when the questionnaire will be returned or col- OO avoid making the survey look like junk mail; lected. This is a matter that needs to be made clear in the OO thank the participants in advance; questionnaire itself. In the case of a postal survey a OO indicate that others have already answered the stamped addressed envelope should always be included. Kenett (2006) and Fowler (2009, p. 52) report that survey (do not be dishonest); responses rates increase when people are interested in OO give pre-p aid stamped addressed envelopes for the subject matter of the survey, or if the subject is very relevant to them, or if completing the survey brings return of the survey; them a sense of satisfaction. Denscombe (2009b, OO offer incentives for return (though increasing the p. 288) reports that response rates increase if the ‘respondent burden’ (the effort required by the respond- financial incentive to a high figure does not bring ent to answer a question) is low. commensurate returns in response rates); Further, the design, layout and presentation of the OO for a follow-u p reminder, include a cover page, as survey may also exert an influence on response rate. It this increases response rates; is important to include a brief covering letter that OO make it easy to answer the survey, keeping the explains the research clearly and introduces the respondent effort and burden to a minimum; researcher. The timing of the survey is important, for OO make the questionnaire topic interesting, the design example, schools will not welcome researchers or attractive and the questions interesting, clear and surveys in examination periods or at special periods, easy to answer, with easy-to-follow instructions and for example, Christmas or inspection times (Fogelman, spacing of the text. Make instructions about 2002, p. 106). responses and return very clear and easy; Finally, it is important to plan the follow-up to OO keep the survey short, easy to read and complete, surveys, to ensure that non‑respondents are called again and very clear; and reminded of the request to complete the survey. OO make response modes easy: giving too many kinds Fowler (2009, p. 57) indicates that between a quarter can lower response rates; and a third of people may agree to completing a survey OO avoid open-e nded questions unless these are really if a follow-up is undertaken. important (as the quality of responses is usually There are several possible ways of increasing poor to open-e nded questions: people tend not to response rates to mailed surveys (Aldridge and Levine, write anything or to write very little). Avoid placing 2001; Diaz de Rada, 2005; Fowler, 2009, p. 56; Dens- open-ended questions at the start of a questionnaire; combe, 2014; Dillman et al., 2014; Williams et al., OO consider asking the respondents for an interview to 2016), including: complete the survey questionnaire; OO deliver the questionnaire personally rather than OO use follow-u ps and polite reminders (e.g. by mail, through mail; email, telephone call) in which the reminder is short, OO ensure that the questions or items are non- polite, indicating the value of the respondent’s par- judgemental (e.g.in sensitive matters); ticipation and, if the reminder is postal, another OO avoid asking for sensitive or personal information clean copy of the questionnaire; unless it is absolutely necessary, particularly if asking for identifying features of children; OO use multiple and mixed modes of responding (i.e. OO indicate you own contact details, relevant and avoid relying on a single mode, such as post, email, authentic professional information about yourself website, cellphone app, interview); and how you can be reached; OO assure confidentiality, anonymity, privacy and secu- OO give advance notification of the survey (e.g. by tele- rity of information; phone, post or email); OO send an email reminder to participants very shortly after the distribution of the survey. OO indicate how the survey is important and the benefits from it, and how (and what) the respondents can Cooper and Schindler (2001, pp. 314–15) and Fowler help in answering the survey; (2009, p. 58) report that the following factors make little or no appreciable difference to response rates: OO indicate the institutional affiliation (with a logo) that is sponsoring or supporting the survey and support OO personalizing the introductory letter; for the survey from high-status or influential OO writing an introductory letter; persons; OO promises of anonymity; 344
S u r v e y s , l o n g i t u d i n a l , c r o ss - s e c t i o n a l a n d t r e n d s t u d i e s OO questionnaire length (it is not always the case that a OO deciding whether a sample or the whole population short questionnaire produces more returns than a is necessary (e.g. it may be possible to have a whole long questionnaire, but researchers will need to con- population if access and size render it feasible, such sider the effect of a long survey questionnaire on the as all the staff of a school); respondents – they may feel positive or negative about it, or set it aside temporarily and forget to OO the sampling frame (all those to be included in the return it later); sample); OO size, reproduction and colour of the questionnaire; OO the sampling strategy (probability and non- OO deadline dates for return (it was found that these did probability) and type of sample; not increase response rate but did accelerate the OO sampling error; return of questionnaires). OO weighted samples for small groups (e.g. before the Potential respondents may be persuaded to participate survey is conducted and post-stratification: after the depending on, for example: survey has been conducted). OO the status and prestige of the institution or researcher Often the researcher will not know the population size carrying out the research; or heterogeneity of the characteristics of the population, and, in this event, it is advisable to have as large a OO the perceived benefit of the research; sample as possible (see Chapter 12 for determining OO the perceived importance of the topic; sample size). OO personal interest in the research; We have already seen that due to factors of expense, OO interest in being interviewed, i.e. the interview time and accessibility, it is not always possible or prac- tical to obtain measures from a population. Indeed experience; Wilson et al. (2006, p. 352) draw attention to the OO personal liking for, or empathy with, the researcher; tension between the need for large samples in order to OO feelings of duty to the public and sense of civic conduct ‘robust statistical analysis’, and issues of resources such as cost and practicability (p. 353). responsibility; Researchers endeavour, therefore, to collect informa- OO loneliness or boredom (nothing else to do); tion from a smaller group or subset of the population in OO sense of self-importance. such a way that the knowledge gained is representative of the total population under study, i.e. a sample. Dillman (2007) suggests that response rates can be Unless researchers identify the total population in increased if, in sequence: (a) non‑respondents are sent advance, it is virtually impossible for them to assess a friendly reminder after ten days, stressing the impor- how representative the sample is which they have tance of the research; (b) non-respondents are sent a drawn. Chapter 12 addresses probability and non- further friendly reminder ten days after the initial probability samples, and we refer readers to the detailed reminder, stressing the importance of the research; (c) a discussion of these in that chapter. The researcher will telephone call is made to the respondents (if the number need to decide the sampling strategy to be used on the is known) shortly after the second reminder, indicating basis of fitness for purpose, for example: the importance of the research. Fowler (2009, p. 60) suggests that the initial question- OO a probability and non-p robability sample; naire might also include a statement to say that comple- OO the desire to generalize, and to whom; tion and return of the questionnaire will ensure that no OO the sampling frame (those who are eligible to be follow-u p reminders will be sent (though this may be regarded by some respondents as presumptuous). included); OO the sample size; 17.8 Survey sampling OO the representativeness of the sample; OO access to the sample; Sampling is a key feature of a survey approach, and we OO the anticipated response rate. advise readers to look closely at Chapter 12 (sampling). Researchers must take sampling decisions early in the Even if the researcher has taken extraordinary care with overall planning of a survey (see Figure 17.1) in light the sampling strategy, there may still be problems (e.g. of the population from which they want to sample, and response rate, respondent characteristics or availability) this involves, for example: that can interfere with the best-laid strategies. In addition to the sampling strategy to be used, there OO identifying the target population (who, how large are the issues of sample size and selection. We discussed and what are their characteristics of interest?); 345
Methodologies for educational research this in Chapter 12, but here we wish to address the order to reach this point a series of other steps has to issue of practicability. For example, let us say that, in be taken. the interests of precision, the researcher wishes to have This raises the matter of deciding the steps to be a sample in which there are four strata (e.g. age groups taken. For example, the researcher could decide the in a primary school), and that each stratum comprised sampling for the survey of the biology students by 50 students, i.e. 200 students in total. If that researcher taking the random sample of 1,000 students from ten wished to increase the sample size of one stratum by, schools. The researcher lists all the 1,000 relevant stu- say, 20 students, this would necessitate an overall dents from the list of 10,000 students, and decides to increase of 80 students (20 × 4) in the sample. Do the select 100 students from each of the ten schools (a benefits outweigh the costs here? biology student, therefore, in one of these ten schools An alternative to increasing the total size of the has a one in ten chance of being selected). Alterna- sample would be to increase the size of one stratum tively, the researcher could decide to sample from five only, under certain conditions. For example, let us say schools only, with 200 students from each of the five that the researcher is studying attitudes of males and schools, so students in each of these five schools have a females to learning science, in a secondary school one in five chance of being selected. Alternatively, the which had only recently moved from being a single-sex researcher could decide to sample from two schools, boys’ school to a mixed sex school, so the ratio of male with 500 students, so students in each of these two to female students is 4:1. The researcher wishes to schools have a one in two chance of being selected. include a minimum of 200 female students. This could There are other permutations. The point here is that, as require a total of 1,000 students in the sample (200 the number of schools decreases, so does the possible females + {200 × 4} male students in the sample); this cost of conducting the survey, but so does the overall could be unmanageable. Rather, the researcher could reliability, as so few schools are included. It is a identify two female students for each male student (i.e. trade-o ff. 400 females) and then, when analysing the data, could In order to reduce sampling error (the variation of give one quarter of the weight to the response of the the mean scores of the sample from the mean score of female students, in order to gain a truer representation the population), a general rule is to increase the sample of the target population of the school. This would bring size, and this is good advice. However, it has to be tem- the total sample to 600 students, rather than 1,000, pered by the fact that the effect of increasing the sample involved in the survey. Oversampling a smaller group size in a small sample reduces sampling error more (in this case the females) and then weighting the analy- than in a large sample, for example, increasing the sis is frequently undertaken in surveys (cf. Fowler, sample size from 50 to 80 (30 persons) will have 2009, p. 27). greater impact on reducing sampling error than increas- In sampling, the probability might also exist of ing the sample size from 500 to 530 (30 persons). excluding some legitimate members of population in Hence it may be of little benefit simply to increase the target sample; however, the researcher will need to sample sizes in already-large samples. weigh the cost of excluding these members (e.g. the The researcher has to exercise his or her judgement very hard to reach) against the cost of ensuring that in attending to sampling. For example, if it is already they are included – the benefit gained from including known that a population is homogeneous, then the them may not justify the time, cost and effort (cf. researcher may feel it a needless exercise in having too Fowler, 2009, p. 179). Similarly, the precision gained large and unmanageable a sample if the results are not from stratified sampling (see Chapter 12) may not be likely to be much different from those of a small worth the price to be paid in necessarily increasing the sample of the same homogeneous group (though theo- sample size in order to represent each stratum. retical sampling (see Chapter 37) may suggest where a In many cases a sampling strategy may be in more researcher needs to include participants from other than one stage. For example, let us consider the small samples). As Fowler (2009, p. 44) remarks, the instance of a survey of 1,000 biology students from results of a sample of 150 people will describe a popu- a population of 10,000 biology students in a city. In lation of 15,000 or 25 million with more or less the the first stage, a cluster group of, say, ten schools same degree of accuracy. He remarks that samples of is identified (A), then, within that, a cluster by age more than 150 or 200 may yield only modest gains to group of students (B), and then, within that, the the precision of the data (p. 45), though this, of course, cluster of individuals in that group who are studying has to be addressed in relation to the population charac- biology (C), and, finally, the sample (D) is taken from teristics, the number, size and kind of strata to be that group. The intention is to arrive at (D), but in included, and the type of sample being used. Sampling 346
S u r v e y s , l o n g i t u d i n a l , c r o ss - s e c t i o n a l a n d t r e n d s t u d i e s errors, he notes (p. 45) are more a function of sample population at a given point in time, and hence provide size than of the proportions of the sample to the popu- aggregated data. lation. Further, he advocates probability rather than By contrast, longitudinal studies can also provide non-p robability samples, unless there are persuasive individual-level data, by focusing on the same individu- reasons for non-probability samples to be used. als over time (e.g. the Household Panel Studies which Whilst sample sizes can be calculated on the basis follow individuals and families over time (Ruspini, of statistics alone (e.g. confidence levels, confidence 2002, p. 4). Lazarsfeld introduced the concept of a intervals, population size, statistical power and so on, panel in the 1940s, attempting to identify causal pat- see Chapter 12), this is often not the sole criterion, as it terns and the difficulties in tracing these (Ruspini, accords a degree of precision to the sample which takes 2002, p. 13)). insufficient account of other sampling issues, for example, access, variation or homogeneity in the popu- Longitudinal studies lation, levels of literacy in the population (e.g. in the case of a self-a dministered questionnaire survey), Longitudinal studies can be of the survey type or of number and type of variables and costs. other types (e.g. case study). Here we confine ourselves Sampling is one of several sources of error in to the survey type. Longitudinal studies can include surveys, as indicated earlier in this chapter. trend studies, cohort studies and panel studies (Creswell, 2012), and we discuss these below. A useful 17.9 Longitudinal and centre for longitudinal studies in education is at the cross-sectional surveys University of London: www.cls.ioe.ac.uk/default.aspx. Longitudinal studies can use repeated cross- The term ‘longitudinal’ describes a variety of studies sectional studies, which are conducted regularly, each that are conducted over a period of time. A clear dis- time with a largely different sample or, indeed, an tinction is drawn between longitudinal and cross- entirely new sample (Ruspini, 2002, p. 3), or use the sectional studies. The longitudinal study gathers data same sample over time. They enable researchers to: over an extended period of time: a short‑term investiga- ‘analyse the duration of social phenomena’ (p. 24); tion may take several weeks or months; a long-term highlight similarities, differences and changes over study can extend over many years. Where successive time in respect of one or more variables or participants measures are taken at different points in time from the (within and between participants); identify long-term same respondents, the term ‘follow-up study’ or ‘cohort (‘sleeper’) effects; and explain changes in terms of study’ is used in the British literature, the equivalent stable characteristics, for example sex, or variable char- term in the US being the ‘panel study’. The term acteristics, such as income. The appeal of longitudinal ‘cohort’ is a group of people with some common research is its ability to establish causality and to make characteristic. inferences. Ruspini adds to these the ability of longitu- Where different respondents are studied at one or dinal research to ‘construct more complicated behav- more different points in time, the study is called ‘cross- ioural models than purely cross-s ectional or time-s eries sectional’, i.e. a cross-s ection of the population is taken data’ (p. 26); they can catch the complexity of human to investigate the topic(s) of interest. Where a few behaviour. Further, longitudinal studies can combine selected factors are studied continuously over time, the numerical and qualitative data. term ‘trend study’ is employed. One example of regular Retrospective analysis is not confined to longitudi- or repeated cross-sectional social surveys is the General nal studies alone. For example, Rose and Sullivan Household Survey, in which the same questions are (1993, p. 185) and Ruane (2005, p. 87) suggest that asked every year, though they are put to a different cross-sectional studies can use retrospective factual sample of the population each time. The British Social questions, for example, previous occupations, dates of Attitudes Survey is an example of a repeated cross- birth within the family, dates of marriage and/or sectional survey, using some 3,600 respondents. divorce, though Rose and Sullivan (1993, p. 185) A famous example of a longitudinal (cohort) study advise against collecting other types of retrospective is the UK’s National Child Development Study, which data in cross-sectional studies, as the quality (e.g. relia- started in 1958. The British General Household Panel bility) of the data diminishes the further back one asks Survey interviewed individuals from a representative respondents to recall previous states or even facts. sample each year in the 1990s. Another example is It is important in longitudinal studies to decide when the British Family Expenditure Survey. These latter and how frequently to collect data over time, and this is two are cross‑sectional in that they tell us about the informed by issues of fitness for purpose as well as practicability. Further, in order to allow for attrition 347
Methodologies for educational research (dropout) of the sample, it is wise to have as large a studied over time. New samples – different people – are sample as practicable and possible at the start of the drawn at each stage of the data collection, but focus on study (Wilson et al., 2006, p. 354). the same factors, and if random samples are used, they can be representative of the wider population. By Cohort studies taking different samples the problem of reactivity is A cohort study focuses on a specific population in which reduced (see below: ‘pre-test sensitisation’), i.e. earlier all its members have the specific defining characteristic surveys affecting the behaviour of participants in the that is of interest to the researcher (e.g. the National later surveys. This is particularly useful if the research Child Development Study in the UK; the Millennium is being conducted on sensitive issues, as raising a sen- Cohort Study). In a cohort study the specific population sitive issue early on in the research may change an indi- is tracked over a specific period of time but selective vidual’s behaviour, which could affect the responses in sampling within that sample occurs. This means that dif- a later round of data collection. By drawing a different ferent members of a cohort are included each time. For sample each time, this problem is overcome. example, the population might be eighteen-year-olds in Trend or prediction studies have an obvious impor- the UK; at one time point (say, when they are twenty- tance to educational administrators or planners. Like five years old) the population might be sampled, and cohort studies, they can be of relatively short or long then at another time point (say, when they are thirty-five) duration. Essentially, the trend study examines recorded the same population might be sampled but different data to establish patterns of change that have already members of the population will be in the sample. occurred in order to predict what will be likely to occur Cohort studies and trend studies can be prospective in the future. In trend studies, two or more cross- longitudinal methods, in that they are ongoing in their sectional studies are undertaken with identical age collection of information about individuals or their groups at more than one point in time in order to make monitoring of specific events. Retrospective longitudi- comparisons over time (e.g. the Scholastic Aptitude and nal studies, on the other hand, focus upon individuals Achievement tests in the US and the National Assess- who have reached some defined end-point or state. For ment of Educational Progress results). A major diffi- example, a group of young people may be the research- culty that researchers face in conducting trend analyses er’s particular interest (intending social workers, con- is the intrusion of unpredictable factors that invalidate victed drug offenders or university dropouts, for forecasts formulated on past data. For this reason, short- example), with questions such as: ‘Is there anything term trend studies tend to be more accurate than long- about your previous experience that can account for term analyses. Trend studies do not include the same your present situation?’ Retrospective longitudinal respondents over time, so the possibility exists for vari- studies will specify the period over which to be retro- ation in data due to the different respondents rather than spective, for example, one year, five years. the change in trends. Gorard (2001b, p. 87) suggests that this problem can be attenuated by a ‘rolling sample’ in Panel studies which a proportion of the original sample is retained in In contrast to a cohort study, in a panel study exactly the second wave of data collection, and a proportion of the same individuals are tracked over time. An example this sample is retained in the third wave, and so on. of this is the Panel Study of Income Dynamics in the US. Another example from the UK is the ‘7 Up’ study Cross-s ectional studies which started in 1964 and tracks a small group of indi- viduals every seven years, yielding insights into social A cross-sectional study is one that produces a ‘snap- and cultural stratification, reproduction and the self- shot’ of a population at one particular point in time. fulfilling prophecy. The epitome of the cross-s ectional study is a national Whilst this type of study has the attraction of track- survey in which a representative sample of the popula- ing the same people over time, this same requirement tion consisting of individuals of different ages, differ- also has its disadvantages in terms of keeping contact ent occupations, different educational and income with those individuals and addressing attrition. Panel levels, and residing in different parts of the country, is studies are useful for investigating causality and change interviewed on the same day. In education, cross- over time. sectional studies can involve indirect measures of the nature and rate of changes in the physical and intellec- Trend studies tual development of samples of children drawn from Trend studies focus on factors (e.g. mathematics per- representative age levels. The single ‘snapshot’ of the formance) rather than people, and these factors are cross-s ectional study provides researchers with data for either a retrospective or a prospective enquiry. 348
S u r v e y s , l o n g i t u d i n a l , c r o ss - s e c t i o n a l a n d t r e n d s t u d i e s A cross-sectional study can also bear several hall- other hand, they suffer from problems of attrition (par- marks of a longitudinal study of parallel groups (e.g. ticipants leaving the research over time, a particular age groups) which are drawn simultaneously from the problem in panel studies which research the same indi- population. For example, drawing students aged five, viduals over time), and they can be expensive to seven, nine and eleven at a single point in time would conduct in terms of time and money (Ruspini, 2002, bear some characteristics of a longitudinal study in that p. 71). Gorard (2001b) reports a study of careers and developments over age groups could be seen, though, identities that had an initial response rate of between 60 of course, it would not have the same weight as a longi- and 70 per cent in the first round, and then risked drop- tudinal study conducted on the same age group over ping to 25 per cent by the third round, becoming time. This is the case for international studies of educa- increasingly more middle class in each wave of the tional achievement, requiring samples to be drawn from study; the same publication discusses a Youth Cohort the same population (Lietz and Keeves, 1997, p. 122) Study in which only 45 per cent of the respondents took and for factors that might influence changes in the part in all three waves of the data collection. Ruspini dependent variables to remain constant across the age (2002, p. 72) identifies an attrition rate of 78 per cent in groups. the three waves of the European Community House- Cross‑sectional studies, catching a frozen moment hold Panel survey of the UK in 1997. in time, may be ineffective for studying change or cau- Ruspini also indicates how a small measurement sality. If changes are to be addressed through cross- error in a longitudinal study may be compounded over sectional surveys, then this suggests the need for time. She gives the example of an error in income repeated applications of the survey, or the use of trend occurring at a point in time (p. 72) that could lead to analysis. ‘false transitions’ appearing over time in regard to The main types of longitudinal study are illustrated poverty and unemployment. in Figure 17.2. Further, long-term studies, Gorard (2001b, p. 86) avers, face ‘a threat to internal validity’ that stems from 17.10 Strengths and weaknesses of the need ‘to test and re-test the same individuals’. longitudinal, cohort and cross- Dooley (2001, p. 120) terms this ‘pre-test sensitisa- sectional studies tion’; it is also termed ‘panel conditioning’ or ‘time-in sample bias’ (Ruspini, 2002, p. 73). Here the first inter- Longitudinal studies of the cohort analysis type have an view in an interview survey can cause changes in the important place in the armoury of educational research- second interview, i.e. the first interview might set up a ers. Longitudinal studies have considerable potential self-fulfilling prophecy that is recorded in the second for yielding rich data that can trace changes over time, interview. He gives the example of a health survey and with great accuracy (Gorard, 2001b, p. 86). On the in the first round of data collection, which may raise LONGITUDINAL Cohort study: different Panel study: same Trend study: same people in sample of people each time factor(s) but different same population each time people in sample each time CROSS-SECTIONAL ‘Snapshot’ of sample ‘Snapshot’ of same at one moment kind of sample in time over time FIGURE 17.2 Types of survey 349
Methodologies for educational research participants’ awareness of the dangers of smoking, such which are not easily addressed in cross-sectional that they reduce or give up smoking by the time the designs. second round takes place. Trend studies overcome this Retrospective longitudinal studies rely on partici- problem by drawing different populations at each round pants’ memories which may be faulty; the further back of data collection. one’s memory reaches, the greater is the danger of dis- Dooley (2001) also identifies difficulties caused by tortion or inability to recall. Memory is affected by, for changes in the research staff over time in longitudinal example (Ruspini, 2002, p. 97): surveys. Changes in interviewee response, he suggests, may be due to having different researchers rather than OO the time that has elapsed since the event took to the respondents themselves. Even using the same place; instruments, different researchers may use them differ- ently (e.g. in interviewing behaviour). OO the significance of the event for the participant; To add to these matters, Ruspini (2002, p. 73) sug- OO the amount of information required for the study – gests that longitudinal data are affected by: the greater the amount, the harder it is to provide; OO history (events occurring may change the observa- OO the contamination/interference effect of other mem- tions of a group under study); ories of a similar event (i.e. the inability to separate OO maturation (participants mature at different speeds similar events); and in different ways); OO the emotional content or the social desirability of the content; OO testing (test sensitization may occur – participants OO the psychological condition of the participant at learn from exposure to repeated testing/interviews); interview. OO the timing of cause and effect (some causes may Further, participants will look at past events through produce virtually instantaneous effects and others the lens of hindsight and subsequent events rather than may take a long time for the effects to show); what those events meant at the time. Moreover, it is not always easy for these participants to recall their emo- OO the direction of causality not always being clear or tional state at the time in question. Factually speaking, singular. it may not be possible to gather data from some time past, as they simply do not exist (e.g. medical records, A major concern in longitudinal studies concerns the data on income) or they cannot be found, recovered or comparability of data over time. For example, though accessed. public examinations may remain constant over time Cohort studies of human development conducted on (e.g. GCSE, A levels), the contents and format of those representative samples of populations are uniquely able examinations do not. (This rehearses the argument that to identify typical patterns of development and to reveal public examinations are becoming easier over time.) factors operating on those samples which elude other This issue concerns the need to ensure consistency in research designs. They permit researchers to examine the data-c ollection instruments over time. Further, if individual variations in characteristics or traits, and to comparability of data in a longitudinal study is to be produce individual development curves. Cohort studies, addressed then this means that the initial rounds of data too, are particularly appropriate when investigators collection will need to anticipate and include all the attempt to establish causal relationships, as this variables that will be addressed over time. involves identifying changes in certain characteristics Longitudinal studies are more prone to attrition than which result in changes in others. cross-sectional studies, and are more expensive to Cross-sectional designs are inappropriate in causal conduct in terms of time and cost. On the other hand, research as they cannot sustain causal analysis unless whereas trend studies change their populations, thereby they are repeated over time, as causality has a neces- disabling micro-level – individual-level – analysis from sary time dimension. Cohort analysis is especially being conducted, longitudinal analysis enables such useful in sociological research because it can show how individual‑level analysis to be performed. Indeed changing properties of individuals fit together into whereas cross-s ectional designs (even if they are changing properties of social systems as a whole. For repeated cross-s ectional designs) may be unsuitable for example, the study of staff morale and its association studying developmental patterns and causality within with the emerging organizational climate of a newly cohorts, in longitudinal analysis this is a strength. Lon- opened school would lend itself to this type of develop- gitudinal data can supply ‘satisfactory answers to ques- mental research. A further strength of cohort studies in tions concerning the dynamics and the determinants of schools is that they provide longitudinal records whose individual behaviour’ (Ruspini, 2002, p. 71), issues 350
S u r v e y s , l o n g i t u d i n a l , c r o ss - s e c t i o n a l a n d t r e n d s t u d i e s value takes account of the known fallibility of any ulating them to communicate with others on unwanted single test or assessment. Finally, time, often a limiting topics (see Riley, 1963). Fourth, cohort studies can factor in experimental and interview settings, is gener- suffer from the interaction of biological, environmental ally more readily available in cohort studies, allowing and intervention influences (Keeves, 1997a, p. 139). the researcher greater opportunity to observe trends and Finally, cohort studies in education pose considerable to distinguish ‘real’ changes from chance occurrences problems of organization due to the continuous changes (see Bailey, 1994). that occur in pupils, staff, teaching methods and the In longitudinal, cohort and trend studies the charac- like. Such changes make it highly unlikely that a study teristics of respondents are likely to affect results will be completed in the way that it was originally (Robson, 1993, p. 128). For example, their memory, planned. knowledge, motivation and personality may affect their Cohort studies, as we have seen, are particularly responses, and indeed they may withhold information, appropriate in research on human growth and develop- particularly if it is sensitive. ment. Why then are so many studies cross-sectional Longitudinal research indicates the influence of bio- rather than cohort studies? The reason is that they have logical factors over time (e.g. human development), a number of advantages over cohort studies: they are environmental influences and intervention influences less expensive; they produce findings more quickly; (Keeves, 1997a, p. 139) and their interactions. Address- they are less likely to suffer from control effects; and ing these, the appeal of longitudinal analysis is that it they are more likely to secure the cooperation of enables researches to conduct causal analysis. Time respondents on a ‘one-off ’ basis. Generally, cross- series studies in longitudinal research also enable emer- sectional designs are able to include more subjects than gent patterns to be observed over time, by examining a are cohort designs. given range of variables over time, in addition to other The strengths of cohort analysis are the weaknesses factors. This enables individual and group profiles to be of the cross-s ectional design. The cross-s ectional study examined over time and development, indicating simi- is a less effective method for the researcher who is con- larities and differences within and between individuals cerned to identify individual variations in growth or to and groups in respect of given variables. establish causal relationships between variables. Sam- Longitudinal studies suffer several disadvantages pling in a cross-sectional study is complicated because (though the gravity of these weaknesses is challenged different subjects are involved at each age level and by supporters of cohort analysis). The disadvantages may not be comparable. Further problems arising out are, first, that they are time-consuming and expensive, of selection effects and obscuring irregularities in because the researcher is obliged to wait for growth growth weakens the cross-s ectional study so much that data to accumulate. Second, there is the difficulty of one observer dismisses the method as a highly unsatis- sample mortality. Inevitably during the course of a factory way of obtaining developmental data except for long-term cohort study, subjects drop out, are lost or the crudest purposes. refuse further cooperation. Such attrition makes it Douglas (1976a), who pioneered the first national unlikely that those who remain in the study are as cohort study in any country, makes a spirited defence representative of the population as the original of the method against the common criticisms that are sample. Sometimes attempts are made to lessen the levelled against it – that it is expensive and time- effects of sample mortality by introducing aspects of consuming. His account of the advantages of cohort cross-sectional study design, that is, ‘topping up’ analysis over cross-sectional designs is summarized in the original cohort sample size at each time of re- Box 17.1. testing with the same number of respondents drawn Cross-sectional studies require attention to sampling from the same population. The problem here is that in order to ensure that the information on which the differences arising in the data from one survey to the sample is based is comprehensive (Lietz and Keeves, next may then be accounted for by differences in the 1997, p. 124). Further, there is a risk that some poten- persons surveyed rather than by genuine changes or tial participants may decline to take part, thereby weak- trends. ening the sample, or some may not answer specific A third difficulty has been termed the ‘control questions or, wittingly or unwittingly, give incorrect effect’ (sometimes referred to as ‘measurement effect’). answers. Measurement error may also occur if the Often, repeated interviewing results in an undesired and instrument is faulty, for example, using inappropriate confusing effect on the actions or attitudes under study, metrics or scales. influencing the behaviour of subjects, sensitizing them The comparative strengths and weaknesses of to matters that have hitherto passed unnoticed, or stim- longitudinal studies (including retrospective studies), 351
Methodologies for educational research Box 17.1 Advantages of cohort over cross-sectional designs 1 Some types of information, for example, on attitudes or assessment of potential ability, are only meaning- ful if collected contemporaneously. Other types are more complete or more accurate if collected during the course of a longitudinal survey, though they are likely to have some value even if collected retrospectively, for example, length of schooling, job history, geographical movement. 2 In cohort studies, no duplication of information occurs, whereas in cross-sectional studies the same type of background information has to be collected on each occasion. This increases the interviewing costs. 3 The omission of even a single variable, later found to be important, from a cross-s ectional study is a disaster, whereas it is usually possible in a cohort study to fill the gap, even if only partially, in a subsequent interview. 4 A cohort study allows the accumulation of a much larger number of variables, extending over a much wider area of knowledge than would be possible in a cross-sectional study. This is of course because the collection can be spread over many interviews. Moreover, information may be obtained at the most appro- priate time, for example, information on job entry may be obtained when it occurs even if this varies from one member of the sample to another. 5 Starting with a birth cohort removes later problems of sampling and allows the extensive use of sub- samples. It also eases problems of estimating bias and reliability. 6 Longitudinal studies are free of one of the major obstacles to causal analysis, namely, the reinterpretation of remembered information so that it conforms to conventional views on causation. It also provides the means to assess the direction of effect. Source: Adapted from Douglas (1976b) cross-s ectional analysis and trend studies are summarized not least because the non-p resence of another person in Table 17.2 (see also Rose and Sullivan, 1993, (e.g. an interviewer) can increase the honesty and pp. 184–8). richness of the data, whereas the presence of an inter- Several of the strengths and weaknesses of retro- viewer might inhibit the respondent. Further, in a spective longitudinal studies share the same character- postal survey, the relations of power between the istics as those of ex post facto research, discussed in researcher and the respondent are often more equal Chapter 20. than in an interview situation (in which the former often controls the situation more than the latter) 17.11 Postal, interview and (p. 308). telephone surveys On the other hand, postal surveys typically suffer from a poor response rate, even though Dillman et al. Postal surveys (2014) comment they have moved from having the lowest response rate to having response rates higher There are strengths and difficulties with postal and than telephone surveys. Mailed surveys are reported to interview surveys. Postal surveys can reach a large have an approximately 20 per cent response rate, which number of people, gather data at comparatively low is far lower than telephone and face-to-face surveys cost and quite quickly, and can give assurances of con- (Colorado State University, 2016). Diaz de Rada and fidentiality (Robson, 1993; Bailey, 1994, p. 148; Dominguez (2015) note that postal surveys feature Dillman et al., 2014). Similarly they can be completed greater acquiescence than other kinds of survey, with at the respondents’ own convenience and in their pre- more unanswered questions. ferred surroundings and own time; this can enable them Because researchers may not have any information to check information, if necessary (e.g. personal docu- about non-respondents, they may not know whether the ments), and think about responses. As standardized sample is representative of the wider population. wording is used, there is a useful degree of comparabil- Further, respondents may not take the care required to ity across the responses, and, as no interviewer is complete the survey carefully, and, indeed, may misun- present, there is no risk of interviewer bias. Further, derstand the questions. There is no way of checking postal questionnaires enable widely scattered popula- this. Bailey (1994, p. 149) suggests that the very issues tions to be reached. that make postal surveys attractive might also render Postal surveys can also be used to gather detailed them less appealing, for example: sensitive qualitative data (Beckett and Clegg, 2007), 352
S u r v e y s , l o n g i t u d i n a l , c r o ss - s e c t i o n a l a n d t r e n d s t u d i e s TABLE 17.2 THE CHARACTERISTICS, STRENGTHS AND WEAKNESSES OF LONGITUDINAL, CROSS-SECTIONAL, TREND ANALYSIS AND RETROSPECTIVE LONGITUDINAL STUDIES Study type Features Strengths Weaknesses Longitudinal 1 Single sample over 1 Useful for establishing causal 1 Time-consuming – it takes a long studies (cohort/panel extended period of relationships and for making time for the studies to be studies) time. reliable inferences. conducted and the results to 2 Enables the same 2 Shows how changing properties of emerge. individuals to be individuals fit into systemic 2 Problems of sample mortality compared over change. heighten over time and diminish time (diachronic 3 Operates within the known limits of initial representativeness. analysis). instrumentation employed. 3 Control effects – repeated 3 Micro-level 4 Separates real trends from chance interviewing of the same sample analysis. occurrence. influences their behaviour. 5 Brings the benefits of extended 4 Intervening effects attenuate the time frames. initial research plan. 6 Useful for charting growth and 5 Problem of securing participation development. as it involves repeated contact. 7 Gathers data contemporaneously 6 Data, being rich at an individual rather than retrospectively, thereby level, are typically complex to avoiding the problems of selective analyse. or false memory. 8 Economical in that a picture of the sample is built up over time. 9 In-depth and comprehensive coverage of a wide range of variables, both initial and emergent – individual specific effects and population heterogeneity. 10 Enables change to be analysed at the individual/micro-level. 11 Enables the dynamics of change to be caught, the flows into and out of particular states and the transitions between states. 12 Individual level data are more accurate than macro-level, cross- sectional data. 13 Sampling error reduced as the study remains with the same sample over time. 14 Enables clear recommendations for intervention to be made. continued 353
Methodologies for educational research TABLE 17.2 continued Study type Features Strengths Weaknesses Cross- 1 Snapshot of 1 Comparatively quick to conduct. 1 Do not permit analysis of causal sectional different samples 2 Comparatively cheap to administer. relationships. studies at one or more 3 Limited control effects as subjects points in time 2 Unable to chart individual Trend (synchronic only participate once. variations in development or analysis analysis). 4 Stronger likelihood of participation changes, and their significance. 2 Large-scale and as it is for a single time. 3 Sampling not entirely comparable representative 5 Charts aggregated patterns. at each round of data collection as sampling. 6 Useful for charting population-wide different samples are used. 3 Macro-level features at one or more single 4 Can be time-consuming as analysis. points in time. background details of each sample 7 Enable researchers to identify the have to be collected each time. 4 Enables different proportions of people in particular groups to be groups or states. 5 Omission of a single variable can compared. 8 Large samples enable inferential undermine the results significantly. statistics to be used, e.g. to 5 Can be compare sub-groups within the 6 Unable to chart changing social retrospective and/ sample. processes over time. or prospective. 7 They only permit analysis of overall, 1 Selected factors net change at the macro-level studied through aggregated data. continuously over time. 1 Maintains clarity of focus throughout 1 Neglects influence of unpredicted 2 Uses recorded the duration of the study. factors. data to predict future trends. 2 Enables prediction and projection 2 Past trends are not always a good on the basis of identified and predictor of future trends. monitored variables and 3 Formula-driven, i.e. could be too assumptions. conservative or initial assumptions might be erroneous. 4 Neglects the implications of chaos and complexity theory, e.g. that long- range forecasting is dangerous. 5 The criteria for prediction may be imprecise. Retrospective 1 Retrospective 1 Useful for establishing causal 1 Remembered information might be relationships. faulty, selective and inaccurate. longitudinal analysis of history 2 Clear focus (e.g. how did this 2 People might forget, suppress or studies of a sample. particular end state or set of fail to remember certain factors. circumstances come to be?). 2 Individual- and 3 Individuals might interpret their 3 Enables data to be assembled that own past behaviour in light of their micro-level data. are not susceptible to experimental subsequent events, i.e. the analysis. interpretations are not contemporaneous with the actual events. 4 The roots and causes of the end state may be multiple, diverse, complex, unidentified and unstraightforward to unravel. 5 Simple causality is unlikely. 6 A cause may be an effect and vice versa. 7 It is difficult to separate real from perceived or putative causes. 8 It is seldom easily falsifiable or confirmable. 354
S u r v e y s , l o n g i t u d i n a l , c r o ss - s e c t i o n a l a n d t r e n d s t u d i e s OO the standardization of wording; an on-the-spot researcher rather than an anonymous OO the inability to catch anything other than a verbal researcher known only through an introductory letter (Robson, 1993). Indeed face-to-face encounters can response; improve response rates. Further, as interviews can be OO the lack of control over the environment in which flexible, questioners are able both to probe and to explain more fully (Bailey, 1994, p. 174). Interviews the survey questionnaire is completed; are also useful when respondents have problems with OO the lack of control over the order in which the ques- reading and writing. Using non-verbal behaviour to encourage respondents to participate is also possible. tions are read and answered; Moreover, with interviews there are greater opportuni- OO the risk that some questions will not be answered; ties to control the environment in which the survey is OO the inability to record spontaneous answers; conducted, particularly in respect of privacy, noise and OO the difficulty in separating non-response from bad external distractions. The effective interviewer, Fowler (2009, p. 128) response, the former being where the intended claims, is business-like and assertive whilst being respondent receives the survey but does not reply to engaging, friendly and kind. Fowler argues for great it, and the latter being where the intended recipient care with choosing interviewers and training them, as does not receive the survey, for example, because much can hang on their behaviour. she/he has moved house; The potential for trust, rapport and cooperation OO the need for simplicity in format as there is no inter- between the interviewer and the respondent is strong in viewer present to guide the respondent through a face-to-face encounters (Dooley, 2001, p. 122; Gwart- more complex format. ney, 2007, p. 16). Further, interviewers can either ensure that the sequence of the survey protocol is Postal surveys are an example of self-administered strictly adhered to or they can tailor the order of surveys. The anonymity and absence of face-to-face responses to individual participants, making certain that interaction between the interviewer and the respondent all questions are answered. Interview surveys, moreo- can render these useful for asking sensitive questions ver, can guarantee that it is the respondent alone who (Strange et al., 2003, p. 337), though Fowler (2009, answers the questions, whereas in postal surveys the p. 74) also counsels that sensitive questions can some- researcher never knows what help or comments are times be handled better in private face-to-face inter- solicited from, or given by, other parties. Bailey (1994) views. In self-administered surveys, Fowler (2009, adds that the opportunity for spontaneous behaviour p. 72) remarks that it is advisable to keep to closed and responses is also possible in interview surveys, and questions and make the response categories simple and interviews can use more complex structures than postal explicit (e.g. ticking a box). If open questions are to be questionnaires, the researcher being on hand to take asked then, he indicates, it is better to gather the survey participants through the items. data in a face-to-face interview. On the other hand, the very features which make Further, Diaz de Rada (2005) reports that the design, interview methods attractive may also make them prob- size and colour of the paper used in postal surveys lematic. For example, interview survey methods may affects response rates. Small-sized questionnaires were be affected by the characteristics of the interviewer (e.g. mostly returned by males and those under sixty-four sex, race, age, ethnicity, personality, skills, perceived years of age (p. 69), whilst larger-s ized questionnaires social status, clothing and appearance). They may also were mostly returned by females and those over the age be affected by the conduct of the interview itself (e.g. of sixty-five (p. 70). He recommends using paper size rapport between the interviewer and the interviewee), 14.85 × 21 cm (i.e. a sheet of A4-sized paper folded in and interviewees may be reluctant to disclose some half ), white paper, and including a cover page (p. 73) information if they feel that the interview will not be (though this inevitably increases the number of pages anonymous or if sensitive information is being in a questionnaire, and this can be off‑putting for requested. The flexibility which the interview gives respondents). He reports that paper size has no effect also contributes to the potential lack of standardization on the quality of the responses. of the interview survey, and this may render consist- ency, and thereby reliability, a problem. Interview surveys Interview surveys are costly in time for the researcher and the interviewee, and, as they are con- Whereas postal surveys are self-administered, interview ducted at a fixed time, they may prevent the interviewee surveys are supervised and hence potentially prone to fewer difficulties. Interview methods of gathering survey data are useful in that the presence of the inter- viewer can help clarify queries from the respondents and can stimulate the respondent to give full answers to 355
Methodologies for educational research from consulting records that may be important to Further, many people are ‘ex-directory’, i.e. their answer the questions. Further, they may require the numbers are withheld from public scrutiny. In addition, interviewer to travel long distances to reach interview- Dooley (2001, p. 123) reports that younger, single and ees, which can be expensive both in time and travel higher occupational status groups use electronic facilities costs (Bailey, 1994, p. 175). If interviews are intended that screen out and delete researchers’ calls and these to be conducted in the participants’ own homes, then could lead to a skewed sample. Indeed Fowler (2009, participants may be unwilling to admit strangers. More- p. 75) indicates that telephone surveys tend to elicit more over, neighbourhoods may be dangerous for some socially desirable answers than face-to-face interviews. researchers to visit (e.g. a white researcher with a clip- Even when the telephone is answered, the person board going into a non-white area of great deprivation, responding may not be the most suitable one to take the or a black researcher going into a conservative white call; she/he may not know the answer to the questions area). or have access to the kind of information required. For example, in an inquiry about household budgets, the Telephone surveys respondent may simply be ignorant about a family’s income or expenditure on particular items. A child may Telephone surveys are located between mailed ques- answer the call, or an elderly person who may not be tionnaires and personal interviews (Arnon and Reichel, the householder. Interviewers will need to prepare a set 2009). Dillman et al. (2014) note the rapid decline in of preliminary screening questions or arrange a call- telephone interviewing (p. 11) with the reduction in back time when a more appropriate person can be landlines, the rise in cellphones, the lack of listing of interviewed. call numbers and the rise in screening callers. However, Telephone interviewing has its own strengths and telephone interviews have the attraction of overcoming weaknesses. For example, more often than not a bias in the researcher or the interviewee that may be respondent’s sex will be clear from their voice, so some caused by social characteristics or matters of age, dress, questions may be unnecessary or inappropriate. On the race, ethnicity, appearance etc. (e.g. Gwartney, 2007, other hand, it is unwise to have several multiple choices p. 16). Indeed Denscombe (2014) suggests that people in a telephone interview, as respondents will simply are ‘more honest and open’ on the phone than in a forget the categories available, there being no written postal questionnaire (p. 12). prompts to which the respondent can refer. Telephone surveys require the interviewer to be an Similarly, order effects can be high: items appearing articulate, clear speaker and a good listener, and able to early in the interview exert an influence on responses to key in interviewee responses onto a computer whilst later ones, whilst items appearing early in a list of listening and speaking (Denscombe, 2014, pp. 42–3). responses may be given greater consideration than They have the advantage of reducing costs in time and those occurring later, a matter not confined to telephone travel, for when a potential respondent is not at home, a surveys but to questionnaires in general. Dooley (2001, call-b ack is cheap and the time to redial is short p. 136) indicates a 17 per cent difference in agreement (Dooley, 2001, p. 122; Arnon and Reichel, 2009, recorded to a general statement question when it p. 179), and, using Internet services such as Skype, tel- appeared before rather than after a specific statement, ephone surveys can be almost free of charge and and other research demonstrates that responses to par- include face-to-face viewing. Revisits to often distant ticular questions are affected by questions surrounding locations, on the other hand, can incur considerable them. His advice is to ask general questions before spe- expense in time and travel. Furthermore, if the intended cific ones, otherwise the general questions are influ- participant is unable or unwilling to respond, then it is enced by earlier responses to specific questions. Once a relatively easy matter to maintain the required sample again, this is a matter not confined to telephone surveys size by calling a replacement. Again, where respond- but to questionnaires in general. ents are unable or unwilling to answer all the questions Further, if the questioning becomes too sensitive, required, then their partial replies may be discarded and respondents may simply hang up in the middle of the further substitutes sought from the sample listing. It is survey interview, tell lies or withhold information. easy to see why telephone interviews must always have Dooley (2001, p. 123) reports that, in comparison to a much longer list of potential respondents in order to face-to-face interviews, telephone respondents tend to attain the required sample size. produce more missing data, to be more evasive, more Not everyone has a telephone (e.g. the poor, the acquiescent (i.e. they tend to agree more with state- young) and this may lead to a skewed sample (Arnon ments) and more extreme in their responses (e.g. opting and Reichel, 2009, p. 179). Nor, for that matter, is every- for the extreme ends of rating scales). one available for interview, particularly if they work. 356
S u r v e y s , l o n g i t u d i n a l , c r o ss - s e c t i o n a l a n d t r e n d s t u d i e s Fowler (2009, pp. 73–4) also indicates that, in a tele- OO the caller’s pronunciation, enunciation and reading phone survey, it is unwise to have too many response out loud; scale points, that it is better to avoid long lists of items and that it is advisable to read the statement before indi- OO the caller’s ability to clarify, summarize, reiterate, cating the response categories, unless a long list of items probe (and when to stop probing), prompt (if the is to be given (i.e. is unavoidable), in which case he sug- receiver does not understand), confirm, affirm, gests that it is better to read and re-read the response cate- respond, give feedback, encourage respondents, gories to the respondent before the list of statements. All keep respondents focused and to the point; of these points take account of the limits of the short-term memories on which respondents often rely in a telephone OO how to conduct closed and open questions, sensi- interview. He also suggests (p. 73) that complex ques- tive, factual and opinion-b ased questions; tions can be approached in a staged manner. For example, if a researcher wishes to ask about a ten-c ategory item OO how to indicate the nature and format of the (e.g. income level of the teacher), then the researcher responses sought; could start with a general question (e.g. above or below a particular figure), and then, once that category has been OO the caller’s ability to handle the called person’s initial identified, proceed to a sub‑category, for example, hostility, refusal, reluctance to take part, feelings of between such-and-such a figure; this avoids overload of invasion of privacy, lack of interest, reluctance to dis- asking a respondent to remember ten categories. close information, feelings of being harassed or Because telephone interviews lack the sensory stim- singled out, anger, antagonism, lack of interest, ulation of visual or face-to-face interviews or written incomplete answers, hurriedness to complete, slow- instructions and presentation, it is unwise to plan a long ness or hesitancy, mistrust, rudeness, abusive telephone survey call. Ten to fifteen minutes is often responses, or simply saying that they are too busy; the maximum time tolerable to most respondents, and indeed fifteen minutes for many people is too long. OO the caller’s ability to remain neutral, impartial and This means that careful piloting must take place in non-judgemental; order to include those items, and only those items, that are necessary for the research. The risk to reliability OO how to record responses; and validity is considerable, as the number of items OO how to end the interview. may be fewer than in other forms of data collection. Procedures for telephone interviews also need to be It is also advisable, in order to avoid the frequent decided (Gwartney, 2007), for example: responses to ‘cold-c alling’ (where the called person simply slams down the telephone), for the interviewer OO how many times to let the telephone ring before to contact the person in advance of the call, perhaps by conceding that there is nobody to answer the call mail, to indicate that the call will come, when, what it (Gwartney (2007, p. 99) suggests eight rings); is about, and to ask for the recipient’s cooperation in the project. OO how to introduce the caller and the project; Many of the features of telephone interviewing are OO what to say and how to introduce items and conduct similar to those of effective interviewing per se, and we advise the reader to consult the comments on interview- the interview; ing earlier and also in Chapter 25. OO how to determine who is receiving the call and 17.12 Comparing methods of data whether he/she is the appropriate person to answer collection in surveys the call; OO whether to leave a message on an answerphone/ Aldridge and Levine (2001, pp. 51–4) and Fowler voicemail/call-back facility and, if so, what that (2009, pp. 80–3) offer useful summary guides to the message will be; advantages and disadvantages of several methods of OO how to handle language problems (e.g. which data collection in surveys: personal face-to-face inter- language is being used, meanings/explanations/ viewing; telephone interviewing; self‑administered/ vocabulary); self-completion versus interviewer-administered; group OO how to handle the situation if the receiver asks to administered; mailed surveys; delivered (distributed) call back later; surveys (e.g. personally delivered or delivered to an OO what to say and how to control the caller’s voice/ institution); Internet surveys. We refer the reader to tone/pitch/speed/pace of questions/repetitions/lan- these useful sources. guage/intonation/register; Additionally, Fowler (2009) and Dillman et al. (2014) discuss the benefits of combining methods of data collection (e.g. face-to-face interviews with telephone interviews, Internet surveys with postal surveys, advance 357
Methodologies for educational research TABLE 17.3 ADVANTAGES AND DISADVANTAGES OF DATA-COLLECTION METHODS IN SURVEYS Advantages Disadvantages Either advantages or disadvantages Postal Time to think Cost: printing, postage Self-completion Interviews Costs may not be too expensive Time: response time and data entry Impersonal face-to-face (individual) Opportunity for attractive survey Low response rates Need for simple format design and graphics Interviews Need for contact details Completion of sensitive (group) Complete at respondent’s information convenience, with opportunity for Risk of superficial coverage of respondent to check topics Can reach many people No checking on understanding or seriousness of response No risk of interviewer intrusion or bias Missing data Can reach scattered populations Respondents may misunderstand instructions or items Can gather sensitive data (nobody else is present) Can offer secure confidentiality, anonymity and non‑traceability Standardized wording Opportunity for gathering in-depth Potential for perceived threat and Location of interviews data bias in face-to-face meeting Participation Reduction of false responses Costly: time for conducting interview, data entry and travel Personal Benefits of human-to-human contact and interpersonal Not possible for large-scale survey Interviewer and behaviour interviewee Need to train interviewers characteristics High response rate Long data-collection period Conduct of interview Useful for exploring complex affects responses issues Access to sample Small samples Opportunity to explain and clarify Little time to think or reflect items and take questions from Standardization respondents Flexibility can reduce standardization Flexibility in item sequence Can build trust and rapport Ensure that only the respondent answers Time-saving (compared to Risk of ‘group think’ Participation individual) Potential for perceived threat and Personal Opportunity for gathering in-depth bias in face-to-face meeting Interviewer and interviewee data Threat to confidentiality characteristics Reduction of false responses Not possible for large-scale Conduct of interview affects responses Benefits of human-to-human survey Small samples contact and interpersonal Scheduling time and location for behaviour whole group to be present Standardization High response rate Costly: time for conducting Useful for exploring complex issues interview, data entry Little time to think 358
S u r v e y s , l o n g i t u d i n a l , c r o ss - s e c t i o n a l a n d t r e n d s t u d i e s Advantages Disadvantages Either advantages or disadvantages Telephone Opportunity to explain and clarify Lack of visuals and non-verbal Sensitive questions: items and take questions from respondents cues: oral and aural medium only absence of an Flexibility in item sequence Finding telephone numbers interviewer may Can build trust and rapport (particularly with cellphones) encourage or Ensure that only the respondent Easy for respondents to refuse or discourage honesty of answers quit through the survey (i.e. to hang response Honesty up) Personal and yet impersonal Anonymity (absence of the human Limited time (no more than ten face) Well-prepared and minutes) Reduction in costs: time, money trained interviewer and travel Cold calling has a bad name Rapid contact Time of day for calling may be inappropriate Random dialling Biased sampling (no telephone) Access to dispersed sample and distant locations Respondents are ex‑directory Response rate higher than postal Immediacy: no time to think of survey responses Short data-collection period Cost (phone charges) Opportunity to explain and clarify items Opportunity to probe participants Reduced interviewer and interviewee bias Personal answering the call may not be suitable Multiple-choice, rating scale and ranking questions are difficult Order effects can be strong Risk of socially desirable responses, satisficing and acquiescence Internet-based Cost saving: time, money, data Security of data and confidentiality Honesty of responses entry by researcher Biased sampling (no Internet, or Impersonal Speed: rapid distribution, respondents’ limited Internet completion and return expertise, volunteer samples) Anonymity, confidentiality and Wide distribution: no problem of No checking on understanding or privacy time and distance seriousness of response Access to minority and Need for email addresses or marginalized groups posting opportunities Opportunity for large samples and Multiple submissions data volume Risk of superficial coverage of Rapid data entry topics continued 359
Methodologies for educational research TABLE 17.3 continued Advantages Disadvantages Either advantages or disadvantages Easy access to people and Computer software compatibility dispersed populations and technical problems Time to think Limited number of items per screen Opportunity to complete it in Respondents give a minimal stages, i.e. with time breaks response Complete at respondent’s Order effects convenience People quit if it is too long or Opportunity for attractive survey complex design and graphics Missing data (or resentment if Higher response rates than postal forced responses are required) surveys Respondents regard it as spam Environmentally friendly (no paper) Design expertise of the researcher Easy skip and branching arrangements Respondents may misunderstand instructions or items Honest responses to sensitive issues Overall low response rates Standardized wording Satisficing and acquiescence (see Chapter 18) Ease of data entry Dropping off Opportunity to explain the survey Costly: distribution staff and time Impersonal questionnaires face-to-face No training required for distribution staff (i.e. no interviews) Respondents have time to think and reflect Complete at respondent’s convenience Higher response rates than postal survey emails with interviews). Single mode surveys, write We include Internet methods in Table 17.3, for ease Dillman et al. (2014), are less effective than mixed mode of comparison with other methods, and our discussion surveys (e.g. telephone calls and emails, emails and web- turns to Internet surveys, devoting the next chapter sites, etc.) in terms of response rates. entirely to this topic. Table 17.3 sets out advantages and disadvantages of these different types of survey administration. Companion Website The companion website to the book includes PowerPoint slides for this chapter, which list the structure of the chapter and then provide a summary of the key points in each of its sections. These resources can be found online at www.routledge.com/cw/cohen. 360
Internet surveys CHAPTER 18 This chapter will look at: al., 2014; Roberts and Allen, 2015). Indeed they are becoming the predominant mode of conducting OO the advantages and disadvantages of Internet surveys surveys, superseding paper‑based surveys, be they OO constructing Internet-b ased surveys through email or websites, on computers, cellphones, OO ethical issues in Internet-b ased surveys tablets and an ever-increasing range of electronic OO sampling in Internet-b ased surveys devices. Though they have much in common with OO improving response rates in Internet surveys paper-b ased surveys, they also have their own particu- OO technological advances lar features, and we comment on these in this chapter. Internet-b ased surveys can operate through, for 18.1 Introduction example: The inclusion here of a chapter devoted to Internet OO an email with an introductory letter and question- surveys signals not only the prominence that these have naire attachment (which requires the researcher to in contemporary research but also raises key issues which know the email address of the recipient); Internet surveys highlight and which may not have such a high profile in more traditional surveys. Internet surveys OO an email directing readers to a website where the bring a new perspective to existing issues in survey survey can be found, or with a hyperlink to that site; research, and we include these here. We advise readers to read this chapter in conjunction with Chapters 8, 17 and OO an email with a survey embedded in it; 24, and very many of the points in these three chapters OO a website which contains the survey: a web-based apply convincingly to the present chapter. Changes in the Internet and its uses are advancing questionnaire; rapidly, with access through a plethora of devices OO a general request for participation, placed in an elec- increasing exponentially. This chapter introduces issues which transcend particular mobile or computer devices tronic environment, such as an advertisement and con- and which can last over time. We recognize immedi- nection to a survey placed on listservs, contact groups ately that, in a chapter of this nature, huge technologi- (e.g. special interest groups, newsgroups, discussion cal changes will have occurred simply between the time groups), blogs, social network sites and forums; of writing and the appearance of this book, and the OO a general advertisement; future in this field has many unpredictable elements. OO a dedicated website (one’s own or others’); The level of sophistication of mobile devices and their OO public messages and advertising on social network- optimization for all kinds of research is advancing at ing sites; breakneck speed. Could we even have imagined five OO companies who provide Internet survey services. years ago that email would become passé so quickly, or three years ago that SMS messaging would become Internet surveys concern: the design of the survey (e.g. yesterday’s news, consigned to the older generation a questionnaire); its distribution and access to it; and rather than the young digital natives of today whose life data collection, storage and accessing data. We discuss seems to revolve around apps and the absence of face- these below. to-face communication (cf. Turkle, 2015)? Internet surveys, whilst they have been the stuff of 18.2 Advantages of Internet surveys student evaluations of teaching for years (Morrison, 2013b), are becoming commonplace in many branches An Internet survey claims several advantages in com- of educational research (Denscombe, 2014; Dillman et parison to a paper survey:1 OO Costs: it reduces costs (e.g. of postage, paper, print- ing, data entry by researchers, telephones, process- ing data, interviewer costs) and increases efficiency. 361
Methodologies for educational research OO Speed: it reduces the time taken to distribute, com- rating scale), i.e. preventing them from continuing plete, gather and process data (data entered onto a until a screen or item is completed: a ‘forced web-based survey can be processed automatically as response’. The computer can check incomplete or soon as the respondent enters the data rather than inconsistent replies. being keyed in later by the researcher), i.e. real-time OO Progress: for each screen, an on-screen progress bar data capture and processing. can show how much of the questionnaire has been completed (e.g. 50 per cent completed, 75 per cent OO Population and samples: wider and much larger completed). populations and samples can be accessed easily, OO Accuracy: fewer missing or incorrect entries, as enabling greater generalizability where Internet human error is reduced in entering and processing users come from a wide, diverse population. Internet online data, these being done automatically. surveys can yield representative data as they can OO Exportability: data can be exported/imported into reach a wider audience. software (e.g. Excel, SPSS) for processing and sub- sequent analysis. OO Contact: e-s urveys overcome spatial and temporal OO Anonymity, honesty and authenticity: respondents constraints (e.g. researchers and participants sepa- may be more honest if their responses are anony- rated from each other in time, location and physical mous and not face-to-face, i.e. a reduction in distance). researcher effects, particularly if sensitive issues are being explored. Because of volunteered participa- OO Volume: a much larger volume of data can be tion (i.e. an absence of coercion), greater authentic- collected. ity of responses may be obtained. OO Access: researchers can reach populations that are With regard to costs, Watt (1997) notes that cost otherwise difficult to reach (e.g. rare, minority, stig- savings make a difference in comparison to a tele- matized, marginalized or deviant cases, as anonym- phone survey, but that an Internet-based survey is only ity and non-traceability might be effective here). slightly less costly than a mail survey unless a web- based survey gathers data from more than around 500 OO Convenience: respondents can complete the survey participants, as the costs in terms of development and at a time to suit themselves, and they can complete design time are considerable (though survey software the survey over time (i.e. they do not need to do it has mitigated this). Fricker and Schonlau (2002) and all at one sitting) and anywhere, i.e. in self‑chosen Fox et al. (2003) suggest that the claims that Internet- and familiar settings. based surveys are cheaper and faster are not always borne out by the evidence, and that, if Internet survey OO Responses: response rates may be higher (though development, programming, testing and modification some evidence challenges this). Participants tend to time, initial contact time and follow-u p time to ensure respond more quickly and more fully, reflectively an increased response rate are factored in, then the and incisively, particularly when e‑reminders are savings may not be as strong as the claims made. More sent. Data are of a higher quality (richer, fuller recently, many Internet surveys have become less responses with greater depth and reflection by par- costly but not in terms of time for survey design and ticipants). There are higher item completion and development. response, higher item variability and fewer missing values (though some evidence challenges this). 18.3 Disadvantages of Internet surveys OO Ease: it is easy to enter responses (e.g. click a radio button, check/tick a box). An Internet survey also has several potential and actual disadvantages in comparison with a conventional OO Environment: it is environmentally friendly (little or survey (here the same references as those given above no paper). for ‘advantages of Internet surveys’ also give disadvan- tages, see note 1), for example: OO Attractiveness: graphics, different colours and fonts can be used. OO Spam: recipients and servers may regard the request to participate as spam or junk mail, and ignore or OO Design flexibility: skip-patterns and branching can delete it. be created and organized by the computer, so that participants do not have to understand complicated instructions, with automated, flexible navigation through the questionnaire consequent to the answers from respondents and boxes that they check/tick. It can enable diverse questions to be asked (through branching and skip functions). OO Response checking: the software can prompt respondents to complete missed/skipped items or to correct errors (e.g. two ticks for a single item in a 362
Internet surveys OO Expertise: respondents may not have sufficient OO Time: surveys embedded in emails may require sub- Internet expertise to complete it correctly. sequent manual data entry by the researcher. OO Sampling: some target groups may not have access OO Response rates: these tend to be low in Internet to the Internet and some respondents/groups may surveys (calculated in relation to those who accessed not be available or may not answer (non-response), the survey). Response rates drop off if the survey is particularly if the questions are sensitive. There may long or too complicated. be sampling bias/skew, as the researcher has less control over the population and sampling, and OO Misreporting: there may be fake and false reporting: respondents self-select. Random sampling may be respondents may give deliberately fake, incorrect or impossible (which may affect statistics used, i.e. socially desirable responses, or complete the survey preventing use of some parametric statistics). Per- to obtain a promised reward. There may be multiple sonal details may be withheld or unverifiable, hence submissions from the same person. Verifying the sub-sampling and parametric analysis may be identity or details of participants may be impossible. impossible. OO Satisficing: respondents, particularly in a long OO Abandonment and dropout: it is easy for respond- survey, will enter any response rather than no ents to simply stop altogether, or to send incomplete response, compromising the quality, accuracy and surveys, or not to send them at all. This is particu- reliability of the response (discussed below). The larly the case if surveys are long or if researchers try quality of responses becomes increasingly question- to coerce respondents into completion by the use of able as a long survey progresses. a forced response (making them unable to continue until every item has been completed on a single Several of the positive claims made for Internet surveys screen). As a general rule, Internet surveys should are questionable. For example, Fricker and Schonlau be shorter than paper-b ased surveys or non-response (2002) and Fox et al. (2003) note that, in comparison to can occur. conventional surveys: (a) response rates may not be higher; (b) time taken in the several stages of the survey OO Computer difficulties: the different configuration – from design to data collection and analysis – may not and software of computers may affect access to the be shorter, even though delivery time is shorter; (c) survey, its layout, presentation and speed of connec- data quality may be no better: coverage of items, sam- tion. The computer or server may ‘crash’, ‘freeze’ pling, response and non‑response, honesty; and (d) or refuse to work, particularly if the survey is long potential respondents may not have access to the Inter- or contains a lot of graphics. Error messages may net and may be computer illiterate. appear. The survey may require software that respondents do not have or which downloads very 18.4 Constructing Internet-based slowly. Servers, computer networks, connections or surveys the ‘system’ may go down. Surveys embedded in emails should be short, with few or no graphics or Much Internet survey design can use any of the many embedded objects, as the recipients’ computer may online Internet survey templates and (often free) serv- not receive these. ices available, and the companion website provides websites which provide such services. At the time of OO No interviewer: this is an impersonal medium, with writing, widely used providers include: SurveyMonkey; no opportunities for in‑depth probes. Zoomerang; Free Online Surveys; SmartSurvey; Survey Planet; Google Forms. We provide the websites of OO Security, privacy and confidentiality: these are not these and others on the companion website. Several of absolute, as hackers, data miners and spammers may these also automatically collate and present results so intercept and track participants, and results are that they can be downloaded into, for example, Excel, stored by software providers. Emails are particularly SPSS, SAS or STATA. vulnerable. For researchers wishing to construct their own Inter- net surveys with software and templates, there are OO Design matters: technical expertise is needed if the several points of guidance below. Internet surveys can researcher is going to write and design her/his own be made more attractive than their paper-b ased coun- survey, i.e. not using commercially produced, open terparts, with graphics, fonts, colours etc., but there are access or free software. cautions, and we indicate these below. Dillman et al. (1998a, 1998b, 1999, 2014) set out OO Instructions and answering: if instructions are misun- several concerns for Internet surveys, some technical derstood, unclear, too complicated or too many, this may lead to irrelevant, unreliable or no responses, and if respondents are unclear how to answer then they are likely to simply skip items or stop. 363
Methodologies for educational research and some presentational. In terms of technical matters, They also suggest following the natural movement of they found that the difference between simple and the eyes from the top-left (the most important part of ‘fancy’ versions of questionnaires (the former with few the screen, hence the part in which the question is graphics, the latter with many, using sophisticated soft- located) to the bottom-right quadrants of the screen (the ware) could be as much as three times the size of the least important part of the screen which might contain file to be downloaded, thereby increasing download the researcher’s logo). They comment that the natural time. Respondents with slow browsers or limited power movement of the eye is to read prose unevenly (e.g. either spent longer in downloading the file or, indeed, saccadically), with the risk of missing critical words, the machine crashed before the file was downloaded. particularly on long lines; hence they advocate keeping Download speeds continue to vary in different parts of lines and sentences short. It is also useful to include a the world and at different times of the day. They also progress bar to indicate how much of the questionnaire found that recipients of plain versions were more likely has been completed. to complete a questionnaire than those receiving fancy Some respondents may have less developed compu- versions, as it took less time to complete the plain ter skills than others, and may not be familiar with version. Utilizing advanced page layout features does web-b ased questionnaires, for example, with radio not translate into higher completion rates, indeed more buttons, scroll bars, drop‑down menus, where to insert advanced page layout reduced completion rates. Simi- open-e nded responses. Hence the survey designer must larly, Fricker and Schonlau (2002) report studies indi- not overestimate the capability of the respondent to use cating a 43 per cent response rate to an email survey the software. Indeed explanations on how to respond compared to a 71 per cent response rate for the same may have to be outlined in the survey itself. mailed paper questionnaire, and that higher response Dillman et al. (1999, 2014) suggest that the problem rates in an Internet survey are typically only obtained of differential expertise in computer usage can be from specialized samples (e.g. undergraduates). addressed in three ways: For presentational matters, Dillman et al. (1998a, 1999, 2014) comment that in a paper‑based survey 1 Place the instructions for how to complete the item the eyes and the hands focus on the same area, whilst next to the item itself (not all placed together at the in an Internet survey the eyes focus on the screen start of the questionnaire). whilst the hands often focus on the keyboard or the mouse, rendering completion more difficult. This is 2 Ask the respondents at the beginning about their one reason to avoid asking respondents to type in level of computer expertise; if they are more expert, many responses to open-e nded questions, and to offer them the questionnaire with certain instruc- replace this with radio buttons or clicking a check box. tions omitted, and if they are less experienced, direct The researchers also found that ‘check-a ll-that-a pply’ them to instructions and further assistance. lists of factors had questionable reliability, as respond- ents tended to complete those items at the top of the 3 Have a minimized ‘floating window’ that accompa- list and ignore the rest. Hence they recommend avoid- nies each screen and which can be maximized to ing the use of large check-all-that-apply lists in a web- give further instructions. based survey. It is important to keep the introduction to the ques- Some web-based surveys prevent respondents from tionnaire short (no more than one screen) and informa- proceeding until they have completed all the items on tive (e.g. about how to move on) and to avoid giving a the screen (a forced response). Whilst this might ensure long list of instructions. Further, as the first question in coverage, it can also anger respondents – such that they a survey tends to raise in respondents’ minds a particu- give up and abandon the survey. Some web-b ased lar mindset, care is needed in setting the first question, surveys prevent respondents from having a deliberate to entice participants and not to put them off participat- non‑response (e.g. if they do not wish to reveal particu- ing (e.g. not too difficult, not too easy, interesting, lar information, or if, in fact, the question does not straightforward to complete, avoiding drop-down boxes apply to them, or if they do not know the answer). The and scrolling). Dillman et al. (1998a, 1998b, 1999, advice of Dillman et al. (1999, 2014) is generally to 2014) make recommendations about the layout of the avoid forced responses and to give respondents catego- screen, for example, keeping the response categories ries of ‘prefer not to answer’/‘decline to answer’, ‘don’t close to the question for ease of following; using fea- know’, ‘not applicable’ and ‘other’. It is much easier tures like brightness, large fonts and spacing for clarity; for participants in a web-based survey to abandon the and avoiding too many changes of font and font size. survey – a simple click of a button – so more attention has to be given to keeping them participating than in a paper-b ased survey. 364
Internet surveys The location of the instruction (e.g. to the right of systems, screen displays (e.g. partial and wrap- the item, underneath the item, to the right of the answer around text) and browsers. box) is important. Locating the instruction too far to the OO Enable the survey to run on computers, cellphones, right of the answer box (e.g. more than nine characters iPads and other different devices. of text to the right) can mean that it is outside the foveal OO Ensure that security, confidentiality and privacy are view (two degrees) of the respondent’s vision, and in place. hence can be overlooked. Redline et al. (2002) advo- OO Start the web questionnaire with an interesting cate making an instruction easier to detect by locating welcome screen that motivates respondents to con- it within the natural field of vision of the reader, setting tinue, makes it clear that it is easy to complete, gives it in a large font to make it bolder and using a different clear instructions on how to proceed and contains colour. If the researcher wishes to include skips and information and a check box for informed consent. branches then this can be done automatically ‘behind OO Provide a PIN number if you wish to limit access to the scenes’, i.e. if the participant gives a particular those people in the sample. response then the computer automatically takes him or OO Ensure that the first question can be seen in its her past a skipped part or to a branching part. entirety on the first screen, and is easy to understand Redline et al. (2002) identify many other variables and complete. that impact on the success of Internet surveys: OO Embed visual images in a survey and place interest- based questions early, as this reduces premature OO the greater the number of words, the more the reader dropout from the survey. will be absorbed with the question than with the OO Ensure that the layout of each question is similar to instructions; a paper format, as respondents may be familiar with this. OO if there are more than seven response categories per OO Ensure that the use of colour keeps figure/ground item the reader may make errors; consistency and readability, so that it is easy to navi- gate through the questionnaire with navigational OO response-order effects: respondents in a self- flow unimpeded, and so that the measurements used administered survey tend to choose earlier rather in questions are clear and sustained. than later items in a list (the primacy effect); OO Ensure consistency in colours, fonts, layout. OO Keep the line length short, to fit screen size and OO if respondents are asked to write an open-ended respondent focus. response they may overlook instructions, as they are OO Minimize the use of drop-d own boxes and direct absorbed in composing their own response, and the respondents to them where they occur. instruction may be out of their field of vision when OO Give clear instructions for how to move through the writing their answer; questionnaire using the computer, and keep instruc- tions for computer actions at the point where the OO items located at the bottom of a page are more likely action is needed, rather than placing them all at the to elicit a non-response than items further up a page, start of the questionnaire. and instructions near the bottom of a page are more OO Avoid forced responses (requiring respondents to likely to be overlooked, so avoid this; answer each question before being able to move on to the next question/screen). OO if instructions are located too far from the answer OO Ensure that questionnaires scroll easily from ques- box they may be overlooked. tion to question, unless order effects are important. OO If multiple choices are presented, keep them to a This advice applies not only to online survey question- single screen; if this is not possible then consider naires but also to paper-b ased surveys. double columns, with navigational instructions. There are several ‘principles’ for designing web- OO Consider providing a progress bar to indicate how based questionnaires (e.g. Schaefer and Dillman, 1998; far the respondent has reached in the survey (it may Dillman et al., 1999; Dillman and Bowker, 2000, or may not be advisable if the questionnaire is quite pp. 10–11; Shropshire et al., 2009; Dillman et al., 2014): long, hence judgement is required). OO Avoid tick-a ll-those-that-apply kinds of question. OO Consider the capabilities and configurations of OO Enable respondents to save their survey and com- the respondents’ computers and the respondents plete it later, and to keep their own copy (backup) of themselves. their completed survey. OO Ensure that the layout/presentation of the survey will be the same across platforms, servers, browsers and respondents, and avoid differences in the visual appearance of questions that may happen as a result of different computers, configurations, operating 365
Methodologies for educational research OO Have a ‘thank you’ screen for when the respondent Toepel et al. (2009) suggest that account has to be submits the completed survey. taken of the ‘cognitive sophistication’ of the respond- ents, as those with less cognitive sophistication tend to OO Provide a ‘help’ button for further explanation and be affected by contextual clues more than those with contact details of the researcher. more cognitive sophistication. Context effects also occur when a particular item is affected by the items Additionally, Heerwegh et al. (2005) report that per- around it or which precede it, in effect providing cues sonalizing the survey (i.e. using the recipient’s name in for the respondent, or in which a particular mindset of the salutation) increases response rates by 8.6 per cent responses is created in the respondent (Friedman and (e.g. starting a survey letter with Dear [name of specific Amoo, 1999). person]), though care has to be taken to ensure consist- The importance of the visual aspect of question- ency, for example, it is counter-productive to start an naires is heightened in Internet surveys (Smyth et al., Internet survey letter with Dear [name of person] and 2004), and this concerns the layout of questions, then, later, to refer to ‘student’, ‘colleague’ etc. (i.e. a instructions and response lists, the grouping of items, depersonalized version). Personalizing a survey the colours used, the spacing of response categories and increases the chances of a respondent starting the the formatting of responses (e.g. writing in words or survey (p. 92) rather than dropping out during the checking boxes). Smyth et al. report that respondents survey (i.e. they might still drop out later). Whether use ‘preattentive processing’ when approaching Inter- participants continue and complete the survey depends net surveys, i.e. they try to take in and understand the on other factors, for example: difficulty in completing whole scene (or screen) before attending to specific the items; relevance of the topic to the respondents; and items, hence visual features are important, for example, user-friendliness of the survey. They also report (p. 94) emboldened words, large fonts, colours, brightness, that personalizing an e-survey increases participant section headings, spacing, placing boxes around items. honesty on sensitive matters (e.g. number of sexual This rests on Gestalt psychology which abides by the partners), increases adherence to survey instructions principles of: (a) proximity (we tend to group together (p. 96) and increases the tendency of respondents to those items that are physically close to each other); (b) answer questions in a socially desirable way. similarity (we tend to group together those items that Denscombe (2009b, pp. 286–7) reports that, for appear alike); (c) prägnanz (figures or items with sim- online surveys, fixed-choice questions tend to have a plicity, regularity and symmetry are more easily per- ‘lower item-response rate than open-ended questions’, ceived and remembered). and, for open‑ended questions, item non-response is Smyth et al. (2004) also suggest (p. 21) that head- lower in online surveys than in paper‑based surveys. ings and separation of sections take on added signifi- Christian et al. (2009) report that if a survey ques- cance in Internet-b ased surveys. Separating items into tionnaire presents the positive end of a scale first then, two sections with headings had a ‘dramatic effect’ on whilst it may not make a difference to the response, it responses, as respondents felt compelled to answer both does increase response time (which might lead to drop- sub-groups (70 per cent gave an answer in both sub- ping out). They also note that if the positive categories groups whereas only 41 per cent did so when there are given lower numbers (e.g. ‘1’ and ‘2’) and the neg- were no headings or sectionalization). They also found ative categories are given high numbers (e.g. ‘4’ and that separating a vertical list of items into sub-g roups ‘5’), then this can increase response time. Further, they and double columns should be avoided. They report report that displaying categories in several columns that asking for open-ended responses (e.g. writing their increases response time, as does giving the poles of subject specialisms) can be more efficient than having rating scales numbers rather than words. them track down a long list of subjects (e.g. from a Mora (2011b) notes that more positive results are drop-down menu) to find the one that applies to them, obtained if the scale runs from –2 to +2 or from –4 to though this can be mitigated by placing simple lists in +4, that scales with the positive label on the left and alphabetical order. Finally, they found that placing very higher numerical ratings on the left had significantly short guides underneath the write-in box rather than at higher ratings, that scales with ‘definitely agree’ on the its side (e.g. dd/mm/yy for ‘day/month/year’) increased left and ‘definitely disagree’ on the right tended to have response rates, and that placing instructions very close higher agreements, and higher scores were recorded for to the answer box improved response rates. positively worded items. The percentage of affirmative Dillman et al. (2003) also found that having responses was higher in a paper-based survey than in respondents use a ‘yes/no’ format and having a ‘forced an Internet-b ased survey (Dillman et al., 2003; Morri- choice’ (no option but to answer) for responding son, 2013b). 366
Internet surveys resulted in increased numbers of affirmative answers, 18.5 Ethical issues in Internet-b ased even though this requires more cognitive processing surveys than non-forced choice questions (e.g. ‘tick[check]-all- that-apply’ questions) (p. 23). This is because respond- In Internet surveys, issues of informed consent, ano- ents may not wish to answer questions in the outright nymity, privacy and confidentiality, non-traceability, negative (p. 10). Even if they do not really have an protection from harm, the precautionary principle and opinion, or they are neutral, or the item does not really data security are important (e.g. Fox et al., 2003; Ham- apply to them, they may choose a ‘yes’ rather than a mersley and Traianou, 2012). Chapter 8 addressed ‘no’ category. They may leave a blank rather than indi- ethical issues in online research and these apply to cating a ‘no’. Internet surveys. Ethics in Internet surveys also con- Dillman et al. (2003) report that respondents tend to cerns the offering of incentives to participate, to ensure select items higher up a list than lower down a list of that they are appropriate and not excessive. Marshall options (the primacy effect), and opt for the ‘satisfic- and Rossman (2016) suggest that researchers must con- ing’ principle (they are satisfied with a minimum suffi- sider several ethical issues in researching online com- cient response, selecting the first reasonable response in munities (pp. 182–3): a list and then moving on rather than working their way down the list to find the optimal response), i.e. item OO how public and private the data are; order is a significant feature, making a difference to OO the sensitivity of the topic; responses of over 39 per cent (p. 7). This is particularly OO the degree of interaction between the researcher and so when respondents are asked for opinions and beliefs rather than topics seeking factual information. They the participants and between participants; also suggest that the more difficult the item is, the more OO the vulnerability of the participants. respondents will move towards ‘satisficing’. ‘Satisfic- ing’ and the primacy effect were stronger in Internet Given that the URL identifies a specific location and is surveys than paper-based surveys (p. 22), and changing held on servers, and given that some survey providers ‘check-a ll-that-a pply’ to forced responses (‘yes/no’) have control over this, it is impossible to guarantee did not eliminate response order effects. Similarly, Diaz total security of identity and information here. Rather, de Rada and Dominguez (2015) report that participants steps have to be taken to protect data in terms of collec- change their minds about participation as the survey tion, storage, access and electronic transfer (just as with proceeds, but, rather than withdrawing altogether, move non‑Internet surveys). towards satisficing, giving answers that require To address security and confidentiality of identity, minimum effort rather than thinking deeply, and that the researcher may require no identification details they give affirmative answers in the spirit of acquies- from the participant and ask for none in the survey. Or cence or ‘don’t know’ answers. the researcher may take steps to protect data privacy, Dillman et al. (2003) report that the order of anonymity and confidentiality, through password pro- response categories can have an effect on responses, tection, one-w ay scrambling of the machine number citing a study that found that asking college students when the data are being submitted (‘encoding it in whether their male or female teachers were more empa- a manner that cannot subsequently be decoded’; Fox thetic was affected by whether the ‘male’ option was et al., 2003, p. 178). However, this does not give placed before or after the ‘female’ option: female teach- any absolute guarantee that hackers, data miners or ers were evaluated more positively when respondents spammers will not intercept and access the data, were asked to compare them to male teachers than particularly where it can be linked to an email address. when male teachers were compared to female teachers An email survey can reveal the identity and trace (p. 6). Respondents compare the second item in light of ability of the respondent. Security (e.g. through pass- the first item in a list rather than considering the items words and PIN numbers) is one possible solution, independently. though this, too, can create problems as respondents Internet-b ased surveys, then, raise several chal- may feel that they are being identified and tracked, lenges and problems. Some of these are indicated in and indeed some surveys may deposit unwelcome Table 18.1, together with possible solutions. ‘cookies’ onto the respondent’s computer, for future contact. Completing a survey may be taken to indicate informed consent, and Roberts and Allen (2015) note the American Educational Research Association’s (2000) Code of Ethics which indicates (Clause 13.01.b) 367
Methodologies for educational research TABLE 18.1 PROBLEMS AND SOLUTIONS IN INTERNET-BASED SURVEYS Problem (sampling) Possible solution Some subsample groups may be under-represented in Adjust the results by weighting the sample responses the respondents (see the comments on a ‘boosted sample’ and ‘weighting’ in Chapter 12) There may be coverage error (not everyone has a non- zero chance of being included) Disclose the sample characteristics in reporting Non-response and volunteer bias Follow-up messages posted on websites and electronic discussion groups. Use emails to contact potential participants. Require the respondents to submit their replies screen by screen. (This enables the researcher not only to use some data from incomplete responses, but also enables her to identify in detail patterns of non- response, i.e. responding is not an all-or-nothing affair (either submit the whole questionnaire or none of it) but can be partial (a respondent may answer some questions but not others)) Problem (ethics) Possible solution Respondents may wish to keep their identity from the Direct respondents to a website rather than to using researcher, and an email address identifies the email correspondence. Provide advice on using non- respondent (in the case of sensitive research, e.g. on traceable connections to access and return the survey child abuse or drug abuse, this may involve criminal (e.g. an internet café, a library, a university). Advise the proceedings if the identity of the respondent is known or respondent to print off the survey and return it by post to able to be tracked by criminal investigators who break a given address. Avoid asking respondents to enter a into the site). Non-traceability of respondents may be password or to give an email address. Prevent access to problematic unprotected directories and confidential data Respondents may not know anything about the Include the researcher’s affiliation (e.g. university), with a researcher, or if it is a bona fide piece of research and logo if possible not simply a marketing ploy Ensure that it is easy for respondents to withdraw at any Informed consent time (e.g. include a ‘Withdraw’ button at the foot of each screen) Problem (technical: hardware and software) Possible solution The configuration of the questionnaire may vary from one Opt for simplicity. Test the survey on different computer machine to another (because of web browsers, systems/browsers to ensure consistency. Avoid surveys connection, hardware, software) and can lead to dropout that require real-time completion The screen as set out by the survey designer may not Opt for simplicity. Use a commercial survey software appear the same as that which appears on the system for generating the questionnaire. Avoid high level respondent’s screen programs Slow network connections or limited bandwidth can slow Keep the use of graphics to a minimum. Advise on the down loading possible time it takes to load Respondents may not have the same software, or the Avoid the use of graphics and more advanced software same version of the software as the sender, rendering programs downloading of the questionnaire either impossible or distorting the received graphics 368
Internet surveys Problem (technical: hardware and software) Possible solution Graphics may be corrupted/incompatible between the Opt for simplicity. Use commercially available web-based sender and the user, i.e. between one kind of machine, surveying systems and packages. Use image files (e.g. user platform and software and another. Hardware may .jpeg, .gif) to reduce loading time. Avoid pop-ups if differ between sender and receiver possible as they reduce response rate The greater the use of graphics and plug-ins (e.g. using Keep software requirements as low-tech as possible. Java and Applets), the longer it takes to download, and, Avoid questionnaires that use sophisticated computer particularly – though not exclusively – if respondents do graphics not have broadband access then time-consuming downloads could result in either the respondent giving up and cancelling the download, or creating a bad mood in the respondent There may be slow loading times due to internet Avoid sophisticated graphics and ‘fancy’ presentations congestion as these take longer to download The physical distance between points on an attitude Indicate how best the questionnaire may be viewed (e.g. scale may spread out because of configuration 800 × 400) differences between machines The construction procedures for wrap-around text may Keep lines of text short vary between computers Email questionnaires may distort the layout of the Avoid sending a questionnaire directly using email; questionnaire (some email software uses HTML, others rather, post it on a website (e.g. so that respondents visit do not) a website and then click a box for immediate transfer to the questionnaire). Consider using an email to direct participants to a website (e.g. the email includes the website which can be reached by clicking in the address contained in the email). Use an email that includes an attachment which contains the more graphically sophisticated survey instrument itself Problem (respondents) Possible solution Respondents may be unfamiliar or inexperienced with the Keep the questionnaire simple and easy to complete internet and the media Respondents may send multiple copies of their Have a security device that tracks and limits (as far as completed questionnaire from the same or different possible) respondents who may be returning the same addresses questionnaire on more than one occasion. Use passwords (though this, itself, may create problems of identifiability). Collect personal identification items. Check for internal consistency across submissions There may be more than one respondent to a single Include questions to cross-check the consistency of questionnaire (the same problem as in, for example, a replies to similar items postal questionnaire) Respondents may not be used to pull-down menus Provide clear instructions Drop-down boxes take up more space on a screen than Avoid their overuse conventional questionnaires Respondents dislike the situation where the computer Avoid this unless considered absolutely necessary prevents them from continuing to the next screen until all continued the items on a particular screen have been completed 369
Methodologies for educational research TABLE 18.1 continued Possible solution Problem (respondents) The language of email surveys can risk offending Check the language used to avoid angering the potential participants (‘flaming’) participants Respondents’ difficulty in navigating the pages of the Keep instructions to the page in question. Make the online survey instructions for branching very clear (font size, colour, etc.) Problem (layout and presentation) Possible solution A page of paper is longer than it is wide, but a screen is Remember that screen-based surveys take a greater wider than it is long, and a screen is smaller than a page, number of screens than their equivalent number of pages i.e. layout becomes a matter of concern in a paper copy. Sectionalize the questionnaire so that each section fills the screen, and does not take more than one screen The layout of the text and instructions assumes greater Opt for clarity and simplicity importance than for paper questionnaires The layout uses a lot of grids and matrices Avoid grids and matrices: they are a major source of non- response The order of items affects response rates Locate requests for personal information at the beginning of the survey. Include ‘warm-ups’ and early ‘high hurdles’ to avoid dropout Respondents may be bombarded with too much Place the advertisement for the survey on user groups as information in an introductory message well as for the general public, inviting participants to contact such-and-such a person or website for further information and the questionnaire itself, i.e. separate the questionnaire from the advertisement for/introduction to the questionnaire Respondents may be overloaded with instructions at the Avoid placing all the instructions at the start of the beginning of the survey questionnaire, but keep specific instructions for specific questions Respondents may be overloaded with information at the Keep the initial information brief and embed further beginning of the survey information deeper in the survey Respondents may have to take multiple actions in order Keep the number of actions required in order to move on to answer each question (e.g. clicking on an answer, to a minimum moving the scroll bar, clicking for the next screen, clicking to submit a screen of information) Respondents may not be able to see all the option Ensure that the whole item and options are contained on choices without scrolling down the screen a single screen Respondents may not understand instructions Provide a helpline, email address or contact details of the researcher. Pilot the instrument Instructions about options may be unclear Use radio buttons for single choice items, and try to keep layout similar to a paper layout Respondents only read part of each question before Keep instructions and words to a necessary minimum going to the response category 370
Internet surveys Problem (reliability) Possible solution Respondents may alter the instrument itself. The Include technological safeguards to prevent alteration researcher relinquishes a greater amount of control to the and have procedures to identify altered instruments respondents than in conventional questionnaires Respondents may be forced to answer every question Pilot the survey. Include options such as ‘don’t know’ and even when they consider some response categories ‘do not wish to answer’ and avoid forcing respondents to inappropriate reply before they can move on Respondents may not be telling the truth – they may Include questions to cross-check replies (to try to reduce misrepresent themselves the problem of respondents not telling the truth) Problem (dropout) Possible solution Respondents may lose interest after a while and abandon Have a device that requires respondents to send their the survey, thereby losing all the survey data replies screen by screen (e.g. a ‘Submit’ button at the foot of each screen), section by section, or item by item. Put each question or each section on a separate screen, with ‘submit’ at the end of each screen. Adopt a ‘one- item-one-screen’ technique Respondents may not know how long the questionnaire Include a device for indicating how far through the is, and so may lose interest questionnaire the respondent has reached: a progress bar at the bottom or the side of the survey Internet surveys take longer to complete than paper- Keep the internet survey as short, clear and easy to based surveys complete as possible People do not want to take part, and it is easier for Increase incentives to participate (e.g. financial someone to quit or cancel an internet-based survey than incentives, lottery tickets (if they are permitted in the a paper-based survey (simply a click of a button) country)) Diminishing returns (the survey response drops off quite Ensure that the website is re-posted each week during quickly). Newsgroup postings and electronic discussion the data collection period group data are removed, relegated or archived after a period of time (e.g. a week), and readers do not read lower down the lists of postings Non-participation may be high (i.e. potential participants Increase incentives to participate. Locate personal may not choose to start, in contrast to those who start informational questions at the start of the survey and who subsequently drop out) Error messages (e.g. if an item has not been completed) Avoid error messages if possible, but, if not possible, cause frustration and may cause respondents to provide clear reasons why the error was made and how abandon the questionnaire to rectify it that waivers of consent may apply to Internet surveys if Researchers can provide initial information about there is minimal risk or if the obtaining of consent may ethical matters at the front of the survey, with rights: be impractical. Nevertheless this does not exonerate the (a) to withdraw; (b) not to answer specific questions; researcher from indicating to participants that, whilst and (c) to withdraw their consent freely. Researchers every reasonable step has been taken to protect confi- can provide a check box for participants to give dentiality and to prevent unwanted access, it cannot be a consent, and indicate to what they are giving consent. watertight guarantee, even if this means that some Such consent, Roberts and Allen (2015) aver, should be respondents decline to participate. Transparency and free of coercion, and care should be taken to ensure that honesty trump the researcher’s personal wishes for this is the case for potentially vulnerable participants responses here. (e.g. those with special needs, minorities, children etc.). 371
Search
Read the Text Version
- 1
- 2
- 3
- 4
- 5
- 6
- 7
- 8
- 9
- 10
- 11
- 12
- 13
- 14
- 15
- 16
- 17
- 18
- 19
- 20
- 21
- 22
- 23
- 24
- 25
- 26
- 27
- 28
- 29
- 30
- 31
- 32
- 33
- 34
- 35
- 36
- 37
- 38
- 39
- 40
- 41
- 42
- 43
- 44
- 45
- 46
- 47
- 48
- 49
- 50
- 51
- 52
- 53
- 54
- 55
- 56
- 57
- 58
- 59
- 60
- 61
- 62
- 63
- 64
- 65
- 66
- 67
- 68
- 69
- 70
- 71
- 72
- 73
- 74
- 75
- 76
- 77
- 78
- 79
- 80
- 81
- 82
- 83
- 84
- 85
- 86
- 87
- 88
- 89
- 90
- 91
- 92
- 93
- 94
- 95
- 96
- 97
- 98
- 99
- 100
- 101
- 102
- 103
- 104
- 105
- 106
- 107
- 108
- 109
- 110
- 111
- 112
- 113
- 114
- 115
- 116
- 117
- 118
- 119
- 120
- 121
- 122
- 123
- 124
- 125
- 126
- 127
- 128
- 129
- 130
- 131
- 132
- 133
- 134
- 135
- 136
- 137
- 138
- 139
- 140
- 141
- 142
- 143
- 144
- 145
- 146
- 147
- 148
- 149
- 150
- 151
- 152
- 153
- 154
- 155
- 156
- 157
- 158
- 159
- 160
- 161
- 162
- 163
- 164
- 165
- 166
- 167
- 168
- 169
- 170
- 171
- 172
- 173
- 174
- 175
- 176
- 177
- 178
- 179
- 180
- 181
- 182
- 183
- 184
- 185
- 186
- 187
- 188
- 189
- 190
- 191
- 192
- 193
- 194
- 195
- 196
- 197
- 198
- 199
- 200
- 201
- 202
- 203
- 204
- 205
- 206
- 207
- 208
- 209
- 210
- 211
- 212
- 213
- 214
- 215
- 216
- 217
- 218
- 219
- 220
- 221
- 222
- 223
- 224
- 225
- 226
- 227
- 228
- 229
- 230
- 231
- 232
- 233
- 234
- 235
- 236
- 237
- 238
- 239
- 240
- 241
- 242
- 243
- 244
- 245
- 246
- 247
- 248
- 249
- 250
- 251
- 252
- 253
- 254
- 255
- 256
- 257
- 258
- 259
- 260
- 261
- 262
- 263
- 264
- 265
- 266
- 267
- 268
- 269
- 270
- 271
- 272
- 273
- 274
- 275
- 276
- 277
- 278
- 279
- 280
- 281
- 282
- 283
- 284
- 285
- 286
- 287
- 288
- 289
- 290
- 291
- 292
- 293
- 294
- 295
- 296
- 297
- 298
- 299
- 300
- 301
- 302
- 303
- 304
- 305
- 306
- 307
- 308
- 309
- 310
- 311
- 312
- 313
- 314
- 315
- 316
- 317
- 318
- 319
- 320
- 321
- 322
- 323
- 324
- 325
- 326
- 327
- 328
- 329
- 330
- 331
- 332
- 333
- 334
- 335
- 336
- 337
- 338
- 339
- 340
- 341
- 342
- 343
- 344
- 345
- 346
- 347
- 348
- 349
- 350
- 351
- 352
- 353
- 354
- 355
- 356
- 357
- 358
- 359
- 360
- 361
- 362
- 363
- 364
- 365
- 366
- 367
- 368
- 369
- 370
- 371
- 372
- 373
- 374
- 375
- 376
- 377
- 378
- 379
- 380
- 381
- 382
- 383
- 384
- 385
- 386
- 387
- 388
- 389
- 390
- 391
- 392
- 393
- 394
- 395
- 396
- 397
- 398
- 399
- 400
- 401
- 402
- 403
- 404
- 405
- 406
- 407
- 408
- 409
- 410
- 411
- 412
- 413
- 414
- 415
- 416
- 417
- 418
- 419
- 420
- 421
- 422
- 423
- 424
- 425
- 426
- 427
- 428
- 429
- 430
- 431
- 432
- 433
- 434
- 435
- 436
- 437
- 438
- 439
- 440
- 441
- 442
- 443
- 444
- 445
- 446
- 447
- 448
- 449
- 450
- 451
- 452
- 453
- 454
- 455
- 456
- 457
- 458
- 459
- 460
- 461
- 462
- 463
- 464
- 465
- 466
- 467
- 468
- 469
- 470
- 471
- 472
- 473
- 474
- 475
- 476
- 477
- 478
- 479
- 480
- 481
- 482
- 483
- 484
- 485
- 486
- 487
- 488
- 489
- 490
- 491
- 492
- 493
- 494
- 495
- 496
- 497
- 1 - 50
- 51 - 100
- 101 - 150
- 151 - 200
- 201 - 250
- 251 - 300
- 301 - 350
- 351 - 400
- 401 - 450
- 451 - 497
Pages:
