-Earl_Babbie-_The_Practice_of_Social_Research(BookFi)

Selecting Subjects 225 medical case, those researchers who were respon- defect in social research. Simply put, college under- sible for administering the drug and for noting im- graduates are not typical of the public at large. There provements would not be told which subjects were is a danger, therefore, that we may learn much receiving the drug and which the placebo. Con- about the attitudes and actions of college under- versely, the researcher who knew which subjects graduates but not about social attitudes and actions were in which group would not administer the in generaL experiment. However, this potential defect is less significant In social scientific experiments, as in medical in explanatory research than in descriptive re- experiments, the danger of experimenter bias is search. True, having noted the level of prejudice further reduced to the extent that the operational among a group of college undergraduates in our definitions of the dependent variables are clear and pretesting, we would have little confidence that the precise. Thus, medical researchers 'would be less same level existed among the public at large. On likely to unconSciously bias their reading of a pa- the other hand, if we found that a documentary tient's temperature than they would be to bias their film reduced whatever level of prejudice existed assessment of how lethargic the patient was. For among those undergraduates, we would have more the same reason, the small-group researcher would confidence-without being certain-that it would be less likely to misperceive which subject spoke, or have a comparable effect in the community at to whom he or she spoke, than whether the sub- large. Social processes and patterns of causal rela- ject's comments sounded cooperative or competi- tionships appear to be more generalizable and tive, a more subjective judgment that's difficult to more stable than specific characteristics such as an define in precise behavioral terms. individual's level of prejudice. As I've indicated several times, seldom can we Aside from the question of generalizability, the devise operational definitions and measurements cardinal rule of subject selection in experimentation that are wholly precise and unambiguous. This concerns the comparability of experimental and is another reason why it can be appropriate to control groups. Ideally, the control group represents employ a double-blind design in social research what the experimental group would be like if it had eX1Jeriments. /lor been exposed to the experimental stimulus. The logic of experiments requires, therefore, that exper- Selecting Subjects imental and control groups be as similar as possible. There are several ways to accomplish this. In Chapter 7 we discussed the logic of sampling, which involves selecting a sample that is represen- Probability Sampling tative of some population. Similar considerations apply to experiments. Because most social research- The discussions of the logic and techniques of prob- ers work in colleges and universities, it seems likely ability sampling in Chapter 7 provide one method that research laboratory experiments would be con- for selecting two groups of people that are similar ducted with college undergraduates as subjects. to each other. Beginning v'lith a sampling frame Typically, the eX1Jerimenter asks students enrolled composed of all the people in the population under in his or her classes to participate in experiments or study, the researcher might select two probability advertises for subjects in a college newspaper. Sub- samples. If these samples each resemble the total jects mayor may not be paid for participating in population from which they're selected, they'll also such experiments (recall also from Chapter 3 the resemble each other. ethical issues involved in asking students to partici- pate in such studies). Recall also, however, that the degree of resem- blance (representativeness) achieved by probability In relation to the norm of generalizability in sampling is largely a function of the sample size. As science, this tendency clearly represents a potential a general guideline, probability samples of less than 100 are not likely to be terribly representative, and

226 Chapter 8: Experiments social scientific experiments seldom involve that sample reflects the characteristics of the total popu- many subjects in either experimental or control lation, the two samples will mirror each other. groups. As a result, then, probability sampling is seldom used in experiments to select subjects from As we saw in Chapter 7, our assumption of a larger population. Researchers do, however, use similarity in the two groups depends in part on the the logic of random selection when they assign number of subjects involved. In the extreme case, if subjects to groups. we recruited only two subjects and assigned, by the flip of a coin, one as the experimental subject and Randomization one as the controL there would be no reason to assume that the two subjects are similar to each Having recruited, by whatever means, a total group other. With larger numbers of subjects, however, of subjects, the experimenter may randomly assign randomization makes good sense. those subjects to either the experimental or the control group. The researcher might accomplish Matching such randomization by numbering all of the sub- jects serially and selecting numbers by means of a Another way to achieve comparability between random number table. Alternatively, the experi- the experimental and control groups is through menter might assign the odd-numbered subjects to matching. This process is similar to the quota sam- the experimental group and the even-numbered pling methods discussed in Chapter 7. If 12 of our subjects to the control group. subjects are young white men, we might assign 6 of them at random to the experimental group and the Let's return again to the basic concept of proba- other 6 to the control group. If 14 are middle-aged bility sampling. If we recruit 40 subjects all to- African Anlerican women, we might assign 7 to gether, in response to a newspaper advertisement. each group. We repeat this process for every rele- for example, there's no reason to believe that the vant grouping of subjects. 40 subjects represent the entire population from which they've been drawn. Nor can we assume The overall matching process could be most that the 20 subjects randomly assigned to the ex- efficiently achieved through the creation of a quota perimental group represent that larger population. matrix constructed of all the most relevant charac- We can have greater confidence, however, that the teristics. Figure 8-2 provides a simplified illustration 20 subjects randomly assigned to the experimental of such a matrix. In this example, the experimenter group will be reasonably similar to the 20 assigned has decided that the relevant characteristics are to the control group. race, age, and gender. Ideally, the quota matrix is constructed to result in an even number of subjects Follmving the logic of our earlier discussions of in eadl cell of the matrix. Then, half the subjects in sampling, we can see our 40 subjects as a population each cell go into the experimental group and half from which we select two probability samples- into the control group. each consisting of half the population. Because each Alternatively, we might recruit more subjects randomization A technique for assigning experi- than our experimental design requires. We might mental subjects to experimental and control groups then examine many characteristics of the large ini- randomly. tial group of subjects. Whenever we discover a pair matching In connection vvith experiments, the of quite similar subjects, we might assign one at procedure whereby pairs of subjects are matched on random to the experimental group and the other to the basis of their similarities on one or more vari- the control group. Potential subjects who are un- ables, and one member of the pair is assigned to the like anyone else in the initial group might be left experimental group and the other to the control group. out of the experiment altogether. Whatever method we employ, the desired re- sult is the same. The overall average description of

Selecting Subjects 227 Control group 6 7 etc. FIGURE 8-2 Quota Matrix Illustration, Sometimes the experimental and control groups are created by finding pairs of matching subjects and aSSigning one to the experimental group and the other to the control group. the experimental group should be the same as that experimental and control groups on this variable to of the control group. For example, on average both assure ourselves that the two groups exhibit the groups should have about the same ages, the same same overall level of prejudice. gender composition, the same racial composition, and so forth. This test of comparability should be Matching or Randomization? used whether the two groups are created through probability sampling or through randomization. When assigning subjects to the experimental and control groups, you should be aware of two argu- Thus far I've referred to the \"relevant\" variables ments in favor of randomization over matching. without saying dearly what those variables are. Of First. you may not be in a position to know in course, these variables cannot be specified in any advance which variables will be relevant for the definite way, any more than I could specify in Chap- matching process. Second, most of the statistics ter 7 which variables should be used in stratified used to analyze the results of experiments assume sampling. Which variables are relevant ultimately randomization. Failure to design your experiment depends on the nature and purpose of an experi- that way, then, makes your later use of those statis- menlo As a general rule, however, the control and tics less meaningfuL experimental groups should be comparable in terms of those variables that are most likely to be related On the other hand, randomization only makes to the dependent variable under study. In a study sense if you have a fairly large pool of subjects, so of prejudice, for example, the two groups should that the laws of probability sampling apply. With be alike in terms of education, etlmicity, and age, only a few subjects, matching would be a better among other characteristics. In some cases, more- procedure. over, we may delay assigning subjects to experi- mental and control groups until we have initially Sometimes researchers can combine matching measured the dependent variable. Thus, for ex- and randomization. When conducting an experi- ample, we might administer a questionnaire mea- ment on the educational enridmlent of young ado- suring subjects' prejudice and then match the lescents, for example, J. Milton Yinger and his col- leagues (1977) needed to assign a large number of

228 Chapter 8: Experiments students, aged 13 and 14, to several different exper- film, mentioned earlier, to a group of people and imental and control groups to ensure the compara- then administer a questionnaire that seems to mea- bility of students composing each of the groups. sure prejudice against African Americans. Suppose They achieved this goal by the following method. further that the answers given to the questionnaire seem to represent a low level of prejudice. We Beginning with a pool of subjects, the re- might be tempted to conclude that the film reduced searchers first created strata of students nearly prejudice. Lacking a pretest, however, we can't be identical to one another in terms of some 15 vari- sure. Perhaps the questionnaire doesn't really rep- abIes. From each of the strata, students were ran- resent a sensitive measure of prejudice, or perhaps domly assigned to the different experimental and the group we're studying was low in prejudice to control groups. In this fashion, the researchers ac- begin with. In either case, the film might have tually improved on conventional randomization. made no difference, though our experimental re- Essentially, they had used a stratified-sampling pro- sults might have misled us into thinking it did. cedure (Chapter 7). except that they had employed far more stratification variables than are typically The second preexperimental design discussed used in, say, survey sampling. by Campbell and Stanley adds a pretest for the ex- perimental group but lacks a control group. This Thus far I've described the classical experi- design-which the authors call the one-group pretest- ment-the experimental design that best repre- posrtest design-suffers from the possibility that some sents the logic of causal analysis in the laboratory. factor other than the independent variable might In practice, however, social researchers use a cause a change between the pretest and posttest re- great variety of experimental designs. Let's look at sults, such as the assassination of a respected Afri- some now. can American leader. Thus, although we can see that prejudice has been reduced, we can't be sure Variations on Experimental that the film is what caused that reduction. DeSign To round out the possibilities for preexperimen- Donald Campbell and Julian Stanley (1963), in a tal designs, Campbell and Stanley point out that classic book on research design, describe some some research is based on experimental and con- 16 different experimental and quasi-experimental trol groups but has no pretests. They call this designs. This section describes some of these varia- design the sratic-group comparison. For example, we tions to better show the potential for experimenta- might show the African American history film to tion in social research. one group and not to another and then measure prejudice in both groups. If the experimental Preexperimental Research DeSigns group had less prejudice at the conclusion of the experiment, we might assume the film was To begin, Campbell and Stanley discuss three \"pre- responsible. But unless we had randomized our experimental\" designs, not to recommend them subjects, we would have no way of knovving that but because they're frequently used in less-than- the two groups had the same degree of prejudice professional research. These designs are called \"pre- initially; perhaps the experimental group started experimental\" to indicate that they do not meet the out \"l'vith less. scientific standards of experimental designs. In the first such design-the one-shot case study-the re- Figure 8-3 graphically illustrates these three searcher measures a single group of subjects on a preexperimental research designs by using a differ- dependent variable following the administration ent research question: Does exercise cause weight of some experimental stimulus. Suppose, for ex- reduction? 10 make the several designs clearer, the ample, that we show the African American history figure shows individuals rather than groups, but the same logic pertains to group comparisons. Let's review the three preexperimental designs in this new example.

Variations on Experimental Design 229 One-Shot Case Study Some intuitive standard of A man who exercises what constitutes is observed to be in a trim shape trim shape Time 1 Time 2 Time 3 One-Group Pretest-PosHest Design An overweight man who exercises is later observed to be in trim shape Time 1 Time 2 Time 3 Static-Group Comparison A man who exercises is observed to be in trim shape while one who doesn't is observed to be overweight Time 1 Time 2 Time 3 FIGURE 8-3 Three Preexperimental Research Designs. These preexperimental designs anticipate the logic of true experiments but leave them- selves open to errors of interpretation, Can you see the errors that might be made in each of these designs? The various risks are solved by the addition of control groups, pretesting, and postlesting. The one-shot case study represents a common beginning to exercise. Or perhaps he became thin form of logical reasoning in everyday life. Asked for some other reason, like eating less or getting whether exercise causes weight reduction, we sick. The observations shown in the diagram may bring to mind an example that would seem to do not guard against these other possibilities. support the proposition: someone who exercises Moreover, the observation that the man in the and is thin. There are problems with this reasoning, diagram is in trim shape depends on our intuitive however. Perhaps the person was thin long before idea of what constitutes trim and overweight body

230 Chapter 8: Experiments shapes. All told, this is very weak evidence for gone on in the e;qJeriment itself. The threat of in- testing the relationship between exercise and ternal invalidity is present whenever anything 'weight loss. other than the experimental stimulus can affect the dependent variable. The one-group pretest-posttest design offers somewhat better evidence that exercise produces Campbell and Stanley (1963: 5-6) and Cook weight loss. Specifically, we've ruled out the and Campbell (1979: 51-55) point to several possibility that the man was thin before beginning sources of internal invalidity. Here are 12: to exercise. However, we still have no assurance that his exercising is what caused him to lose 1. HistOlY. During the course of the eA'Periment, weight. historical events may occur that will confound the eA'Perimental results. The assassination of an Afri- Finally, the static-group comparison eliminates can American leader during the course of an ex- the problem of our questionable definition of what periment on reducing anti-African American prej- constitutes trim or overweight body shapes. In this udice is one example; the arrest of an African case, we can compare the shapes of the man who American leader for some heinous crime, which exercises and the one who does not. This design, might increase prejudice, is another. however, reopens the possibility that the man who exercises was thin to begin with. 2. Maturation. People are continually growing and changing, and such changes can affect the re- Validity Issues in sults of the eA'Periment. In a long-term experiment, Experimental Research the fact that the subjects grow older (and wiser?) may have an effect. In shorter experiments, they At this point I want to present in a more systematic may grow tired, sleepy, bored, or hungry, or change way the factors that affect the validity of experi- in other ways that affect their behavior in the mental research. First we'll look at what Campbell experiment. and Stanley call the sources of il1ternal invalidity, reviewed and expanded in a follow-up book by 3. TeSTing. As we have seen, often the process of Thomas Cook and Donald Campbell (1979). Then testing and retesting influences people's behavior, we'll consider the problem of generalizing experi- thereby confounding the experinlental results. mental results to the \"real\" world, referred to as Suppose we administer a questionnaire to a group eXTernal invalidity. Having examined these, we'll be as a way of measuring their prejudice. Then we in a position to appreciate the advantages of some administer an experimental stimulus and remea- of the more sophisticated experimental and quasi- sure their prejudice. By the time we conduct the experimental designs social science researchers posttest, the subjects 'will probably have become sometimes use. more sensitive to the issue of prejudice and will be more thoughtful in their answers. In fact, they may Sources ofIntemallnvalidity have figured out that we're trying to find out how prejudiced they are, and, because few people like to The problem of internal invalidity refers to the appear prejudiced, they may give answers that they possibility that the conclusions drawn from experi- think we want or that will make them look good. mental results may not accurately reflect what has 4. I11STrUmentatioll. The process of measurement in internal invalidity Refers to the possibility that pretesting and posttesting brings in some of the is- the conclusions drawn from experimental results sues of conceptualization and operationalization may not accurately reflect what went on in the discussed earlier in the book. If we use different experiment itself. measures of the dependent variable in the pretest and posttest (say, different questionnaires about prejudice), how can we be sure tney're comparable to each other? Perhaps prejudice will seem to decrease simply because the pretest measure was

Variations on Experimental Design 231 more sensitive than the posttest measure. Or if the prejudiced to stan with, so the group results 'will measurements are being made by the experi- reflect a substantial \"decrease\" in prejudice. menters, their standards or their abilities may change over the course of the experiment. S. Causal rime order. Though rare in social research, ambiguity about the time order of the experimental 5.. Starisrical regression Sometimes it's appropriate stimulus and the dependent variable can arise. to conduct experiments on subjects who start out Whenever this occurs, the research conclusion with extreme scores on the dependent variable. If that the stimulus caused the dependent variable you were testing a new method for teaching math can be challenged with the explanation that the to hard-core failures in math, you'd want to con- \"dependent\" variable actually caused changes in duct your experiment on people who previously the stimulus. had done extremely poorly in math. But consider for a minute what's likely to happen to the math 9. DUfllSiol1 or imitariol1 oftreatments. When achievement of such people over time without any experimental and control-group subjects can experimental interference. They're starting out so communicate with each other, experimental sub- low that they can only stay at the bottom or inl- jects may pass on some elements of the experimen- prove: They can't get worse. Even without any ex- tal stimulus to the control group. For example, sup- perimental stimulus, then, the group as a whole is pose there's a lapse of time between our shOwing of likely to show some improvement over time. Refer- the African American history film and the posttest ring to a regressioll TO rlze mea11, statisticians often administration of the questionnaire. Members of point out that extremely tall people as a group are the experimental group might tell control-group likely to have children shorter than themselves, subjects about the film. In that case, the control and extremely short people as a group are likely to group becomes affected by the stimulus and is not a have children taller than themselves. There is a real control. Sometimes we speak of the control danger, then, that changes occurring by virtue of group as having been \"contaminated.\" subjects starting out in extreme positions will be at- tributed erroneously to the effects of the experi- 10. COlllpcllSarioll. As you'll see in Chapter 12, in mental stimulus. experiments in real-life situations-such as a spe- cial educational program-subjects in the control 6. Selectio11 biases. We discussed selection bias earlier group are often deprived of something considered when we examined different ways of selecting sub- to be of value. In such cases, there may be pres- jects for experiments and assigning them to experi- sures to offer some form of compensation. For ex- mental and control groups. Comparisons don't ample, hospital staff might feel sorry for control- have any meaning unless the groups are compara- group patients and give them extra \"tender loving ble at the start of an experiment. care.\" In such a situation, the control group is no longer a genuine control group. 7. Experimemalmorrality. Although some social experiments could, I suppose, kill subjects, experi- 11. COlllpensarOlY rivaby. In real-life experiments, the memallllorrality refers to a more general and less- subjects deprived of the experimental stimulus may extreme problem. Often, experimental subjects will try to compensate for the missing stimulus by work- drop out of the experiment before it's completed, ing harder. Suppose an experimental math program and this can affect statistical comparisons and con- is the experimental stinlUlus; the control group may clusions. In the classical experiment involving an work harder than before on their math in an at- experimental and a control group, each with a tempt to beat the \"special\" experimental subjects. pretest and posttest, suppose that the bigots in the experimental group are so offended by the African 12. Demoralizatiol1. On the other hand, feelings of American history film that they tell the experi- deprivation within the control group may result in menter to forget it, and they leave. Those subjects their giving up. In educational experiments, de- sticking around for the posttest will have been less moralized control-group subjects may stop study- ing, act up, or get angry.

232 Chapter 8: Experiments Stimulus Postiest Pretest Experimental Group Control Group FIGURE 8-4 The Classical Experiment: Using an African American History Film to Reduce Prejudice. TIlis diagram illustrates the basiC structure of the classical experiment as avehicle for testing the impact of afilm on prejudice. Notice how the control group, the pretesting, and the posttesting function. These, then, are some of the sources of internal and instrumentation can't be problems, because invalidity in experiments. Aware of these, experi- both the experimental and control groups are sub- menters have devised designs aimed at handling ject to the same tests and experimenter effects. If them. The classical e:.:periment, if coupled with the subjects have been assigned to the two groups proper subject selection and assignment, addresses randomly, statistical regression should affect both each of these problems. Let's look again at that equally, even if people with extreme scores on study design, presented graphically in Figure 8-4. prejudice are being studied. Selection bias is ruled out by the random assignment of subjects. Experi- If we use the experimental design shoH'I1 in mental mortality is more complicated to handle, Figure 8-4, we should expect two findings. For the but the data provided in this study design offer experimental group, the level of prejudice mea- several ways to deal with it. Slight modifications to sured in their posttest should be less than was found the design-administering a placebo (such as a in their pretest. In addition, when the two posttests film having nothing to do with African Americans) are compared, less prejudice should be found in the to the control group, for example-can make the experin1ental group than in the control group. problem even easier to manage. This design also guards against the problem of The remaining five problems of internal inva- history in that anything occurring outside the ex- lidity are avoided through the careful administra- periment that might affect the experimental group tion of a controlled experimental design. The ex- should also affect the control group. Consequently, perimental design we've been discussing facilitates there should still be a difference in the two posttest the dear specification of independent and depen- results. The same comparison guards against prob- dent variables. Ex-perimental and control subjects lems of maturation as long as the subjects have can be kept separate, reducing the possibility of been randomly assigned to the two groups. Testing

Variations on Experimental Design 233 diffusion or imitation of treatments. Administrative Stimulus controls can avoid compensations given to the (film) control group, and compensatory rivalry can be watched for and taken into account in evaluating No the results of the ex-periment, as can the problem of stimulus demoralization. =====TIME=_ _- ' Sources of Exlemallnvalidily Expected Findings Internal invalidity accounts for only some of the In Group 1, postlest prejudice should be less than complications faced by experimenters. In addition, pretest prejudice. there are problems of what Campbell and Stanley In Group 2, prejudice should be the same in the call external inValidity, which relates to the gen- pretest and the postlest eralizability of ex-perimental findings to the \"real\" The Group 1 postlest should show less prejudice world. Even if the results of an ex-periment provide than the Group 2 postlest an accurate gauge of what happened during that The Group 3 postlest should show less prejudice experiment do they really teU us anything about than the Group 4 postles! life in the wilds of society? FIGURE 8-5 Campbell and Stanley describe four forms of The Solomon Four-Group Design. The classical experiment this problem; I'll present one as an illustration. runs the risk that pretesting will have an effect on subjects, so The generalizability of experimental findings is the Solomon Four Group Design adds experimental and con- jeopardized, as the authors point out if there's an trol groups that skip the pretest. interaction between the testing situation and the experimental stimulus (1963: 18). Here's an ex- Notice that Groups land 2 in Figure 8-5 ample of what they mean. compose the classical experiment, with Group 2 being the control group. Group 3 is administered Staying with the study of prejudice and the the experimental stimulus without a pretest and African American history film, let's suppose that our Group 4 is only posttested. This ex-perimental de- experimental group-in the classical experiment- sign permits four meaningful comparisons, which has less prejudice in its posttest than in its pretest are described in the figure. If the African American and that its posttest shows less prejudice than that history film really reduces prejudice-unaccounted of the control group. We can be confident that the for by the problem of internal validity and unac- film actually reduced prejudice among our experi- counted for by an interaction between the testing mental subjects. But would it have the same effect if the film were shown in theaters or on television? external invalidity Refers to the possibility that We can't be sure, because the film might be effec- conclusions drawn from experimental results may tive only when people have been sensitized to the not be generalizable to the \"reaJ\" world. issue of prejudice, as the subjects may have been in taking the pretest. This is an example of interaction between the testing and the stimulus. The classical experimental design cannot control for that possi- bility. Fortunately, experimenters have devised other designs that can. The So!olllonfouT-group design (D. Campbell and Stanley 1963: 24-25) addresses the problem of testing interaction with the stimulus. As the name suggests, it involves four groups of subjects, as- signed randomly from a pool. Figure 8-5 presents this design graphically.

234 Chapter 8: Experiments and the stimulus-we should expect four designs that include it Be clear, however, that this findings: point applies only to experiments in which subjects have been assigned to experimental and control L In Group L posttest prejudice should be less groups randomly, because that's what justifies the than pretest prejudice. assumption that the groups are equivalent without having been measured to find out. 2. In Group 2, prejudice should be the same in the pretest and the posttest. This discussion has introduced the intricacies of experimental design, its problems, and some solu- 3. The Group 1 posttest should show less preju- tions. There are, of course, a great many other ex- dice than the Group 2 posttest. perimental designs in use. Some involve more than one stimulus and combinations of stimuli. Others 4. The Group 3 posttest should show less preju- involve several tests of the dependent variable over dice than the Group 4 posttest. time and the administration of the stimulus at dif- ferent times for different groups. If you're inter- Notice that finding (4) rules out any interaction ested in pursuing this topic, you might look at the between the testing and the stimulus. And remem- Campbell and Stanley book. ber that these comparisons are meaningful only if subjects have been assigned randomly to the differ- An Illustration ent groups, thereby providing groups of equal prej- of Experimentation udice initiaily, even though their preexperimental prejudice is only measured in Groups 1 and 2. Experiments have been used to study a ,'\\'ide vari- ety of topics in the social sciences. Some experi- There is a side benefit to this research design, as ments have been conducted within laboratory the authors point out. Not only does the Solomon situations; others occur out in the \"real world.\" The four-group design rule out interactions between folloyving discussion provides a glimpse of both. testing and the stimulus, it also provides data for comparisons that \\vill reveal how much of this in- Let's begin with a \"real world\" example. In teraction has occurred in a classical experiment George Bernard Shaw's well-loved play, Pygma- This knowledge allows a researcher to review and lion-the basis of the long-running Broadway mu- evaluate the value of any prior research that used sical, My Fair Lady-Eliza Doolittle speaks of the the simpler design. powers others have in determining our social iden- tity. Here's how she distinguishes the way she's The last experimental design I'll mention here treated by her tutor, Professor Higgins, and by Hig- is what Campbell and Stanley (1963: 25-26) call gins's friend, Colonel Pickering: the posttest-only control group design; it consists of the second half-Groups 3 and 4-of the Solomon You see, really and truly, apart from the things design. As the authors argue persuasively, vvith anyone can pick up (the dressing and the proper proper randomization, only Groups 3 and 4 are way of speaking, and so on), the difference be- needed for a true experiment that controls for the tween a lady and a flower girl is not how she problems of internal invalidity as well as for the in- behaves, but how she's treated. I shall always be teraction between testing and stimulus. With ran- a flower girl to Professor Higgins, because he domized assignment to experimental and control always treats me as a flower girl, and always groups (which distinguishes this design from the will, but I know I can be a lady to you, because static-group comparison discussed earlier), the sub- you always treat me as a lady, and always wilL jects will be initially comparable on the dependent variable-comparable enough to satisfy the con- (Act V) ventional statistical tests used to evaluate the re- sults-so it's not necessary to measure them. In- The sentiment Eliza expresses here is basic so- deed, Campbell and Stanley suggest that the only cial science, addressed more formally by sociologists justification for pretesting in this situation is tradi- such as Charles Horton Cooley (the \"looking-glass tion. Experimenters have simply grown accustomed to pretesting and feel more secure with research

An Illustration of Experimentation 235 self\") and George Herbert Mead (\"the generalized Subsequent experiments have focused on specific other\"). The basic point is that who we think we aspects of what has become known as the attribu- are-our self-concept-and how we behave are rion process, or the expectatiollS communication model. largely a function of how others see and treat us. This research, largely conducted by psychologists, Related to this, the way others perceive us is largely parallels research primarily by sociologists, which conditioned by expectations they have in advance. If takes a slightly different focus and is often gathered they've been told we're stupid, for example, they're under the label expectations-states theory. Psychologi- likely to see us that way-and we may come to see cal studies focus on situations in which the expec- ourselves that way and actually act stupidly \"Label- tations of a dominant individual affect the perfor- ing theory\" addresses the phenomenon of people mance of subordinates-as in the case of a teacher acting in accord with the ways that others perceive and students, or a boss and employees. The socio- and label them. These theories have served as the logical research has tended to focus more on the premise for numerous movies, such as the 1983 film role of expectations among equals in smail, task- Trading Places, in which Eddie Murphy and Dan oriented groups. In a jury, for example, how do ju- Ackroyd playa derelict converted into a stock- rors initially evaluate each other, and how do those broker and vice versa. initial assessments affect their later interactions? (You can learn more about this phenomenon, in- The tendency to see in others what we've been cluding attempts to find practical applications, by led to expect takes its name from Shaw's play. searching the web for \"Pygmalion Effect.\") Called the Pygmalion effect, it's nicely suited to con- trolled experiments. In one of the best-known ex- Here's an example of an experiment conducted perimental investigations of the Pygmalion effect, to examine the way our perceptions of our abilities Robert Rosenthal and Lenore Jacobson (l968) ad- and those of others affect our willingness to accept ministered what they called the \"Harvard Test of the other person's ideas. Martha Foschi, G. Keith Inflected Acquisition\" to students in a West Coast Warriner, and Stephen Hart (1985) were particu- school. Subsequently, they met with the students' larly interested in the role \"standards\" play in that teachers to present the results of the test In partic- respect: ular, Rosenthal and Jacobson identified certain students as very likely to exhibit a sudden spurt in In general terms, by \"standards\" we mean academic abilities during the coming year, based on how well or how poorly a person has to per- the results of the test. form in order for an ability to be attributed or denied him/her. In our view, standards are a When IQ test scores were compared later, key variable affecting how evaluations are the researchers' predictions proved accurate. processed and what expectations result For The students identified as \"spurters\" far exceeded example, depending on the standards used, their classmates during the following year, sug- the same level of success may be interpreted gesting that the predictive test was a powerful one. as a major accomplishment or dismissed as In fact, the test was a hoax! The researchers had unimportant made their predictions randomly among both good and poor students. What they told the teachers (1985 108-9) did not really reflect students' test scores at alL The progress made by the \"spurters\" was simply a To begin examining the role of standards, the result of the teachers expecting the improvement researchers designed an experiment involving four and paying more attention to those students, en- eX'J)erimental groups and a control. Subjects were couraging them, and rewarding them for achieve- told that the experiment involved something cailed ments. (Notice the similarity between this situation \"pattern recognition ability,\" defined as an innate and the Hawthorne effect discussed earlier in this ability some people had and others didn't. The re- chapteL) searchers said subjects would be working in pairs on pattern recognition problems\" The Rosenthal-Jacobson study attracted a great deal of popular as well as scientific attention. In fact, of course, there's no such thing as pattern recognition ability. The object of the

236 Chapter 8: Experiments experiment was to determine how information 2. You are possibly better than your partner. about this supposed ability affected subjects' subse- 3. You are possibly WOloe than your partner. quent behavior. 4. You are definitely worse than your partner. The first stage of the experiment was to \"test\" The control group for this experiment was told each subject's pattern recognition abilities. If you nothing about their own abilities or their partners'. had been a subject in the experin1ent, you would In other words, they had no expectations. have been shovvn a geometrical pattern for 8 sec- onds, followed by two more patterns, each of The final step in the e}q)eriment was to set the which was similar to but not the same as the first \"teams\" to work. As before, you and your partner one. Your task would be to choose which of the would be given an initial pattern, followed by a subsequent set had a pattern closest to the first one comparison pair to choose from. When you en- you saw. You would be asked to do this 20 times, tered your choice in this round, however, you and a computer would print out your \"score.\" Half would be told what your partner had answered; the subjects would be told that they had gotten 14 then you would be asked to choose again. In your correct; the other half would be told that they had final choice, you could either stick with your origi- gotten only 6 correct-regardless of which patterns nal choice or switch. The \"partner's\" choice was, of they matched vvith which. Depending on the luck course, created by the computer, and as you can of the draw, you would think you had done either guess, there were often a disagreements in the quite well or quite badly. Notice, however, that teams: 16 out of 20 times, in fact you wouldn't really have any standard for judging your performance-maybe getting 4 correct would The dependent variable in this experiment be considered a great performance. was the extent to which subjects would switch their choices to match those of their partners. The At the same tin1e you were given your score, researchers hypothesized that the definitely better however, you would also be given your \"partner's group would svvitch least often, followed by the score,\" although both the \"partners\" and their probably better group, followed by the control group, \"scores\" would also be computerized fictions. (Sub- followed by the probably worse group, followed jects were told they would be communicating with by the definirely worse group, who would s,vitch their partners via computer terminals but would most often. not be allowed to see each otheL) If you were as- signed a score of 14, you would be told your part- The number of times subjects in the five groups ner had a score of 6; if you were assigned 6, you switched their answers follows. Realize that each would be told your partner had 14. had 16 opportunities to do so. These data indicate that each of the researchers' ex-pectations was This procedure meant that you would enter correct-with the exception of the comparison be- the teamwork phase of the experiment believing tween the possibly worse and definitely worse groups. either (1) you had done better than your partner Although the latter group was in fact the more or (2) you had done worse than your partner. This likely to switch, the difference was too small to be information constituted part of the \"standard\" you taken as a confirmation of the hypothesis. (Chapter would be operating under in the experiment. In 16 will discuss the statistical tests that let re- addition, half of each group was told that a score of searchers make decisions like this.) between 12 and 20 meant the subject definitely had pattern recognition ability; the other subjects were Definitely better Mean Number ofSwitches told that a score of 14 wasn't really high enough to Possibly better 5.05 prove anything definite. Thus, you would emerge Control group 6.23 from this with one of the following beliefs: Possibly worse 7.95 Definitely worse 9.23 1. You are definitely better at pattern recognition 9.28 than your partner.

Alternative Experimental Settings 237 In more-detailed analyses, it was found that the most experiments, researchers can often use volun- same basic pattern held for both men and women, teers who respond to invitations online. Here are though it was somewhat clearer for women than two sites you might visit to get a better idea of this for men. Here are the actual data: form of experimentation. Mean Number <Il Online Sodal Psychology Studies: Imp://www ofSwitches .socialpsychology.org/expts.htm. This website offers hot links to numerous profeSSional and Women Men student research projects on such topics as \"in- terpersonal relations,\" \"beliefs and attitudes,\" Definitely better 4.50 5.66 and \"personality and individual differences.\" Possibly better In addition, the site offers some resources for Control group 6.34 6.10 conducting web experiments. Possibly worse Definitely worse 7.68 8.34 <Il Small World Phenomenon: http://smallworld .columbia.edu. This study, by a group of sociol- 9.36 9.09 ogists at Columbia University, is seeking to re- search the concept of \"six degrees of separa- 10.00 8.70 tion.\" Based on the idea that you know someone who knows someone who knows someone, Because specific research efforts like this one this experiment seeks to determine the average sometimes seem extremely focused in their scope, number of steps it takes to get from one person you might wonder about their relevance to any- to another around the world. thing. As part of a larger research effort, however, You may have heard this concept expressed studies like this one add concrete pieces to our un- as something like this: \"There are no more than derstanding of more general social processes. six degrees of separation between any two people on earth,\" but that's not exactly accu- It's worth taking a minute or so to consider rate. In 1967 Stanley Milgram selected 300 some of the life situations where \"expectation people at random in Omaha and asked them to states\" might have very real and important conse- contact a specific person in Boston, through a quences. I've mentioned the case of jury delibera- chain of personal contacts. The average num- tions. How about all forms of prejudice and dis- ber of steps required was indeed six, but only crimination? Or, consider how expectation states 20 percent succeeded. The Columbia study is figure into job interviews or meeting your heart- an attempt to examine the matter in a more throb's parents. If you think about it, you'll un- rigorously controlled manner. doubtedly see other situations where these labora- tory concepts apply in real life. \"Natural\" Experiments Alternative Experimental Important social scientific ex-perin1ents can occur in Settings the course of normal social events, outside con- trolled settings. Sometimes nature designs and exe- Although we tend to equate the terms experiment cutes experiments that we can observe and ana- and laboratOlyexperiment, many important social lyze; sometimes social and political decision makers scientific experiments occur outside controlled set- serve this natural function. tings, such as on the Internet or in \"real life.\" Imagine, for example, that a hurricane has Web-Based Experiments struck a particular town. Some residents of the town suffer severe financial damages, and others Increasingly, researchers are using the World Wide Web as a vehicle for conducting experiments. Be- cause representative samples are not essential in

238 Chapter 8: Experiments escape relatively lightly. What, we might ask, are failed in the number 2 reactor and began melting the behavioral consequences of suffering a natural the uranium core. The survey was conducted five disaster? Are those who suffer most more likely to to six months after the accident. Among other take precautions against future disasters than those things, the survey questionnaire measured work- who suffer least are? To answer these questions, we ers' attitudes toward working at nuclear power might interview residents of the town some time plants. If they had measured only the TMI workers' after the hurricane. We might question them re- attitudes after the accident, the researchers would garding their precautions before the hurricane and have had no idea whether attitudes had changed as the ones they're currently taking, comparing the a consequence of the accident. But they improved people who suffered greatly from the hurricane their study design by selecting another, nearby- with those who suffered relatively little. In this seemingly comparable-nuclear power plant (ab- fashion, we might take advantage of a natural ex- breviated as PB) and surveyed workers there as a periment, which we could not have arranged even control group: hence their reference to a static- if we'd been perversely willing to do so. group comparison. A similar example comes from the annals of so- Even with an experimental and a control cial research concerning World War II. After the group, the authors were wary of potential problems war ended, social researchers undertook retrospec- in their design. In particular, their design was tive surveys of wartime morale among civilians in based on the idea that the two sets of workers were several German cities. Among other things, they equivalent to each other, except for the single wanted to determine the effect of mass bombing fact of the accident. The researchers could have on the morale of civilians. They compared the assumed this if they had been able to assign work- reports of wartime morale of residents in heavily ers to the two plants randomly. but of course that bombed cities with reports from cities that received was not the case. Instead, they needed to compare relatively little bombing. (Bombing did not reduce characteristics of the two groups and infer whether morale.) or not they were equivalent Ultimately, the re- searchers concluded that the two sets of workers Because the researcher must take things pretty were very much alike, and the plant the employees much as they occur, natural experiments raise worked at was merely a function of where they many of the validity problems discussed earlieL lived. Thus, when Stanislav Kasl. Rupert Chisolm, and Brenda Eskenazi (1981) chose to study the impact Even granting that the two sets of workers that the Three Mile Island (TMI) nuclear accident were equivalent the researchers faced another in Pennsylvania had on plant workers, they had to problem of comparability. They could not contact be especially careful in the study design: all the workers who had been employed at TMI at the time of the accident The researchers discussed Disaster research is necessarily opportunistic, the problem as follows: quasi-experimental. and after-the-fact In the terminology of Campbell and Stanley's classical One special attrition problem in this study was analysis of research designs, our study falls into the possibility that some of the no-contact the \"static-group comparison\" category, consid- nonrespondents among the TMI subjects, but ered one of the weak research designs. How- not PB subjects, had permanently left the area ever. the weaknesses are potential and their because of the accident This biased attrition actual presence depends on the unique circum- would, most likely, attenuate the estin1ated stances of each study. extent of the impact. Using the evidence of disconnected or \"not in service\" telephone (1981.474) numbers, we estimate this bias to be negligible (1 percent). The foundation of this study was a survey of the people who had been working at Three Mile Is- ([(asl, Cizisolm, and Eskenazi 1981.' 475) land on March 28, 1979, when the cooling system

Strengths and Weaknesses of the Experimental Method 239 The TMI example points to both the special evaluation research involves taking the logic of ex- problems involved in natural experiments and the perimentation into the field to observe and evalu- possibility for taking those problems into account. ate the effects of stimuli in real life. Because this is Social research generally requires ingenuity and in- an increasingly important form of social research, sight; natural experiments call for a little more than an entire chapter is devoted to it the average. Strengths and Weaknesses Earlier in this chapter, we used a hypothetical of the Experimental Method example of studying whether an African American history film reduced prejudice. Sandra Ball- Experiments are the primary tool for studying Rokeach, Joel Grube, and Milton Rokeach (1981) causal relationships. However. like all research were able to address that topic in real life through a methods, experin1ents have both strengths and natural experiment. In 1977, the television drama- weaknesses. tization of Alex Haley's Roots, a historical saga about African Americans, was presented by ABC on eight The chief advantage of a controlled experiment consecutive nights. It garnered the largest audiences lies in the isolation of the experimental variable's in television history up to that time. Ball-Rokeach impact over time. This is seen most clearly in terms and her colleagues wanted to know whether Roots of the basic experimental model. A group of experi- changed white Americans' attitudes toward African mental subjects are found, at the outset of the ex- Americans. Their opportunity arose in 1979, when periment, to have a certain characteristic; follOwing a sequel-Roots: The Ne>.1 Generation-was televised. the administration of an experimental stimulus, Although it would have been nice (from a re- they are found to have a different characteristic. To searcher's point of view) to assign random samples the extent that subjects have experienced no other of Americans either to watch or not to watch the stimuli, we may conclude that the change of charac- show, that wasn't possible. Instead, the researchers teristics is attributable to the experimental stimulus. selected four samples in Washington State and mailed questionnaires that measured attitudes to- Further, because individual experiments are of- ward African Americans. FollO'i'Ving the last episode ten rather limited in scope, requiring relatively little of the show, respondents were called and asked time and money and relatively few subjects, we of- how many, if any, episodes they had watched. ten can replicate a given experiment several times Subsequently, questionnaires were sent to respon- using several different groups of subjects. (This isn't dents, remeasuring their attitudes toward African always the case, of course, but it's usually easier to Americans. repeat experiments than, say, surveys.) As in all other forms of scientific research, replication of re- By comparing attitudes before and after for search findings strengthens our confidence in the both those who watched the show and those who validity and generalizability of those findings. didn't, the researchers reached several conclusions. For example, they found that people with already The greatest weakness of laboratory experi- egalitarian attitudes were much more likely to ments lies in their artificiality. Social processes that watch the show than were those who were more occur in a laboratory setting might not necessarily prejudiced toward African Americans: a self- occur in natural social settings. For example, an selection phenomenon. Comparing the before African American history film might genuinely re- and after attitudes of those who watched the show, duce prejudice among a group of experimental moreover, suggested the show itself had little or no subjects. This would not necessarily mean, how- effect. Those who watched it were no more egali- ever, that the same film shown in neighborhood tarian afterward than they had been before. movie theaters throughout the country would re- duce prejudice among the general public. Artificial- This example anticipates the subject of Chapter ity is not as much of a problem, of course, for 12, evaluation research, which can be seen as a special type of natural experiment. As you'll see,

240 Chapter 8: Experiments natural experiments as for those conducted in the Variations on Experimental Design laboratory. 61 Campbell and Stanley describe three forms of In discussing several of the sources of internal preexperiments: the one-shot case study, the and external invalidity mentioned by Campbell, one-group pretest-posttest design, and the Stanley, and Cook, we saw that we can create ex- static-group comparison. None of these designs perimental designs that logically control such prob- features all the controls available in a true lems. This possibility points to one of the great ad- experiment vantages of experiments: They lend themselves to a logical rigor that is often much more difficult to 61 Campbell and Stanley list, among others, 12 achieve in other modes of observation. sources of internal invalidity in experimental design. The classical experiment with random MAIN POINTS assignment of subjects guards against each of these problems. Introduction 61 In experiments, social researchers typically se- 61 Experiments also face problems of external in- validity: Experimental findings may not reflect lect a group of subjects, do something to them, real life. and observe the effect of what was done. 61 The interaction of testing and stimulus is an ex- TOpics Appropriate to Experiments ample of external invalidity that the classical 61 Experiments are an excellent vehicle for the experiment does not guard against. controlled testing of causal processes. 61 The Solomon four-group design and other vari- ations on the classical experinlent can safe- The Classical Experiment guard against external invalidity. 61 The classical experiment tests the effect of an 61 Campbell and Stanley suggest that, given proper experimental stimulus (the independent vari- randomization in the assignment of subjects to able) on a dependent variable through the the experimental and control groups, there is pretesting and posttesting of experimental and no need for pretesting in experiments. control groups. 61 It is generally less important that a group of An Illustration of Experimentation e}.'Perimental subjects be representative of 61 Experiments on \"expectation states\" demon- some larger population than that experimental and control groups be similar to each other. strate experinlental designs and show how ex- 61 A double-blind e}.'Periment guards against ex- periments can prove relevant to real-world perimenter bias, because neither the experi- concerns. menter nor the subject knows which subjects are in the control and experimental groups. Alternative Experimental Settings Selecting Subjects 61 More and more, researchers are using the In- 61 Probability sampling, randomization, and ternet for conducting experiments. matching are all methods of achieving compa- 61 Natural experiments often occur in the course rability in the experimental and control groups. of social life in the real world, and social re- Randomization is the generally preferred searchers can implement them in somewhat method. In some designs, it can be combined the same way they would design and conduct with matching. laboratory experinlents. Strengths and Weaknesses of the Experimental Method 61 Like all research methods, experiments have strengths and weaknesses. Their primary weakness is artificiality: What happens in an

Online Study Resources 241 experinlent may not reflect what happens in social research. This book is especially useful in the outside world. Their strengths include the its application of the logic of experiments to isolation of the independent variable, which other social research methods. Though fairly permits causal inferences; the relative ease of old. this book has attained the status of a classic replication; and scientific rigor. and is still frequently cited. KEY TERMS Cook. Thomas D.. and Donald T. Campbell. 1979. Quasi-Experimemario/Z:' Design and Analysis Issues The following terms are defined in context in the for Field Settillgs . Chicago: Rand McNally. An ex- chapter and at the bottom of the page where the term panded and updated version of Campbell and is introduced, as well as in the comprehensive glossary Stanley. at the back of the book. Jones. Stephen R. G. 1990. \"Worker Independence control group matching and Output: The Hawthorne Studies Reevalu- double-blind experiment posttesting ated.\" American Sociological Review 55:176-90. experimental group pretesting This article reviews these classical studies and external invalidity randomization questions the traditional interpretation (which internal invalidity was presented in this chapter). REVIEW QUESTIONS AND EXERCISES Martin. David W. 1996. Doillg Psychology Experimellts. 4th ed. Monterey. CA: Brooks/Cole. With thor- 1. In the library or on the web, locate a research ough explanations of the logic behind research report of an experiment. Identify the dependent methods. often in a humorous style. this book variable and the stimulus. emphasizes ideas of particular importance to the beginning researcher. such as getting an idea for 2. Pick 6 of the 12 sources of internal invalidity an experiment and reviewing the literature. discussed in this chapter and make up examples (not discussed in the chapter) to illustrate each. Ray. William J. 2000. AIethods toward a Science of Behavior alld Erperience. 6th ed. Belmont. CA: 3. Create a hypothetical experimental design Wadsworth. A comprehensive examination of that illustrates one of the problems of external social scientific research methods. with a special invalidity. emphasis on experimentation. This book is es- pecially strong in the philosophy of science. 4. Think of a recent natural disaster you've wit- nessed or read about. Frame a research question SPSS EXERCISES that might be studied by treating that disaster as a natural experiment. In two or three para- See the booklet that accompanies your text for exer- graphs. outline how the study might be done. cises using SPSS (Statistical Package for the Social Sci- ences). There are exercises offered for each chapter. 5. In this chapter. we looked briefly at the problem and you'll also find a detailed primer on using SPSS. of \"placebo effects.\" On the web. find a study in which the placebo effect figured importantly. Online Study Resources Write a brief report on the study. including the source of your information. (Hint: you might Sociology ~ Now\"': Research Methods want to do a search on \"placebo.\") 1. Before you do your final review of the chapter, ADDITIONAL READINGS take the SociologyNow. Research /vletlzods diagnos- tic quiz to help identify the areas on which you Campbell, Donald. and Julian Stanley. 1963. Erperi- should concentrate. You'lI find information on melllal alld Quasi-Experimental Desiglls for Research. this online tool. as well as instructions on how Chicago: Rand McNally. An excellent analysis of to access all of its great resources. in the from of the logic and methods of experimentation in the book.

242 Chapter 8: Experiments 20 As you review, take advantage of the Sociology Yahoo! Directory: Tests and Experiments Now.: Research Methods customized study plan, http://dir.yahoo.com /SociaCScience/Psychology/ based on your quiz results. Use this study plan Research/ Tests_and_Experiments / with its interactive exercises and other re- Here you'll find an extensive list of websites relating to sources to master the material. various kinds of social science experiments. Some tell you about past or ongoing experiments, and some in- 3.. When you're finished with your review, take vite you to participate, either as a subject or as an the posttest to confirm that you're ready to experimenter. move on to the next chapter. William D. Hacker Social Science WEBSITE FOR THE PRACTICE Experimental Laboratory OF SOCIAL RESEARCH 11TH EDITION http://sseLcaItech.edu/ This laboratory at the California Institute of Technol- Go to your book's website at http://sociology ogy gives students the opportunity to participate as .wadsworth.com/babbie_practicelle for tools to subjects in experiments online-for pay. aid you in studying for your exams. You'll find Tlllo- rial Qllizzes with feedback, Intemet Exercises, Flashcards, Stanford Prison Experiment and Chapter Tutorials, as well as Exte/lded Projects, Info- http://vvww.prisonexp.org/ Trac College Edition search terms, Sodal Research ill Cyber- This website provides a slide show relating a famous space, GSS Data, Web Lillks, and primers for using vari- social science experiment that reveals some of the ous data-analysis software such as SPSS and ]\\,'Vivo. problems that can occur in this type of study. WEB LINKS FOR THIS CHAPTER Please realize that the Internet is an evolving entity, subject to change. Nevertheless, these few websites should be fairly stableo Also, check your book's website for even more Web Lillks. These websites, current at the time of this book's publication, provide opportunities to learn about experiments.

Survey Research Introduction Self-Administered Questionnaires Mail Distribution and Return Topics Appropriate Monitoring Returns for Survey Research Follow-up Mailings Acceptable Response Rates Guidelines for Asking Questions A Case Study Choose Appropriate Question Interview Surveys Forms The Role of the Survey Interviewer Make Items Clear General Guidelines Avoid Double·Barreled Questions for Survey Interviewing Respondents Must Be Competent Coordination and Control to Answer Respondents Must Be Willing Telephone Surveys to Answer Computer Assisted Telephone Questions Should Be Relevant Interviewing (CAn) Short Items Are Best Response Rates in Interview Avoid Negative Items Surveys Avoid Biased Items and Terms New Technologies and Questionnaire Construction Survey Research General Questionnaire Format Comparison of the Different Formats for Respondents Survey j\\lethods Contingency Questions Matrix Questions Strengths and Weaknesses of Ordering Items in a Questionnaire Survey Research Questionnaire Instructions Pretesting the Questionnaire Secondary Analysis A Composite Illustration Sociology~ Now''': Research Methods Use this online tool to help you make the grade on your next exam. After reading this chapter, go to the \"Online Study Resources\" at the end of the chapter for instructions on how to benefit from SodologyNow: Research lvlel/lOds.

244 Chapter 9: Survey Research Introduction options for administering it so that respondents an- swer your questions adequately. Surveys are a very old research techniqueo In the Old Testament, for example, we find the following: The chapter concludes with a short discussion of secondmy mzaZysis, the analysis of survey data col- After the plague the Lord said to Moses and to lected by someone else. This use of survey results Eleazar the son of Aaron, the priest, \"Take a has become an important aspect of survey research census of all the congregation of the people of in recent years, and it's especially useful for stu- IsraeL from twenty old and upwardo\" dents and others with scarce research funds. (NlllI/bers 261-2) Let's begin by looking at the kinds of topics that researchers can appropriately study by using survey Ancient Egyptian rulers conducted censuses to research. help them administer their domains. Jesus was born away from home because Joseph and Mary Topics Appropriate were journeying to Joseph's ancestral home for a for Survey Research Roman census. Surveys may be used for descTiptive, explanatory, A little-known survey was attempted among and eX\"ploratory purposes. They are chiefly used in French workers in 1880. A German political sociol- studies that have individual people as the units of ogist mailed some 25,000 questionnaires to work- analysiso Although this method can be used for ers to determine the extent of their exploitation by other units of analysis, such as groups or interac- employers. The rather lengthy questionnaire in- tions, some individual persons must serve as cluded items such as these: respondents or informants. Thus, we could un- dertake a survey in which divorces were the unit of Does your employer or his representative resort analysis, but we would need to administer the sur- to trickery in order to defraud you of a part of vey questionnaire to the participants in the di- your earnings? vorces (or to some other informants). If you are paid piece rates, is the quality of Survey research is probably the best method the article made a pretext for fraudulent deduc- available to the social researcher who is interested tions from your wages? in collecting original data for describing a popula- tion too large to observe directly. Careful probabil- The survey researcher in this case was not ity sampling provides a group of respondents George Gallup but Karl Marx ([1880]1956: 208). whose characteristics may be taken to reflect those Though 25,000 questionnaires were mailed out, of the larger population, and carefully constructed there is no record of any being returned. standardized questionnaires provide data in the same form from all respondents. Today, survey research is a frequently used mode of observation in the sodal sciences. In a typ- Surveys are also excellent vehicles for measur- ical survey, the researcher selects a sample of re- ing attitudes and orientations in a large population. spondents and administers a standardized question- Public opinion polls-for example, Gallup, Harris, naire to them. Chapter 7 discussed sampling Roper, and Yankelovich-are well-known exam- techniques in detaiL This chapter discusses how to ples of this useo Indeed, polls have become so preva- prepare a questionnaire and describes the various lent that at times the public seems unsure what to think of them. Pollsters are criticized by those who respondent A person who provides data for analy- don't think (or want to believe) that polls are sis by responding to a survey questionnaire.

Guidelines for Asking Questions 245 accurate (candidates who are \"losing\" in polls often Opinion Polling had to say in condemning this tell voters not to trust the polls). But polls are also practice: criticized for being too accurate-for example, when exit polls on election day are used to predict a win- A \"push poll\" is a telemarketing technique in ner before the actual voting is complete. which telephone calls are used to canvass po- tential voters, feeding them false or misleading The general attitude toward public opinion re- \"information\" about a candidate under the pre- search is further complicated by scientifically un- tense of taking a poll to see how this \"informa- sound \"surveys\" that nonetheless capture people's tion\" affects voter preferences. In fact, the in- attention because of the topics they cover and/or tent is not to measure public opinion but to their \"findings.\" A good example is the \"Hite Re- manipulate it-to \"push\" voters away from ports\" on human sexualityo While enjoying consid- one candidate and toward the opposing candi- erable attention in the popular press, Shere Hite date. Such polls defame selected candidates by was roundly criticized by the research community spreading false or misleading information about for her data-collection methods. For example, a them, The intent is to disseminate campaign 1987 Hite report was based on questionnaires com- propaganda under the guise of conducting a pleted by women around the country-but which legitimate public opinion polL women? Hite reported that she distributed some 100,000 questionnaires through various organiza- (Bednarz 1996) tions, and around 4,500 were returned. In short, the labels \"survey\" and \"poll\" are Now 4,500 and 100,000 are large numbers in sometimes misusedo Done properly, however, sur- the context of survey sampling. However, given vey research can be a useful tool of social inquiry. Hite's research methods, her 4,500 respondents Designing useful (and trustworthy) survey research didn't necessarily represent US, women any more begins with formulating good questions. Let's turn than the Literary Digests enormous 1936 sample to that topic nowo represented the U.So electorate when their 2 mil- lion sample ballots indicated that Alf Landon would Guidelines bury FDR in a landslide. for Asking Questions Sometimes, people use the pretense of survey In sodal research, variables are often operational- research for quite different purposes. For example, ized when researchers ask people questions as a you may have received a telephone call indicating way of getting data for analysis and interpretation. you've been selected for a survey, only to find that Sometimes the questions are asked by an inter- the first question was \"How would you like to viewer; sometimes they are written down and make thousands of dollars a week right there in given to respondents for completion. In other cases, your ovvn home?\" Or you may have been told you several general guidelines can help researchers could win a prize if you could name the president frame and ask questions that serve as excellent whose picture is on the penny, (Tell them it's Elvis.) operationalizations of variables while avoiding pit- Unfortunately, a few unscrupulous telemarketers falls that can result in useless or even misleading try to prey on the general cooperation people have information. given to survey researchers. Surveys include the use of a questionnaire- By the same token, political parties and charita- an instrument specifically designed to elicit infor- ble organizations have begun conducting phony mation that ,viII be useful for analysis. Although \"surveys.\" Often under the guise of collecting pub- some of the specific points to follow are more ap- lic opinion about some issue, callers ultimately ask propriate to structured questionnaires than to the respondents for a monetary contribution. Recent political campaigns have produced another form of bogus survey, the \"push polL\" Here's what the American Association for Public

246 Chapter 9: Survey Research more open-ended questionnaires used in qualita- Both questions and statements can be used tive, in-depth interviewing, the underlying logic is profitably. Using both in a given questionnaire valuable whenever we ask people questions in or- gives you more flexibility in the design of items der to gather data. and can make the questionnaire more interesting as welL Choose Appropriate Question Forms Open-Ended and Closed-Ended Questions Let's begin with some of the options available to In asking questions, researchers have two options. you in creating questionnaires. These options in- They can ask open-ended questions, in which clude using questions or statements and choosing case the respondent is asked to provide his or her open-ended or closed-ended questions. own answer to the question. For example, the respondent may be asked, \"What do you feel is the Questions and Statements most in1portant issue facing the United States to- day?\" and be provided with a space to write in the Although the term questionnaire suggests a collec- answer (or be asked to report it verbally to an in- tion of questions, an examination of a typical ques- terviewer). As we'll see in Chapter 10, in-depth, tionnaire will probably reveal as many statements qualitative interviewing relies almost exclusively on as questions. This is not without reason. Often, the open-ended questions. However, they are also used researcher is interested in determining the extent in survey research. to which respondents hold a particular attitude or perspective. If you can summarize the attitude in a In the case of closed-ended questions, the fairly brief statement, you can present that state- respondent is asked to select an answer from ment and ask respondents whether they agree or among a list provided by the researcher, Closed- disagree with it. As you may remember, Rensis Lik- ended questions are very popular in survey re- ert greatly formalized this procedure through the search because they provide a greater uniformity of creation of the Likert scale, a format in which re- responses and are more easily processed than spondents are asked to strongly agree, agree, dis- open-ended ones. agree, or strongly disagree, or perhaps strongly approve, approve, and so forth. Open-ended responses must be coded before they can be processed for computer analysis, as questionnaire A document containing questions we'll see in Chapter 14. This coding process often and other types of items designed to solicit informa- requires the researcher to interpret the meaning of tion appropriate for analysis. Questionnaires are used responses, opening the possibility of misunder- primarily in survey research but also in eXlleriments, standing and researcher bias. There is also a danger field research, and other modes of observation. that some respondents will give answers that are open-ended questions Questions for which the essentially irrelevant to the researcher'S intent. respondent is asked to provide his or her own an- Closed-ended responses, on the other hand, can swers. In·depth, qualitative interviewing relies al- often be transferred directly into a computer most exclusively on open-ended questions. formal. closed-ended questions Survey questions in which the respondent is asked to select an answer The chief shortcoming of closed-ended ques- from among a list provided by the researcher. Popu- tions lies in the researcher'S structuring of re- lar in survey research because they provide a greater sponses. When the relevant answers to a given uniformity of responses and are more easily question are relatively clear, there should be no processed than open-ended questions. problem. In other cases, however, the researcher'S structuring of responses may overlook some impor- tant responses. In asking about \"the most important issue facing the United States,\" for example, his or

Guidelines for Asking Questions 247 her checklist of issues might omit certain issues that CPS, which measures, among other critical data, the nation'S unemployment rate. A part of the mea- respondents would have said were important. surement of employment patterns focuses on a re- spondent's activities during \"last week,\" by which The construction of closed-ended questions the Census Bureau means Sunday through Satur- day. Studies undertaken to deterntine the accuracy should be guided by two structural requirements. of the survey found that more than half the re- spondents took \"last week\" to include only Mon- First, the response categories provided should be day through Friday. By the same token, whereas the Census Bureau defines \"working full-tin1e\" as exhaustive: They should include all the possible re- 35 or more hours a week, the same evaluation studies showed that some respondents used the sponses that might be ex-pected. Often, researchers more traditional definition of 40 hours per week. As a consequence, the wording of these questions ensure this by adding a category such as \"Other in the CPS was modified in 1994 to specify the Census Bureau's definitions. (Please specify: ).\" Second, the answer cat- Similarly, the use of the term Narive American to egories must be mutually exclusive: The respon- mean American Indian often produces an overrepre- sentation of that ethnic group in surveys. Clearly, dent should not feel compelled to select more than many respondents understand the term to mean \"born in the United States.\" one. (In some cases, you may wish to solicit mul- Avoid Double-Barreled Questions tiple answers, but these may create difficulties in Frequently, researchers ask respondents for a single data processing and analysis later on.) To ensure answer to a question that actually has multiple parts. That seems to happen most often when the that your categories are mutually exclusive, care- researcher has personally identified with a complex question. For example, you might ask respondents fully consider each combination of categories, ask- to agree or disagree with the statement \"The United States should abandon its space program and spend ing yourself whether a person could reasonably the money on domestic programs.\" Although many people would unequivocally agree v'lith the state- choose more than one answer. In addition, it's use- ment and others would unequivocally disagree, still others would be unable to answer. Some would ful to add an instruction to the question asking the vvant to abandon the space program and give the money back to the taxpayers. Others would want respondent to select the one best answer, but this to continue the space program but also put more money into domestic programs. These latter re- technique is not a satisfactory substitute for a care- spondents could neither agree nor disagree without misleading you. fully constructed set of responses. As a general rule, whenever the word and ap- Make Items Clear pears in a question or questionnaire statement, check whether you're asking a double-barreled It should go without saying that questionnaire question. See \"Double-Barreled and Beyond\" for items need to be clear and unambiguous, but the some imaginative variations on this theme. broad proliferation of unclear and ambiguous ques- tions in surveys makes the point worth emphasiz- ing. We can become so deeply involved in the topic under examination that opinions and perspectives are clear to us but not to our respondents-many of whom have paid little or no attention to the topic. Or, if we have only a superficial understand- ing of the topic, we may fail to specify the intent of a question sufficiently. The question \"What do you think about the proposed peace plan?\" may evoke in the respondent a counterquestion: \"Which pro- posed peace plan?\" Questionnaire items should be precise so that the respondent knows exactly what the researcher is asking. The possibilities for misunderstanding are end- less, and no researcher is immune (Polivka and Rothgeb 1993). One of the most established re- search projects in the United States is the Census Bureau's ongoing \"Current Population Survey\" or

248 Chapter 9: Survey Research Even established, professional researchers have sometimes created u.s. will War isprobable double-barreled questions and worse. Consider this question, asked butnot Waris of US citizens in April 1986, at atime when the country's relationship notgo inevitable inevitable with Libya was at an especially low point Some observers suggested to war that the United States might end up in ashooting war with the small 2 North African nation. The Harris Poll sought to find out what US.public U.s. will not invade Libya opinion was. 45 US.will invade Libya but it If Libya now increases its terrorist acts againstthe US.and we keep would be wrong 67 infiicting more damage on Libya, then inevitably it will all end in the US going to war and finally invading that country which US will invade Libya and it would be wrong would be right Respondents were given the opportunity ofanswering\"Agree,\" The examination ofprognoses aboutthe Libyan situation is notthe \"Disagree,\" or\"Not sure.\" Notice the elements contained in the complex only example ofdouble-barreled questions sneaking into public opinion statement research Here are some questions the Harris Poll asked in an attemptto gauge U.s.public opinion about then Soviet General Secretary Gorbachev: 1. Will Libya increase its terrorist acts against the US? 2 Will the U.S.infiict more damage on Libya? He looks like the kind of Russian leader who will recognize that 3 Will the U.S. inevitably or otherwise go to war against Libya? both the Soviets and the Americans can destroy each other with 4. Would the US invade Libya? nuclear missiles so it is better to come to verifiable arms control 5 Would that be right or wrong? agreements. These several elements offer the possibility of numerous points of He seems to be more modern, enlightened,and attractive, view-far more than the three alternatives offered to the survey re- which is agood sign for the peace ofthe world. spondents Even if we were to assume hypothetically that Libya would \"increase its terrorist attacks\"and the United States would\"keep infiicting Even though he looks much more modern and attractive,it more damage\"in return, you might have anyone ofat least seven dis- would be amistake to think he will be much different from other tinct expectations about the outcome Russian leaders. How many elements can you identify in each ofthe questions? How many possible opinions could people have in each case? What does asimple\"agree\"or\"disagree\"really mean in such cases? Source. Reported in World Opi,~ion UpdoiE,October 1985 and May 1986, respectively Respondents Must Be how students' fees ought to be spent. Typically, re- Competent to Answer spondents are asked to indicate the percentage of available funds that should be devoted to a long list In asking respondents to provide information, you of activities. Without a fairly good knowledge of should continually ask yourself whether they can the nature of those activities and the costs involved do so reliably In a study of child rearing, you might in them, the respondents cannot provide meaning- ask respondents to report the age at which they ful answers. Administrative costs, for example, will first talked back to their parents. Quite aside from receive little support although they may be essen- the problem of defining talkillg back to parents, it's tial to the program as a whole. doubtful that most respondents would remember with any degree of accuracy. One group of researchers examining the driv- ing experience of teenagers insisted on asking an As another example, student government lead- open-ended question concerning the number of ers occasionally ask their constituents to indicate miles driven since receiving a license. Although

Guidelines for Asking Questions 249 consultants argued that few drivers would be able the ridicule of a disbelieving majority, but en- to estimate such information with any accuracy, forced separatism diminishes opportunities for the question was asked nonetheless. In response, recruitment and information exchange... , , some teenagers reported driving hundreds of thou- sands of miles. \"Secretive\" survivalists eschew telephones, launder their mail through letter exchanges, Respondents Must Be use nicknames and aliases, and carefully con- Willing to Answer ceal their addresses from strangers. Yet once I was invited to group meetings, I found them Often, we would like to learn things from people cooperative respondents. that they are unwilling to share with us. For ex- ample, Yanjie Bian indicates that it has often been Questions Should Be Relevant difficult to get candid answers from people in China. Similarly, questions asked in a questionnaire should [Here] people are generally careful about what be relevant to most respondents. When attitudes they say on nonprivate occasions in order to are requested on a topic that few respondents have survive under authoritarianismo During the thought about or really care about, the results are Cultural Revolution between 1966 and 1976, not likely to be usefuL Of course, because the re- for example, because of the radical political spondents may express attitudes even though agenda and political intensity throughout the they've never given any thought to the issue, you country, it was almost impossible to use survey run the risk of being misled. techniques to collect valid and reliable data in- side China about the Chinese people's life expe- This point is illustrated occasionally when re- riences, characteristics, and attitudes towards searchers ask for responses relating to fictitious the Communist regime, people and issues. In one political poll I conducted, I asked respondents whether they were familiar (1994.19-20) with each of 15 political figures in the community. As a methodological exercise, I made up a name: Sometimes, U.S. respondents say they're unde- Tom Sakumoto. In response, 9 percent of the re- cided when, in fact, they have an opinion but think spondents said they were familiar with him. Of they're in a minority. Under that condition, they those respondents familiar ,vith him, about half re- may be reluctant to tell a stranger (the interviewer) ported seeing him on television and reading about what that opinion is. Given this problem, the Gallup him in the newspapers. Organization, for example, has used a \"seCTet ballot\" format, which simulates actual election conditions, When you obtain responses to fictitious issues, in that the \"voter\" enjoys complete anonymity. In you can disregard those responses. But when the is- an analysis of the Gallup Poll election data from sue is real, you may have no way of telling which re- 1944 to 1988, Andrew Smith and G. E Bishop sponses genuinely reflect attitudes and which reflect (1992) have found that this technique substantially meaningless answers to an irrelevant question. reduced the percentage of respondents who said they were undecided about how they would voteo Ideally, we would like respondents to simply report that they don't know, have no opinion, or This problem is not limited to survey research, are undecided in those instances where that is the howevec Richard Mitchell (1991: 100) faced a case. Unfortunately, however, they often make up similar problem in his field research among U.S. answers. survivalists: Short Items Are Best Survivalists, for example, are ambivalent about concealing their identities and inclinations. In the interests of being unambiguous and precise They realize that secrecy protects them from and of pointing to the relevance of an issue, re- searchers tend to CTeate long and complicated

250 Chapter 9: Survey Research items. That should be avoided. Respondents are of- In 1993 a national survey conmlissioned by the ten unwilling to study an item in order to under- Anlerican Jewish Committee produced shocking stand it The respondent should be able to read an results: One American in five believed that the Nazi item quickly, understand its intent and select or Holocaust-in which six million Jews were report- provide an answer without difficulty. In general, edly killed-never happened; further, one in three assume that respondents will read items quickly Anlericans expressed some doubt that it had oc- and give quick answers. Accordingly, provide clear, curred. This research finding suggested that the short items that will not be misinterpreted under Holocaust Revisionist movement in America was those conditions. powerfully influencing public opinion (\"1 in 5 Polled Voices Doubt on Holocaust\" 1993). Avoid Negative Items In the aftermath of this shocking news, re- The appearance of a negation in a questionnaire searchers reexamined the actual question that had item paves the way for easy misinterpretation. been asked: \"Does it seem possible or does it seem Asked to agree or disagree with the statement \"The impossible to you that the Nazi extermination of United States should not recognize Cuba,\" a sizable the Jews never happened?\" On reflection, it portion of the respondents will read over the word seemed clear that the complex, double-negative nor and answer on that basis. Thus, some will agree question could have confused some respondents. with the statement when they're in favor of recog- nition, and others \"viII agree when they oppose it A new survey was commissioned and asked, And you may never know which are which. \"Does it seem possible to you that the Nazi extermi- nation of the Jews never happened, or do you feel Similar considerations apply to other \"negative\" certain that it happened?\" In the follow-up survey, words. In a study of support for civil liberties, for only 1 percent of the respondents believed the example, respondents were asked whether they Holocaust never happened, and another 8 percent felt \"the following kinds of people should be prohib- said they weren't sure (\"Poll on Doubt of Holocaust ired from teaching in public schools\" and were pre- Is Corrected\" 1994). sented 'with a list including such items as a Com- munist, a Ku Klux Klansman, and so forth. The Avoid Biased Items and Terms response categories \"yes\" and \"no\" were given be- side each entry. A comparison of the responses to Recall from our discussion of conceptualization and this item vvith other items reflecting support for operationalization in Chapter 5 that there are no ul- civil liberties strongly suggested that many respon- timately tme meanings for any of the concepts we dents gave the answer \"yes\" to indicate willingness typically study in social science. Prejudice has no ul- for such a person to teach, rather than to indicate timately correct definition; whether a given person that such a person should be prohibited from is prejudiced depends on our definition of that term. teaching. (A later study in the series using the an- The same general principle applies to the responses swer categories \"permit\" and \"prohibit\" produced we get from people completing a questionnaire. much clearer results.) The meaning of someone's response to a ques- bias That quality of a measurement device that tion depends in large part on its wording. This is tends to result in a misrepresentation of what is be- true of every question and answer. Some questions ing measured in a particular direction. For example, seem to encourage particular responses more than the questionnaire item \"Don't you agree that the other questions do. In the context of questionnaires, president is doing a good job?\" would be biased in bias refers to any property of questions that en- that it would generally encourage more favorable courages respondents to answer in a particular way. responses. Most researchers recognize the likely effect of a question that begins, \"Don't you agree with the President of the United States that ...\" No reputable

Questionnaire Construction 251 researcher would use such an item. Unhappily, the however, he may want to avoid looking like a biasing effect of items and terms is far subtler than chauvinist. Recognizing that his views are out this example suggests. of step with current thinking, he may choose to say \"yes.\" The mere identification of an attitude or posi- tion \\vith a prestigious person or agency can bias The best way to guard against this problem is to responses. The item \"Do you agree or disagree with imagine how you would feel giving each of the an- the recent Supreme Court decision that ...\" would swers you intend to offer to respondents. If you have a similar effect. Such wording may not pro- would feel embarrassed, perverted, inhumane, stu- duce consensus or even a majority in support of pid, irresponsible, or otherwise socially disadvan- the position identified with the prestigious person taged by any particular response, give serious or agency, but it will likely increase the level of thought to how willing others will be to give those support over what would have been obtained with- answers . out such identification. The biasing effect of particular wording is often Sometimes the impact of different forms of difficult to anticipate. For example, in both surveys question wording is relatively subtle. For example, and experiments, researchers sometimes ask re- when Kenneth Rasinski (1989) analyzed the re- spondents to consider hypothetical situations and sults of several General Social Survey studies of at- say how they think they would behave. Those situ- titudes toward government spending, he found that ations often involve other people, however, and the the way programs were identified had an impact names used can affect responses. For instance, re- on the amount of public support they received. searchers have long known that male names for Here are some comparisons: the hypothetical people can produce different re- sponses than female names do. Research by Joseph More Support Less Support Kasof (1993) points to the importance of what the specific names are: whether they generally evoke \"Assistance to the poor\" \"Welfare\" positive or negative images in terms of attractive- \"Halting rising crime rate\" \"Law enforcement\" ness, age, intelligence, and so forth. Kasof's review \"Dealing with drug addiction\" \"Drug rehabilitation\" of past research suggests there has been a tendency \"Solving problems of big cities\" \"Assistance to big cities\" to use more positively valued names for men than \"Improving conditions of blacks\" \"Assistance to blacks\" for women. \"Protecting social security\" \"Social security\" As in all other research, carefully examine the In 1986, for example, 62.8 percent of the respon- purpose of your inquiry and construct items that dents said too little money \\vas being spent on \"as- will be most useful to it. You should never be mis- sistance to the poor,\" whereas in a matched survey led into thinking there are ultimately \"right\" and that year, only 23.1 percent said we were spending \"wrong\" ways of asking the questions. When in too little on \"welfare.\" doubt about the best question to ask, moreover, re- member that you should ask more than one. In this context be wary of what researchers call the sodal desirabiliry of questions and answers. These, then, are some general guidelines for Whenever we ask people for information, they an- writing questions to elicit data for analysis and in- swer through a filter of what will make them look terpretation. Next we look at how to construct good. This is especially true if they're interviewed questionnaires. face-to-face. Thus, for example, a particular man may feel that things would be a lot better if women Questionnaire Construction were kept in the kitchen, not allowed to vote, forced to be quiet in public, and so forth. Asked Questionnaires are used in connection vvith many whether he supports equal rights for women, modes of observation in sodal research. Although structured questionnaires are essential to and most

252 Chapter 9: Survey Research directly associated with survey research, they are what respondents to a self-administered question- also widely used in experiments, field research, and naire would need. other data-collection activities. For this reason, questionnaire construction can be an important The desirability of spreading out questions in practical skill for researchers. As we discuss the es- the questionnaire cannot be overemphasized. tablished teclmiques for constructing question- Squeezed-together questionnaires are disastrous, naires, let's begin with some issues of questionnaire whether completed by the respondents themselves formal. or administered by trained interviewers. The pro- cessing of such questionnaires is another night- mare; I'll have more to say about that in Chapter 14. General Questionnaire Format Formats for Respondents The format of a questionnaire is just as important In one of the most common types of questionnaire as the nature and wording of the questions asked. items, the respondent is expected to check one re- An improperly laid out questionnaire can lead re- sponse from a series. For this purpose my experi- spondents to miss questions, confuse them about ence has been that boxes adequately spaced apart the nature of the data desired, and even lead them are the best formaL Modern word processing makes to throw the questionnaire away. the use of boxes a practical technique these days; setting boxes in type can also be accomplished eas- As a general rule, a questionnaire should be ily and neatly. You can approximate boxes by using spread out and uncluttered. If a self-administered brackets: [ j, but if you're creating a questionnaire questionnaire is being designed, inexperienced re- on a computer, you should take the few extra nun- searchers tend to fear that their questionnaire will utes to use genuine boxes that will give your ques- look too long; as a result, they squeeze several tionnaire a more professional look. Here are some questions onto a single line, abbreviate questions, easy examples: and try to use as few pages as possible. These ef- forts are ill-advised and even dangerous. Putting D0 0 more than one question on a line will cause some respondents to miss the second question altogethec Rather than providing boxes to be checked, you Some respondents will misinterpret abbreviated might print a code number beside each response questions. More generally, respondents who find and ask the respondent to circle the appropriate they have spent considerable time on the first page number (see Figure 9-1). This method has the of what seemed like a short questionnaire will be added advantage of specifying the code number to more demoralized than respondents who quickly be entered later in the processing stage (see Chap- complete the first several pages of what initially ter 14). If numbers are to be circled, however, you seemed like a rather long form. Moreover, the lat- should provide clear and prominent instructions to ter will have made fewer errors and will not have the respondent, because many vvill be tempted to been forced to reread confusing, abbreviated ques- cross out the appropriate number, which makes tions. Nor vvill they have been forced to write a data processing more difficult. (Note that the tech- long answer in a tiny space. nique can be used more safely when interviewers administer the questionnaires, because the inter- Sinular problems can arise for interviewers in a viewers themselves record the responses.) face-to-face or telephone interview. Like respon- dents to a self-administered questionnaire, inter- Contingency Questions viewers may miss questions, lose their place, and generally become frustrated and flustered. Inter- Quite often in questionnaires, certain questions view questionnaires need to be laid out in a way \\·vill be relevant to some of the respondents and that supports the interviewer'S work, including spe- irrelevant to others. In a study of birth control cial instructions and guidelines that go beyond

Questionnaire Construction 253 Did you happen to vote in the last presidential 23. Have you ever smoked marijuana? election? DYes ONo (DYes If yes: About how many times have 2. No you smoked marijuana? 3. Don't know o Once Have you ever felt you were the victim of 02t05times sexual discrimination? 06 to 10 times 1. Yes 011 to 20 times ®NO o More than 20 times 3. Don't know FIGURE 9·2 FIGURE 9-1 Contingency Question FOmlat. Contingency questions offer a Circling the Answer structure for exploring subject areas logically in some depth. methods, for instance, you would probably not Note that the questions shown in Figure 9-2 want to ask men if they take birth control pills. could have been dealt with in a single question. The question might have read, \"How many times, if This sort of situation often arises when re- any, have you smoked marijuana?\" The response searchers wish to ask a series of questions about a categories, then, might have read: \"Never,\" \"Once,\" certain topic. You may want to ask whether your \"2 to 5 times,\" and so forth. This single question respondents belong to a particular organization would apply to all respondents, and each would and, if so, how often they attend meetings, find an appropriate answer category. Such a ques- whether they have held office in the organization, tion, however, might put some pressure on respon- and so forth. Or, you might want to ask whether dents to report having smoked marijuana, because respondents have heard anything about a certain the main question asks how many times they have political issue and then learn the attitudes of those smoked it, even though it allows for those excep- who have heard of it. tional cases who have lIever smoked marijuana even alice. (The emphases used in the previous sentence Each subsequent question in series such as give a fair indication of how respondents might these is called a contingency question: Whether read the question.) The contingency question for- it is to be asked and answered is contingent on re- mat illustrated in Figure 9·2 should reduce the sponses to the first question in the series. The subtle pressure on respondents to report having proper use of contingency questions can facilitate smoked marijuana. the respondents' task in completing the question- naire, because they are not faced with trying to an- contingency question A survey question intended swer questions irrelevant to them. for only some respondents, detemlined by their re- sponses to some other question. For example, all re- There are several formats for contingency ques- spondents might be asked whether they belong to tions. The one shown in Figure 9-2 is probably the the Cosa Nostra, and only those who said yes would clearest and most effective. Note two key elements be asked how often they go to company meetings in this format. First, the contingency question is and picnics. The latter would be a contingency isolated from the other questions by being set off to question. the side and enclosed in a box. Second, an arrow connects the contingency question to the answer on which it is contingent. In the illustration, only those respondents answering yes are expected to answer the contingency question. The rest of the respondents should simply skip it.

254 Chapter 9: Survey Research 24. Have you ever been abducted by aliens? 13. Have you ever voted in a national, state, or DYes local election? ONo o Yes (Please answer questions 14-25.) If yes: Did they let you steer the ship? o No (Please skip questions 14-25. Go directly DYes No to question 26 on page 8.) If yes: How fast did you go? FIGURE 9-4 Instructions to Skip o Warp speed increase the likelihood of responses from those for o Weenie speed whom the questions are relevant. FIGURE 9-3 Matrix Questions Contingency Table. Sometimes it will be appropriate for certain Quite often, you'll want to ask several questions kinds of respondents to skip over inapplicable questions. To that have the same set of answer categories. This is avoid confusion, you should be sure to provide clear instruc- typically the case whenever the Likert response tions to that end. categories are used. In such cases, it is often pos- sible to construct a matrix of items and answers as Used properly, even rather complex sets of con- illustrated in Figure 9-5. tingency questions can be constructed without confusing the respondent. Figure 9- 3 illustrates a This format offers several advantages over other more complicated example. formats. First. it uses space efficiently. Second, re- spondents vvill probably find it faster to complete a Sometimes a set of contingency questions is set of questions presented in this fashion than in long enough to extend over several pages. Suppose other ways. In addition, this format may increase you're studying political activities of college stu- the comparability of responses given to different dents, and you wish to ask a large number of ques- questions for the respondent as well as for the re- tions of those students who have voted in a na- searcher. Because respondents can quickly review tionaL state, or local election. You could separate their answers to earlier items in the set. they out the relevant respondents ,\"lith an initial ques- might choose between, say, \"strongly agree\" and tion such as \"Have you ever voted in a nationaL \"agree\" on a given statement by comparing the state, or local election?\" but it would be confusing strength of their agreement with their earlier to place the contingency questions in a box stretch- responses in the set. ing over several pages. It would make more sense to enter instructions, in parentheses after each an- There are some dangers inherent in using this swer. telling respondents to answer or skip the con- format. however. Its advantages may encourage tingency questions. Figure 9-4 provides an illustra- you to structure an item so that the responses fit tion of this method. into the matrix format when a different. more idio- syncratic set of responses might be more appropri- In addition to these instructions, it's worth- ate. Also, the matrix question format can foster a while to place an instruction at the top of each response-set among some respondents: They may page containing only the contingency questions. develop a pattern of. say, agreeing with all the state- For example, you might say, \"This page is only for ments. This would be especially likely if the set of respondents who have voted in a nationaL state, or statements began with several that indicated a par- local election.\" Clear instructions such as these ticular orientation (for example, a liberal political spare respondents the frustration of reading and perspective) with only a few later ones representing puzzling over questions irrelevant to them and

Questionnaire Construction 255 17. Beside each of the statements presented below, please indicate whether you Strongly Agree (SA), Agree (A), Disagree (D), Strongly Disagree (SO), or are Undecided (U). a. What this country needs SA A 0 SO U is more law and order.......... 0 0 0 0 0 b. The porice should be 0000 disarmed in America........... c. During riots, looters 0 00 should be shot on sight. ........ etc. FIGURE 9-5 Matrix Question Format. Matrix questions offer an efficient format for presenting aset of closed-ended questionnaire items that have the same response categories. the oppOsite orientation. Respondents might as- of responses vvithin a question can also make a dif- sume that all the statements represented the same ference (Bishop and Smith 2001) . orientation and, reading quickly, misread some of them, thereby giving the wrong answers. This The impact of item order is not uniform. When problem can be reduced somewhat by alternating J. Edwin Benton and John Daly (1991) conducted statements representing different orientations and a local government survey, they found that the by making all statements short and clear. less-educated respondents were more influenced by the order of questionnaire items than those with Ordering Items in a Questionnaire more education were. The order in which questionnaire items are pre- Some researchers attempt to overcome this ef- sented can also affect responses. First. the appear- fect by randomizing the order of items. This effort ance of one question can affect the answers given is usually futile. In the first place, a randomized set to later ones. For example, if several questions have of items will probably strike respondents as chaotic been asked about the dangers of terrorism to the and worthless. The random order also makes it United States and then a question asks respondents more difficult for respondents to answer, because to volunteer (open-ended) what they believe to they must continually switch their attention from represent dangers to the United States, terrorism one topic to another. Finally, even a randomized will receive more citations than would otherwise be ordering of items will have the effect discussed the case. In this situation, it's preferable to ask the previously-except that you'll have no control over open-ended question first. the effect. Similarly, if respondents are asked to assess The safest solution is sensitivity to the problem. their overall religiosity (\"How important is your re- Although you cannot avoid the effect of item order, ligion to you in genera!?\"), their responses to later try to estimate what that effect will be so that you questions concerning specific aspects of religiosity can interpret results meaningfully. If the order of will be aimed at consistency with the prior assess- items seems especially important in a given study, ment. The converse is true as well. If respondents you might construct more than one version of the are first asked specific questions about different as- questionnaire with different orderings of the items. pects of their religiosity, their subsequent overall as- You will then be able to determine the effects by sessment will reflect the earlier answers. The order comparing responses to the various versions. At the very least. you should pretest your questionnaire in the different forms. (We'll discuss pretesting in a moment.)

256 Chapter 9: Survey Research The desired ordering of items differs between used, respondents should be given some guidelines interviews and self-administered questionnaires. In about whether brief or lengthy answers are ex- the latter, it's usually best to begin the question- pected. If you vvish to encourage your respondents naire vvith the most interesting set of items. The to elaborate on their responses to closed-ended potential respondents who glance casually over the questions, that should be noted. first few items should want to answer them. Per- haps the items will ask for attitudes they're aching If a questionnaire has SUbsections-political to express. At the same time, however, the initial attitudes, religious attitudes, background data- items should not be threatening. (It might be a bad introduce each with a short statement concerning idea to begin with items about sexual behavior or its content and purpose. For example, \"In this sec- drug use.) Requests for duller, demographic data tion, we would like to know what people consider (age, gender, and the like) should generally be the most important community problems.\" Demo- placed at the end of a self-administered question- graphic items at the end of a self-administered naire. Placing these items at the beginning, as questionnaire might be introduced thus: \"Finally, many inexperienced researchers are tempted to do, we would like to know just a little about you so we gives the questionnaire the initial appearance of a can see how different types of people feel about the routine form, and the person receiving it may not issues we have been examining.\" be motivated to complete iL Short introductions such as these help the re- Just the opposite is generally true for interview spondent make sense of the questionnaire. They surveys. When the potential respondent's door first make the questionnaire seem less chaotic, espe- opens, the interviewer must gain rapport quickly. cially when it taps a variety of data. And they help After a short introduction to the study, the inter- put the respondent in the proper frame of mind for viewer can best begin by enumerating the members answering the questions. of the household, getting demographic data about each. Such items are easily answered and generally Some questions may require special instruc- nonthreatening. Once the initial rapport has been tions to facilitate proper answering. This is espe- established, the interviewer can then move into the cially true if a given question varies from the area of attitudes and more sensitive matters. An in- general instructions pertaining to the whole ques- terview that began with the question \"Do you be- tionnaire. Some specific examples will illustrate this lieve in vvitchcraft?\" would probably end rather situation. quickly. Despite attempts to provide mutually exclusive Questionnaire Instructions answers in closed-ended questions, often more than one answer \\vill apply for respondents. If you Every questionnaire, whether it is to be completed want a single answer, you should make this per- by respondents or administered by interviewers, fectly clear in the question. An example would be should contain clear instructions and introductory \"From the list below, please check the primary rea- comments where appropriate. son for your decision to attend college.\" Often the main question can be followed by a parenthetical It's useful to begin every self-administered ques- note: \"Please check the one best answer.\" If, on the tionnaire with basic instructions for completing it. other hand, you want the respondent to check as Although many people these days have eX'Perience many answers as apply, you should make this clear. with forms and questionnaires, begin by telling them exactly what you want: that they are to indi- When a set of answer categories are to be rank- cate their answers to certain questions by placing a ordered by the respondent, the instructions should check mark or an X in the box beside the appropri- indicate this, and a different type of answer format ate answer or by writing in their answer when should be used (for example, blanks instead of asked to do so. If many open-ended questions are boxes). These instructions should indicate how many answers are to be ranked (for example: all; only the first and second; only the first and last; the most important and least important). These

Self-Administered Questionnaires 257 instructions should also spell out the order of rank- this discussion with an illustration of a real ques- ing (for example: \"Place a 1 beside the most inlPor- tionnaire, showing how some of these comments rant item, a 2 beside the next most important, and find substance in practice. so forth\"). Rank-ordering of responses is often difficult for respondents, however, because they Before turning to the illustration, however, I may have to read and reread the list several times, want to mention a critical aspect of questionnaire so this technique should be used only in those situ- design: precoding. Because the information col- ations where no other method will produce the lected by questionnaires is typically transformed desired result. into some t}'Pe of computer format, it's usually ap- propriate to include data-processing instructions on In mUltiple-part matrix questions, giving spe- the questionnaire itself. These instructions indicate cial instructions is useful unless the same format is where specific pieces of information will be stored used throughout the questionnaire. Sometimes re- in the machine-readable data files. Notice that the spondents will be expected to check one answer in following illustration has been precoded with the each column of the matrix; in other questionnaires mysterious numbers that appear near questions they'll be expected to check one answer in each and answer categories. row. Whenever the questionnaire contains both formats, it's useful to add an instruction clarifying A CompOSite Illustration which is expected in each case. Figure 9-6 is part of a questionnaire used by the Pretesting the Questionnaire University of Chicago's National Opinion Research Center in its General Social Survey. The question- No matter how carefully researchers design a data- naire deals with people's attitudes toward the gov- collection instrument such as a questionnaire, there ernment and is designed to be self-administered, is always the possibility-indeed the certainty-of though most of the GSS is conducted in face- error. They 'will always make some mistake: an am- to-face interviews. biguous question, one that people cannot answer, or some other violation of the rules just discussed. Self-Administered Questionnaires The surest protection against such errors is to pretest the questionnaire in full or in part. Give the So far we've discussed how to formulate questions questionnaire to the ten people in your bowling and how to design effective questionnaires. As im- league, for example. It's not usually essential that portant as these tasks are, the labor vvill be wasted the pretest subjects comprise a representative unless the questionnaire produces useful data- sample, although you should use people for whom which means that respondents actually complete the questionnaire is at least relevant. the questionnaire. We turn now to the major methods for getting responses to questionnaires. By and large, it's better to ask people to com- plete the questionnaire than to read through it I've referred several times in this chapter to in- looking for errors. All too often, a question seems terviews and self-administered questionnaires. to make sense on a first reading, but it proves to be Actually, there are three main methods of adminis- impossible to answer. tering survey questionnaires to a sample of respon- dents: self-administered questionnaires, in which Stanley Presser and Johnny Blair (1994) de- respondents are asked to complete the question- scribe several different pretesting strategies and re- naire themselves; surveys administered by inter- port on the effectiveness of each. They also provide viewers in face-to-face encounters; and surveys data on the cost of the various methods. There are many more tips and guidelines for questionnaire construction, but covering them all would take a book in itself. For now I'll complete

10. Here are some things the government might do for the economy. Circle one number for each action to show whether you are in favor of it or against it 1. Strongly in favor of 2. In favor of 3. Neither in favor of nor against 4. Against 5. Strongly against PLEASE CIRCLE A NUMBER a. Control of wages by legislation ....... ccccccccc.c...... c..cccc ......... cc 1 2 3 4 5 281 2 3 4 5 291 bc Control of prices by legislation ....... c.co..'\"c.cc........ cc............ cc 1 2 3 4 5 301 c. Cuts in government spending .. cccccc ..cc ......ccccccc ..c.......cccc'\"cc.1 311 d. Government financing of projects to 2 3 4 5 321 create new jobs .......\"'c.c .. c..cc..c\"' ...... c.ccccc.c........ ccccc ..c......c... c1 2 3 4 5 331 e. Less governrnent regulation of business ... cc.ccc ........·· ..·ccccc 1 341 f. Support for industry to develop new 2 3 4 5 products and technology ccc'cccc ... \"'c.ccccccccc........ ccccc... c...... c.c 1 351 g. Supporting declining industries to 2 3 4 5 protect jobs .c.cccc.ccc ..c.. c........\"'c.cccc...c....... c.cccc\"' ........cccc'\"cccc\" 1 hc Reducing the work week to create 2 3 4 5 more jobs c............. ccccccccccccc........ c..ccccccccccc ......cccccccc.... c.... \"'cc 1 11 c Listed below are various areas of governrnent spending. Please indicate whether you would like to see more or less governrnent spending in each area. Remember that if you say \"much more,\" it might require a tax increase to pay for it 1. Spend much more 2. Spend more 3. Spend the same as now 4. Spend less 5. Spend much less 8. Can't choose ac The environment ....c'\"cccccc.c.c ......c.. ccccccc.c.cc .. c.. c..cc....... 1 PLEASE CIRCLE A NUMBER 8 361 b. Health ......ccc.c.'\"ccc.cccc............. c..cc.ccc\"'.c.......... c.c ..c.......· 1 2345 8 371 c. The police and law enforcement ...... c... c.c.'\"........... '\"c 1 2345 8 381 dc Education c....\"'......c.. cc.c.cc.c ..c........... c.. ccccc.......... ccc.c.c.. 1 2345 8 391 ec The military and defense ........cc ...c..............c..cc\"'\"'.\"'... 1 2345 8 401 f. Retirement benefits \"'.....cc.. c.c......c..................c........ cc.1 2 345 8 411 gc Unemployment benefits c.c.............c.c.'\"c.........·c.c...c..c.. 1 2 345 8 421 t1. Culture and the arts '\"'\" ............'\"cccc .........c...c.cc..c....... 1 2345 8 431 2345 441 12. If the government had to choose between keeping down inflation or keeping down unemployment, 451 to which do you think it should give highest priority? Keeping down inflation ....c......ccccccccc ............. ccccccc.. c..........\"'ccc.'\".......c.....\"'cc.ccc.c\"'................·..··c\"'...\"' ..c.. 1 Keeping down unemployment '\"c......c........cc'\"ccc.c........c.. \"'c....cc............ c................c......... c.. cc .....c..cc..·..... 2 Can't choose ........cc... c.c.cc\"'\"c.c.........\"'ccc....c............ c.cc...........c.......\"'....c...c..c.................c.......c.....c........ 8 13c Do you think that labor unions in this country have too much power or to little power? Far too much power ..c..........\"'c.c.cc.cc....c.........c...c...............c........c...........c....ccc...c..·c.c...c...ccc ...........c.... 1 Too much power ....c...cccccccccccc ..... c..... c... ccc.c....c.......c.ccccc............. c.c...cc.....c..c............. cc\"'ccc.ccc........... c.c.. 2 About the right amount of power cccccc.c............'\"cc'cc .............. cccc ...cc.......c....... c....cc.......................\"'.....c3 Too little power .....c...............c.ccccccccc..cc...... c.c.ccc..c.......c.c.cccc..c.cc....c.......... cc ..ccc ...c...................cc....c....... 4 Far too little power cc..... ccc..c'\".c..c.....\"'......cccc.........\"'\"cccc ....c.c.............c.c.cc..c.c......c.... c....\"'\"'..................·· 5 Can't choose cccccccccc...cc.c..............c..c.c.cc.c..c......c.c.ccccc...............c.cccccc.c.c .......c........... c.c...........·......·.c....ccc 8 FIGURE 9-6 ASample Questionnaire. This questionnaire excerpt is from the General Social Survey, amajor source of data for analysis by social researchers around the world.

14c How about business and industry, do they have too much power or too little power? 461 l~~ ~~?U~~::rw~.r . ~c.::::::::::::c::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::: To~~ttlee ~~et amount of power .c\"Cc.c ..c· ... cc..c..ccccccc......ccccc.......c\"'......c..c...\"'.... c.c .............. c...c...... c... 3 ~~~:f~~~~:~~~~~c:::: .. ::.: .. ·:·::·: .. :.:::·::::::·:.::::·:::::.::::::c:c:::::·:::::::::::::::::::::::::::::::::.::::::::::::::::::::::::::::::i 15c And what about the federal governrnent, does it have to much power or too little ! 471 power? ~~~~~~~~~[\"\"totPo~\"iii 16. In general, how good would you say labor unions are for the country as a whole? 481 ~~~€~~,i! 17 What do you think the government's role in each of these industries should be? 1. Own it 2. Control prices and profits but not own it 3. Neither own it nor control its prices and profits 8. Can't choose PLEASE CIRCLE A NUMBER a. Electric power .......c.. c............c.'\"cc.....c..cc\"'.........cccc........ 'ccc ... 1 2 3 8 491 bc The steel industry .c'cccccc.............'\"........ccc'\"cc....c.. cc.c ......Cccc 1 2 3 8 501 c. Banking and insurance .c.c.cccc.......ccccccc...... Ccccc\"'......cccc'\".....cc 1 2 3 8 511 18. On the whole, do you think it should or should not be the government's responsibility to c •• 1. Definitely should be 2. Probably should be 3. Probably should not be 4. Definitely should not be 8. Can't choose PLEASE CIRCLE A NUMBER 521 531 ac Provide a job for everyone who wants one .. cC\"ccc ....c..ccc..\"'....c... 1 2 3 4 8 541 bc Keep prices under control .......ccc\"'....c..c..c.c ...c.....c... c.......c......... 1 2 3 4 8 c. Provide health care for the sick ... c.......c.cccc .........c.............. c..... 1 2 3 4 8 551 d\" Provide a decent standard of living for the old .....cccccc.c.c.CCCCccc.c..... c.cccccc........CC\"Cccc ...... c.ccccc......c..C\" .. c....... 1 2 3 4 8 FIGURE 9-6 (Continued)

260 Chapter 9: Survey Research conducted by telephone. This section and the next keep them in mind any time you plan to send two discuss each of these methods in tufIL questionnaires to others. The most common form of self-administered A common reason for not returning question- questionnaire is the mail survey. However, there naires is that it's too much trouble. To overcome are several other techniques that are often used as this problem, researchers have developed several welL At times, it may be appropriate to administer a ways to make returning them easier. For instance, questionnaire to a group of respondents gathered at a self-mailing questionnaire requires no return en- the same place at the same time. A survey of stu- velope: When the questionnaire is folded a particu- dents taking introductory psychology might be con- lar way, the return address appears on the outside. ducted in this manner during class. High school stu- The respondent therefore doesn't have to worry dents might be surveyed during homeroom period. about losing the envelope. Some recent eXllerimentation has been con- More-elaborate designs are available also. The ducted with regard to the home delivery of ques- tionnaires. A research worker delivers the ques- university student questionnaire to be described tionnaire to the home of sample respondents and later in this chapter was bound in a booklet with a eX1l1ains the study. Then the questionnaire is left special. two-panel back cover. Once the question- for the respondent to complete, and the researcher naire was completed, the respondent needed only picks it up later. to fold out the extra paneL wrap it around the booklet. and seal the whole thing with the adhe- Home delivery and the mail can also be used in sive strip running along the edge of the paneL The combination. Questionnaires are mailed to families, foldout panel contained my return address and and then research workers visit homes to pick up postage. When I repeated the study a couple of the questionnaires and check them for complete- years later. I improved on the design. Both the ness. Just the opposite technique is to have ques- front and back covers had foldout panels: one for tionnaires hand delivered by research workers with sending the questionnaire out and the other for a request that the respondents mail the completed getting it back-thus avoiding the use of envelopes questionnaires to the research office. altogether. On the whole, when a research worker either The point here is that anything you can do to delivers the questionnaire, picks it up, or both, the completion rate seems higher than it is for straight- make the job of completing and returning the fonvard mail surveys. Additional experimentation questionnaire easier will improve your study. Imag- with this technique is likely to point to other ways ine receiving a questionnaire that made no provi- to improve completion rates while reducing costs. sions for its return to the researcher. Suppose you The remainder of this section, however. is devoted had to (1) find an envelope, (2) write the address specifically to the mail survey, which is still the typ- on it, (3) figure out how much postage it required, ical form of self-administered questionnaire. and (4) put the stamps on it. How likely is it that you would return the questionnaire? Mail Distribution and Return A few brief comments on postal options are in The basic method for collecting data through the order. You have options for mailing questionnaires mail has been to send a questionnaire accompanied out and for getting them returned. On outgoing by a letter of explanation and a self-addressed, mail. your choices are essentially between first- stamped envelope for returning the questionnaire. class postage and bulk rate. First class is more cer- The respondent is expected to complete the ques- tain, but bulk rate is far cheaper. (Check your local tionnaire, put it in the envelope, and return it. It post office for rates and procedures.) On return by any chance, you've received such a question- mail, your choice is between postage stamps and naire and failed to return it. it would be valuable to business-reply permits. Here, the cost differential is recall the reasons you had for not returning it and more complicated. If you use stamps, you pay for them whether people return their questionnaires or not. With the business-reply permit, you pay for

Self-Administered Questionnaires 261 only those that are used, but you pay an additional already been assigned. Two examples should illus- surcharge of about a nickel. This means that stamps trate the important advantages of this procedure. are cheaper if a lot of questionnaires are returned, but business-reply permits are cheaper if fewer are Let's assume you're studying attitudes toward a returned (and you won't know in advance how political figure. In the middle of the data collection, many vvill be returned). the media break the story that the politician is hav- ing extramarital affairs. By knowing the date of There are many other considerations involved that public disclosure and the dates when question- in choosing among the several postal options. Some naires were received, you'll be in a position to de- researchers, for example, feel that using postage termine the effects of the disclosure. (Recall the dis- stamps communicates more \"humanness\" and sin- cussion in Chapter 8 of history in connection with cerity than using bulk rate and business-reply per- experiments. ) mits does. Others worry that respondents vvill steam off the stamps and use them for some pur- In a less sensational way, serialized ID numbers pose other than returning the questionnaires. Be- can be valuable in estimating nonresponse biases in cause both bulk rate and business-reply permits re- the survey. Barring more direct tests of bias, you quire establishing accounts at the post office, you'll may wish to assume that those who failed to an- probably find stamps much easier for small surveys. swer the questionnaire \\vill be more like respon- dents who delayed answering than like those who Monitoring Returns answered right away. An analysis of questionnaires received at different points in the data collection The mailing of questionnaires sets up a new re- might then be used for estimates of sampling bias. search question that may prove valuable to a study. For example, if the grade point averages (GPAs) re- Researchers shouldn't sit back idly as question- ported by student respondents decrease steadily naires are returned; instead, they should undertake through the data collection, with those replying a careful recording of the varying rates of return right away having higher GPAs and those replying among respondents. later having lower GPAs, you might tentatively con- clude that those who failed to answer at all have An invaluable tool in this activity is a return lower GPAs yet. Although it would not be advisable rate graph. The day on which questionnaires were to make statistical estimates of bias in this fashion, mailed is labeled Day 1 on the graph, and every you could take advantage of approximate estimates day thereafter the number of returned question- based on the patterns you've observed. naires is logged on the graph. It's usually best to compile two graphs. One shows the number re- If respondents have been identified for pur- turned each day-rising, then dropping. The sec- poses of follow-up mailing, then preparations for ond reports the cumulative number or percentage. those mailings should be made as the question- In part, this activity provides the researchers with naires are returned. The case study later in this sec- gratification, as they get to draw a picture of their tion discusses this process in greater detail. successful data collection. More important, how- ever, it serves as their guide to how the data collec- Follow-up Mailings tion is going. If follow-up mailings are planned, the graph provides a clue about when such mailings Follow-up mailings may be administered in several should be launched. (The dates of subsequent mail- ways. In the sin1plest, nonrespondents are simply ings should be noted on the graph.) sent a letter of additional encouragement to partici- pate. A better method, however, is to send a new As completed questionnaires are returned, each copy of the survey questionnaire with the follow- should be opened, scanned, and assigned an up letter. If potential respondents have not re- identification (ID) number. These numbers should turned their questionnaires after two or three be assigned serially as the questionnaires are re- weeks, the questionnaires have probably been lost turned, even if other identification numbers have or misplaced. Receiving a follow-up letter might

262 Chapter 9: Survey Research encourage them to look for the original question- survey. The body of inferential statistics used in naire, but if they can't find it easily, the letter may connection with survey analysis assumes that all go for naughL members of the initial sample complete and return their questionnaires. Because this almost never The methodological literature strongly suggests happens, response bias becomes a concern, with that follow-up mailings provide an effective the researcher testing (and hoping) for the possibil- method for increasing return rates in mail surveys. ity that the respondents look essentially like a In generaL the longer a potential respondent delays random sample of the initial sample, and thus a replying, the less likely he or she is to do so at aIL somewhat smaller random sample of the total Properly timed follow-up mailings, then, provide population. additional stin1Uli to respond. Nevertheless, overall response rate is one guide The effects of follow-up mailings will be seen in to the representativeness of the sample respondents. the response rate curves recorded during data col- If a high response rate is achieved, there is less lection. The initial mailings will be followed by a chance of significant response bias than \"vith a low rise and subsequent subsiding of returns; the fol- rate. Conversely, a low response rate is a danger sig- low-up mailings will spur a resurgence of returns; naL because the nonrespondents are likely to differ and more follow-ups will do the same. In practice, from the respondents in ways other than just their three mailings (an original and two follow-ups) willingness to participate in your survey. Richard seem the most efficient. Bolstein (1991), for example, found that those who did not respond to a preelection political poll were The timing of follow-up mailings is also impor- less likely to vote that those who did participate. tant. Here the methodological literature offers less- Estimating the turnout rate from just the survey re- precise guides, but I have found that two or three spondents, then, would have overestimated the weeks is a reasonable space between mailings. number who would show up at the polls. (This period might be increased by a few days if the mailing time-out and in-is more than two or But what is a high or low response rate? A three days.) quick review of the survey literature will uncover a wide range of response rates, Each of these may be If the individuals in the survey sample are not accompanied by a statement like \"This is regarded identified on the questionnaires, it may not be pos- as a relatively high response rate for a survey of this sible to remail only to nonrespondents. In such a tnJe.\" (A U.S. senator made this statement regard- case, send your follow-up mailing to all members of ing a poll of constituents that achieved a 4 percent the sample, thanking those who may have already return rate.) Even so, it's possible to state some participated and encouraging those who have not general guidelines about return rates. A review of to do so. (The case study reported later describes published social research literature suggests that a another method you can use in an anonymous response rate of 50 percent is considered adequate mail survey.) for analysis and reporting. A response of 60 percent is good; a response rate of 70 percent is very Acceptable Response Rates good. Bear in mind, however, that these are only rough guides; they have no statistical basis, and a A question that new survey researchers frequently demonstrated lack of response bias is far more im- ask concerns the percentage return rate, or the portant than a high response rate,. If you want to response rate, that should be achieved in a mail pursue this matter further, Delbert Miller (1991: 145-55) has revievved several specific surveys to response rate The number of people participating offer a better sense of the variability of response in a survey divided by the number selected in the rates. sample, in the form of a percentage. This is also called the comp/erioll mie or, in self-administered sur- As you can imagine, one of the more persistent veys, the retum mle: the percentage of questionnaires discussions among survey researchers concerns sent out that are returned.

Self-Administered Questionnaires 263 ways of increasing response rates. You'll recall that serves present purposes all the better. The study this was a chief concern in the earlier discussion of was conducted by the students in my graduate options for mailing out and receiving question- seminar in survey research methods. naires. Survey researchers have developed many ingenious techniques addressing this problem. As you may recall, 1, 100 students were selected Some have experimented with novel formats. Oth- from the university registration tape through a ers have tried paying respondents to participate. stratified, systematic sampling procedure. For each The problem with paying, of course, is that it's ex- student selected, six self-adhesive mailing labels pensive to make meaningfully high payment to were printed by the computer. hundreds or thousands of respondents, but some imaginative alternatives have been used. Some re- By the time we were ready to distribute the searchers have said, \"We want to get your two- questionnaires, it became apparent that our meager cents' worth on some issues, and we're willing to research funds wouldn't cover several mailings to pay\"-enclosing two pennies. Another enclosed a the entire sample of 1, 100 students (questionnaire quarter, suggesting that the respondent make printing costs were higher than anticipated). As a some little child happy. Still others have enclosed result, we chose a systematic two-thirds sample paper money. Similarly, Michael Davern and of the mailing labels, yielding a subsample of his colleagues (2003) found that financial 733 students. incentives also increased completion rates in face- to-face interview surveys (discussed in the next Earlier, we had decided to keep the survey section) anonymous in the hope of encouraging more- candid responses to some sensitive questions. (Later Don Dillman (1978) provides an excellent re- surveys of the same issues among the same pop- view of the various techniques that survey re- ulation indicated this anonyrrtity was unnecessary.) searchers have used to increase return rates on Thus, the questionnaires would carry no identifica- mail surveys, and he evaluates the impact of each. tion of students on them. At the same time, we More important, Dillman stresses the necessity of hoped to reduce the follow-up mailing costs by paying attention to all aspects of the study-what mailing only to nonrespondents. he calls the \"Total Design Method\"-rather than one or two special gimmicks. To achieve both of these aims, a special post- card method was devised. Each student was mailed More recently, Francis Yammarino, Steven a questionnaire that carried no identifying marks, Skinner, and Terry Childers (1991) have under- plus a postcard addressed to the research office- taken an in-depth analysis of the response rates with one of the student's mailing labels affixed to achieved in many studies using different tech- the reverse side of the card, The introductory letter niques. Their findings are too complex to summa- asked the student to complete and return the ques- rize easily, but you might find some guidance there tionnaire-assuring anonymity-and to return for effective survey design. the postcard simultaneously. Receiving the postcard would tell us-without indicating which question- ACase Study naire it was-that the student had returned his or her questionnaire. This procedure would then facil- The steps involved in the administration of a mail itate follow-up mailings. survey are many and can best be appreciated in a walk-through of an actual study. Accordingly, this The 32-page questionnaire was printed in section concludes with a detailed description of booklet form. The three-panel cover described ear- how the student survey we discussed in Chapter 7 lier in this chapter permitted the questionnaire to as an illustration of systematic sampling was ad- be returned vvithout an additional envelope. ministered . This study did not represent the theo- retical ideal for such studies, but in that regard it A letter introducing the study and its purposes was printed on the front cover of the booklet. It ex- plained why the study was being conducted (to learn how students feel about a variety of issues), how students had been selected for the study, the

264 Chapter 9: Survey Research importance of each student's responding, and the The follow-up mailing stimulated a resurgence mechanics of returning the questionnaire. of returns, as expected, and the same logging pro- cedures continued. The returned postcards told us Students were assured that their responses to which additional mailing labels to destroy. Unfortu- the survey were anonymous, and the postcard nately, time and financial pressures made a third method was eXlJlainedo A statement followed about mailing impossible, despite initial plans to do so, the auspices under which the study was being con- but the two mailings resulted in an overall return ducted, and a telephone number was provided for rate of 62 percent. those who might want more information about the study. (Five students called for information.) This illustration should give you a fairly good sense of what's involved in the execution of mailed By printing the introductory letter on the self-administered questionnaireso Let's turn now to questionnaire, we avoided the necessity of enclos- the second principal method of conducting surveys, ing a separate letter in the outgoing envelope, in-person interviews. thereby simplifying the task of assembling mailing pieces. Interview Surveys The materials for the initial mailing were The interview is an alternative method of collect- assembled as follows. (1) One mailing label for ing survey datao Rather than asking respondents to each student was stuck on a postcard. (2) Another read questionnaires and enter their own answers, label was stuck on an outgoing manila envelope. researchers send interviewers to ask the questions (3) One postcard and one questionnaire were orally and record respondents' answers. Interview- placed in each envelope-with a glance to ensure ing is typically done in a face-to-face encounter, that the name on the postcard and on the envelope but telephone interviewing, discussed in the next were the same in each case. section, follows most of the same guidelines. The distribution of the survey questionnaires Most interview surveys require more than one had been set up for a bulk rate mailing. Once the interviewer, although you might undertake a questionnaires had been stuffed into envelopes, small-scale interview survey yourselL Portions of they were grouped by zip code, tied in bundles, and this section will discuss methods for training and delivered to the post office. supervising a staff of interviewers assisting you with a survey. Shortly after the initial mailing, questionnaires and postcards began arriving at the research office. This section deals specifically with survey inter- Questionnaires were opened, scanned, and as- viewing. Chapter 10 discusses the less structured, signed identification numbers as described earlier in in-depth interviews often conducted in qualitative this chapter. For every postcard received, a searcl1 field research. was made for that student's remaining labels, and they were destroyedo The Role of the Survey Interviewer After two or three weeks, the remaining mail- There are several advantages to having a question- ing labels were used to organize a follow-up mail- naire administered by an interviewer rather than a ing. This tinle a special, separate letter of appeal respondenL To begin with, interview surveys typi- was included in the mailing piece. The new letter cally attain higher response rates than mail surveys indicated that many students had returned their do. A properly designed and executed interview questionnaires already, and it was very important survey ought to achieve a completion rate of at for all others to do so as well. least 80 to 85 percenL (Federally funded surveys often require one of these response rates.) interview A data-collection encounter in which one person (an interviewer) asks questions of an- other (a respondent). Interviews may be conducted face-to-face or by telephone.

Interview Surveys 265 Respondents seem more reluctant to turn down The interviewer must also fit into this ideal sit- an interviewer standing on their doorstep than uation. The interviewer'S presence should not affect to throwaway a mailed questionnaire. a respondent's perception of a question or the an- swer given. In other words, the interviewer should The presence of an interviewer also generally be a neutral medium through which questions and decreases the number of \"don't knows\" and \"no answers are transmitted. answers.\" If minimizing such responses is impor- tant to the study, the interviewer can be instructed As such, different interviewers should obtain to probe for answers (\"If you had to pick one of the exactly the same responses from a given respon- answers, which do you think would come closest to dent. (Recall our earlier discussions of reliability.) your feelings?\"). This neutrality has a special importance in area samples. To save time and money, a given inter- Further, if a respondent clearly misunderstands viewer is typically assigned to complete all the in- the intent of a question or indicates that he or she terviews in a particular geographic area-a city does not understand, the interviewer can clarify block or a group of nearby blocks. If the inter- matters, thereby obtaining relevant responses. (As viewer does anything to affect the responses ob- we'll discuss shortly, such clarifications must be tained, the bias thus interjected might be inter- strictly controlled through formal specifications.) preted as a characteristic of that area. Finally, the interviewer can observe respon- Let's suppose that a survey is being done to dents as well as ask questions. For example, the in- determine attitudes toward low-cost housing in tenriewer can note the respondent's race if this is order to help in the selection of a site for a new considered too delicate a question to ask. Similar government-sponsored development. An inter- observations can be made regarding the quality of ,riewer assigned to a given neighborhood might- the dwelling, the presence of various possessions, through word or gesture-communicate his or her the respondent's ability to speak English, the re- own distaste for low-cost housing developments. spondent's general reactions to the study, and so Respondents might therefore tend to give responses forth. In one survey of students, respondents were in general agreement 'Yith the interviewer's ovvn given a short, self-administered questionnaire to positiono The results of the survey would indicate complete-concerning sexual attitudes and behav- that the neighborhood in question strongly resists ior-during the course of the interview. While re- construction of the development in its area when spondents completed the questionnaire, the inter- in fact their apparent resistance simply reflects the viewer made detailed notes regarding their dress interviewer's attitudes. and grooming. General Guidelines This procedure raises an ethical issue. Some re- for Survey Interviewing searchers have objected that such practices violate the spirit of the agreement by which the respon- The manner in which interviews ought to be con- dent has allowed the interview. Although ethical is- ducted will vary somewhat by survey population sues seldom are clear-cut in social research, it's im- and survey contento Nevertheless, some general portant to be sensitive to them, as we saw in guidelines apply to most intenriewing situations. Chapter 3. Appearance and Demeanor Survey research is of necessity based on an un- realistic stimulus-response theory of cognition and As a rule, interviewers should dress in a fashion behavior. Researchers must assume that a question- similar to that of the people they'll be interviewing. naire item will mean the same thing to every re- A richly dressed interviewer will probably have spondent, and every given response must mean the difficulty getting good cooperation and responses same when given by different respondents. Al- from poorer respondents; a poorly dressed though this is an impossible goal. survey questions are drafted to approximate the ideal as closely as possible.

266 Chapter 9: Survey Research interviewer will have similar difficulties with richer Ultimately, the interviewer must be able to read respondents. To the extent that the interviewer's the questionnaire items to respondents without er- dress and grooming differ from those of the respon- ror, \\vithout stumbling over words and phrases. A dents, it should be in the direction of cleanliness good model is the actor reading lines in a play or and neatness in modest apparel. If cleanliness is not movie. The lines must be read as though they con- next to godliness, it appears at least to be next to stituted a natural conversation, but that conversa- neutrality. Although middle-class neatness and tion must follow exactly the language set down in cleanliness may not be accepted by all sectors of the questionnaire. u.s. society, they remain the primary norm and are By the same token, the interviewer must be familiar vvith the specifications prepared in con- the most likely to be acceptable to the largest num- junction vvith the questionnaire. Inevitably some ber of respondents. questions will not exactly fit a given respondent's situation, and the interviewer must determine how Dress and grooming are typically regarded as the question should be interpreted in that situation., signs of a person's attitudes and orientations. Torn The specifications provided to the interviewer jeans, green hair, and razor blade earrings may should give adequate guidance in such cases, but communicate-correctly or incorrectly-that the the interviewer must know the organization and interviewer is politically radicaL sexually permis- contents of the specifications well enough to refer sive, favorable to drug use, and so forth. Any of to them efficiently. It would be better for the inter- these impressions could bias responses or affect the viewer to leave a given question unanswered than willingness of people to be interviewed. to spend five minutes searching through the speci- fications for clarification or trying to interpret the In demeanor, interviewers should be pleasant if relevant instructions. nothing else. Because they'll be prying into a re- spondent's personal life and attitudes, they must Following Question Wording Exactly communicate a genuine interest in getting to know the respondent. \\\"Iithout appearing to spy. They The first part of this chapter discussed the must be relaxed and friendly, \\\"Iithout being too ca- significance of question wording for the responses sual or clinging. Good interviewers also have the obtained. A slight change in the wording of a given ability to determine very quickly the kind of person question may lead a respondent to answer \"yes\" the respondent will feel most comfortable with, the rather than \"no.\" It follows that interviewers must kind of person the respondent would most enjoy be instructed to follow the wording of questions ex- talking to. Clearly, the interview will be more suc- actly. Othen\"lise all the effort that the developers cessful if the interviewer can become the kind of have put into carefully phrasing the questionnaire person the respondent is comfortable \\\"Iith. Fur- items to obtain the information they need and to ther, because respondents are asked to volunteer a ensure that respondents interpret items precisely as portion of their time and to divulge personal infor- intended will be wasted. mation, they deserve the most enjoyable experi- ence the researcher and interviewer can provide. Familiarity with the Questionnaire Recording Responses Exactly If an interviewer is unfamiliar with the question- Whenever the questionnaire contains open-ended naire, the study suffers and the respondent faces an questions, those soliciting the respondent's own an- unfair burden. The interview is likely to take more swers, the interviewer must record those answers time than necessary and be unpleasant. Moreover, exactly as given. No attempt should be made to the interviewer cannot acquire familiarity by skim- summarize, paraphrase, or correct bad grammar. ming through the questionnaire two or three times. He or she must study it carefully, question by ques- This exactness is especially important because tion, and must practice reading it aloud. the interviewer will not know how the responses are to be coded. Indeed, the researchers themselves

Interview Surveys 267 may not know the coding until they've read a hun- questions. For example, in response to a question dred or so responses. For example, the question- about traffic conditions, the respondent might sim- naire might ask respondents how they feel about ply reply, \"Pretty bad.\" The interviewer could ob- the traffic situation in their community. One re- tain an elaboration on this response through a vari- spondent might answer that there are too many ety of probes. Sometimes the best probe is silence; cars on the roads and that something should be if the interviewer sits quietly with pencil poised, done to limit their numbers. Another might say the respondent will probably fill the pause with ad- that more roads are needed. If the interviewer ditional comments. (This technique is used effec- recorded these two responses with the same SUfl1- tively by newspaper reporters.) Appropriate verbal mary-\"congested traffiC-the researchers would probes might be \"How is that?\" or \"In what not be able to take advantage of the important dif- ways?\" Perhaps the most generally useful probe ferences in the original responses. is \"Anything else?\" Sometimes, verbal responses are too inarticu- Often, interviewers need to probe for answers late or ambiguous to permit interpretation. How- that will be sufficiently informative for analytical ever. the interviewer may be able to understand purposes. In every case, however. such probes //lust the intent of the response through the respondent's be completely neutral; they must not in any way gestures or tone. In such a situation, the inter- affect the nature of the subsequent response. viewer should still record the exact verbal response Whenever you anticipate that a given question may but also add marginal comments giving both the require probing for appropriate responses, you interpretation and the reasons for arriving at iL should provide one or more useful probes next to the question in the questionnaire. This practice has More generally, researchers can use any mar- two important advantages. First. you'll have more ginal comments explaining aspects of the response time to devise the best. most neutral probes. Sec- not conveyed in the verbal recording, such as the ond, all interviewers vvill use the same probes respondent's apparent anger. embarrassment un- whenever they're needed. Thus, even if the probe certainty in answering, and so forth. In each case, isn't perfectly neu traL all respondents will be pre- however, the exact verbal response should also be sented with the same stimulus. This is the same recorded. logical guideline discussed for question wording. Although a question should not be loaded or bi- Probing for Responses ased, it's essential that every respondent be pre- sented with the same question, even if it's biased. Sometimes respondents in an interview will give an inappropriate or incomplete answer. In such cases, Coordination and Control a probe, or request for an elaboration, can be use- fuL For example, a closed-ended question may Most interview surveys require the assistance of present an attitudinal statement and ask the re- several interviewers. In large-scale surveys, inter- spondent to strongly agree, agree somewhat dis- viewers are hired and paid for their work. Student agree somewhat. or strongly disagree. The respon- researchers might find themselves recruiting dent, however, may reply: \"I think that's true.\" The friends to help them interview. Whenever more interviewer should follow this reply with \"Would you say you strongly agree or agree somewhat?\" If probe A technique employed in interviewing to necessary, interviewers can explain that they must solicit a more complete answer to a question. It is a check one or the other of the categories provided. If nondirective phrase or question used to encourage the respondent adamantly refuses to choose, the a respondent to elaborate on an answer. Examples interviewer should write in the exact response include \"Anything more?\" and \"How is that?\" given by the respondent. Probes are more frequently required in eliciting responses to open-ended than closed-ended

268 Chapter 9: Survey Research than one interviewer is involved in a survey, their Make sure your interviewers fully understand the efforts must be carefully controlled, This control specifications and the reasons for them as well as has two aspects: training interviewers and supervis- the questions themselves. ing them after they begin work. This portion of the interviewer training is likely The interviewers' training session should begin to generate many troublesome questions from your with the description of what the study is all abouL interviewers. They'll ask, \"What should I do Even though the interviewers may be involved if ... 7\" In such cases, avoid giving a quick, offhand only in the data-collection phase of the project, it answer. If you have specifications, show how the will be useful to them to understand what will be solution to the problem could be determined from done with the interviews they conduct and what the specifications. If you do not have specifications, purpose 'will be served. Morale and motivation are show how the preferred handling of the situation usually lower when interviewers don't know fits within the general logic of the question and what's going on. the purpose of the study. Giving unexplained an- swers to such questions will only confuse the The training on how to interview should begin interviewers and cause them to take their work '.vith a discussion of general guidelines and proce- less seriously. If you don't know the answer to such dures, such as those discussed earlier in this sec- a question when it's asked, admit it and ask for tion. Then the whole group should go through the some time to decide on the best answer. Then questionnaire together-question by question. think out the situation carefully and be sure to give Don't sinlply ask if anyone has any questions about all the interviewers your answer, explaining your the first page of the questionnaire. Read the first reasons. question aloud, explain the purpose of the ques- tion, and then entertain any questions or com- Once you've gone through the whole question- ments the interviewers may have. Once all their naire, conduct one or two demonstration inter- questions and comments have been handled, go on views in front of everyone. Preferably, you should to the next question in the questionnaire. interview someone other than one of the inter- viewers. Realize that your interview 'will be a model It's always a good idea to prepare specifications for those you're training, so make it good. It would to accompany an interview questionnaire. Specifi- be best, moreover, if the demonstration interview catiol1s are explanatory and clarifying comments were done as realistically as possible. Don't pause about handling difficult or confusing situations that during the demonstration to point out how you've may occur with regard to particular questions in the handled a complicated situation: Handle it, and questionnaire. When drafting the questionnaire, try then explain later. It's irrelevant if the person you're to think of all the problem cases that might arise- interviewing gives real answers or takes on some the bizarre circumstances that might make a ques- hypothetical identity for the purpose, as long as the tion difficult to answer. The survey specifications answers are consistent. should provide detailed guidelines on how to handle such situations. For example, even as simple a mat- After the demonstration interviews, pair off ter as age might present problems. Suppose a re- your interviewers and have them practice on each spondent says he or she will be 25 next week. The other. When they've completed the questionnaire, interviewer might not be sure whether to take the have them reverse roles and do it again. Interview- respondent's current age or the nearest one. The ing is the best training for interviewing. As your in- specifications for that question should explain what terviewers practice on each other, wander around, should be done. (Probably, you would specify that listening in on the practice so you'll know how well the age as of last birthday should be recorded in they're doing. Once the practice is completed, the all cases.) whole group should discuss their experiences and ask any other questions they may have. If you've prepared a set of specifications, review them with the interviewers when you go The final stage of the training for interviewers over the individual questions in the questionnaire. should involve some \"real\" interviews. Have

Telephone Surveys 269 them conduct some interviews under the actual subscribers, who were hardly typical in a nation conditions that ,viII pertain to the final survey. You just recovering from the Great Depression. By may want to assign them people to interview, or 1993, however, the Census Bureau (1996: Table perhaps they may be allowed to pick people them- 1224) estimated that 93.4 percent of all housing selves. Don't have them practice on people you've units had telephones, so the earlier form of class selected in your sample, however. After each inter- bias has been substantially reduced. viewer has completed three to five interviews, have him or her check back with you. Look over the A related sampling problem involved unlisted completed questionnaires for any evidence of mis- numbers. A survey sample selected from the pages understanding. Again, answer any questions that of a local telephone directory would totally omit all the interviewers have. Once you're convinced that those people-typically richer-who requested a given interviewer knows what to do, assign some that their numbers not be published. This potential actual interviews, using the sample you've selected bias has been erased through a technique that for the study. has advanced telephone sampling substantially: random-digit dialing. It's essential to continue supervising the work of interviewers over the course of the study. You Telephone surveys offer many advantages that should check in with them after they conduct no underlie the growing popularity of this method. more than 20 or 30 interviews, You might assign Probably the greatest advantages are money and 20 interviews, have the interviewer bring back time, in that order. In a face-to-face, household in- those questionnaires when they're completed, look terview, you may drive several miles to a respon- them over, and assign another 20 or so. Although dent's horne, find no one there, return to the re- this may seem overly cautious, you must continu- search office, and drive back the next day-possibly ally protect yourself against misunderstandings that finding no one there again. It's cheaper and quicker may not be evident early in the study. to let your fingers make the trips. If you're the only interviewer in your study, Interviewing by telephone, you can dress any these comments may not seem relevant However, way you please without affecting the answers re- it would be wise, for example, to prepare specifi- spondents give. And sometimes respondents will be cations for potentially troublesome questions in more honest in giving socially disapproved answers your questionnaire. Otherwise, you run the risk of if they don't have to look you in the eye. Similarly, it making ad hoc decisions, during the course of the may be possible to probe into more sensitive areas, study, that you'll later regret or forget Also, the though this isn't necessarily the case. People are, to emphasis on practice applies equally to the one- some extent, more suspicious when they can't see person project and to the complex funded survey the person asking them questions-perhaps a con- with a large interviewing staff. sequence of \"surveys\" aimed at selling magazine subscriptions and time-share condominiums. Telephone Surveys Interviewers can communicate a lot about For years telephone surveys had a rather bad repu- themselves over the phone, however, even though tation among professional researchers. Telephone they can't be seen. For example, researchers worry surveys are limited by definition to people who about the impact of an interviewer's name (particu- have telephones. Years ago, this method produced larly if ethnicity is relevant to the study) and debate a substantial social-class bias by excluding poor the ethics of having all interviewers use bland people from the surveys, This was vividly demon- \"stage names\" such as Smith or Jones. (Female strated by the LiterCllY Digest fiasco of 1936. Recall interviewers sometimes ask permission to do this, that, even though voters were contacted by maiL to avoid subsequent harassment from men they the sample was partially selected from telephone interview. ) Telephone surveys can allow greater control over data collection if several interviewers are engaged in the projecL If all the interviewers are

270 Chapter 9: Survey Research calling from the research office, they can get The pigs are eating my Volvo!\" (That sort of thing is clarification from the person in charge whenever much harder to fake when the interviewer is sitting problems occur, as they inevitably do. Alone in in your living room.) the boondocks, an interviewer may have to wing it between weekly visits with the interviewing Another potential problem for telephone inter- supervisoL viewing is the prevalence of answering machines. A study conducted by Walker Research (1988) Finally, another important factor involved in found that half of the owners of answering ma- the growing use of telephone surveys has to do chines acknowledged using their machines to with personal safety. Don Dillman (1978: 4) de- \"screen\" calls at least some of the time. Research by scribes the situation this way: Peter Tuckel and Barry Feinberg (1991), however, showed that answering machines had not yet had a Interviewers must be able to operate comfort- significant effect on the ability of telephone re- ably in a climate in which strangers are viewed searchers to contact prospective respondents. Nev- with distrust and must successfully counter re- ertheless, the researchers concluded that as an- spondents' objections to being interviewed. In- swering machines continued to proliferate, \"the creasingly, interviewers must be \\villing to work sociodemographic characteristics of owners will at night to contact residents in many house- change.\" This fact made it likely that \"different be- holds. In some cases, this necessitates providing havior patterns associated with the utilization of protection for interviewers working in areas of the answering machine\" could emerge (1991: 216). a city in which a definite threat to the safety of individuals exists. More-recent research has shown that several factors, including answering machines, have re- Concerns for safety, thus, work in two ways to duced response rates in telephone surveys. Peter hamper face-to-face interviews. Potential respon- Tuckel and Harry O'Neill (2002) and others have dents may refuse to be interviewed, fearing the examined the impact of such factors as Caller ID, an- stranger-interviewer. And the interviewers them- swering machines, telemarketing, and phone lines selves may incur some risks. All this is made even being tied up by faxes and Internet access. All these worse by the possibility of the researchers being constitute difficulties modern survey researchers sued for huge sums if anything goes wrong. must deal with. There are problems involved in telephone in- Computer-Assisted Telephone terviewing, however. As I've already mentioned, Interviewing (CA TI) the method is hampered by the proliferation of bo- gus \"surveys\" that are actually sales campaigns dis- In Chapter 14, we'll see some of the ways com- guised as research. If you have any questions about puters have influenced the conduct of social any such call you receive, by the way, ask the inter- research- particularly data processing and analysis. viewer directly whether you've been selected for a Computers are also changing the nature of tele- survey only or if a sales \"opportunity\" is involved. phone interviewing. One innovation is computer- It's also a good idea, if you have any doubts, to get assisted telephone interviewing (CATI). This the interviewer's name, phone number, and com- method is increasingly used by academic, govern- pany. Hang up if the caller refuses to provide any ment, and commercial survey researchers. Though of these. there are variations in practice, here's what CATI can look like. For the researcher, the ease \\vith which people can hang up is another shortcoming of telephone Imagine an interviewer wearing a telephone surveys. Once you've been let inside someone's headset, sitting in front of a computer terminal and home for an interview, the respondent is unlikely its video screen. The central computer selects a to order you out of the house in midinterview. It's telephone number at random and dials it. (Recall much easier to terminate a telephone interview that random-digit dialing avoids the problem of abruptly, saying something like, \"Whoops! Someone's at the door. I gotta go.\" or \"Omigod!

Telephone Surveys 271 byJames E. Dannemiller reasons for switching TV neVis shows Thus, vie can hear, in turn, the re- sponses ofthe young men, young women, middle-aged men, and so SMS Research, Honolulu forth In one such study we found the younger and older men tending to watch one TV news show, while the middle-aged men watched some- development of various CATI techniques has been aboon to sur- thing else. Listening to the responses of the middle-aged men, one after and marketing research, though mostly it has supported the another, we heard acommon comment \"Well, now that I'm older .. \" This collection, coding, and analysis of\"data as usual \"The Voice Capture tech- kind of aside might have been lost in the notes hastily typed by inter- nique deVeloped by Survey Systems, hOV/ever,offers quite unusual possi- viewers, but such comments stood out dramatically in the oral data The bilities, which we are only beginning to explore middle-aged men seemed to be telling us they felt\"maturity\" required In the course of aCATI-based telephone interview, the interviewer them to watch aparticular show, vlhile more years under their belts let can nigger the computer to begin digitally recording the conversation them drift back to what they liked in the first place with the respondent Having determined that the respondent has re- cently changed his or her favorite TV neVIS show, for example, the inter- These kinds of data are especially compelling to clients, particularly viewer can ask,\"Why did you change!\" and begin recording the verbatim in customer satisfaction studies Rather than summarize what we feel a response. (Early in the interview, the interviewer has asked permission to client's customers like and don't like, we can let the respondents speak record partS of the intervievl) directly to the client in their own vlords.lt's like afocus group on de- Later on, coders can play back the responses and code them- mand Going one step further, we have found that letting line employees much as would do with the interviewer's typescript of the re- (bank tellers, for example) listen to the responses has more impact than sponses This offers an easier and more accurate '/lay of accomplishing a having their supervisors tell them what they are doing right or wrong conventional task. But that's atame use of the new capability It's also possible to incorporate such oral data as parts of across- As exciting as these experiences are, Ihave the strong feeling that tabulation during analysisVle may create atable of gender by age by we have scarcely begun to tap into the possibilities for such unconven- tional forms of data unlisted telephone numbers\") On the video screen analyzing the data before the interviewing is com- is an introduction (\"Hello, my name is .. \") and the plete, thereby gaining an advanced view of how first question to be asked (\"Could you tell me how the analysis will turn OUL Sill another innovation many people live at this address?\") that computer technology makes possible is de- scribed in \"Voice Capture.\" When the respondent answers the phone, the interviewer says hello, introduces the study, and Response Rates in Interview Surveys asks the first question displayed on the screen. When the respondent answers the question, the in- Earlier in this chapter we looked at the issue of re- terviewer tYVes that answer into the computer ter- sponse rates in mail surveys, and this is an equally minal-either the verbatim response to an open- important issue for interview surveys. In Chapter 7, ended question or the code category for the when we discussed formulas for calculating sam- appropriate answer to a closed-ended question\" The pling error to determine the accuracy of survey es- answer is immediately stored in the computer. The timates, the impliCit assumption was that everyone second question appears on the video screen, is selected in a sample would participate-which is asked, and the answer is entered into the com- almost never the case. Lacking perfection, re- puter. Thus, the interview continues. searchers must maximize participation by those se- lected. Although interview surveys tend to produce In addition to the obvious advantages in terms of data collection. CAT! automatically prepares the data for analysis; in fact, the researcher can begin

272 Chapter 9: Survey Research higher response rates than mail surveys do, inter- CASl (computer-assisted self-interviewing): A view success has recently declined. research worker brings a computer to the re- spondent's home, and the respondent reads By analyzing response-rate trends in the Uni- questions on the computer screen and enters versity of lv1ichigan's Survey of Consumer Atti- his or her own answers. tudes, Richard Curtin, Stanley Presser. and Eleanor Singer (2005) have sketched a pattern of general CSAQ (computerized self-administered ques- decline over recent years. Between 1979 and 1996, tionnaire): The respondent receives the ques- the response rate in this telephone survey dropped tionnaire via CD-ROM, bulletin board, or other from 72 to 60 percent representing an average an- means and runs the software, which asks ques- nual decline of three-quarters of a percent. Since tions and accepts the respondent's answers. The 1996, the rate of decline has doubled. The in- respondent then returns the data file. creased nonresponses reflected both refusals and those they were unable to contact TDE (touchtone data entry): The respondent initiates the process by calling a number at the By contrast, the General Social Survey, using research organization. This prompts a series of personal interviews, experienced response rates be- computerized questions, which the respondent tween 73.5 and 82.4 percent in the years from 1975 answers by pressing keys on the telephone to 1998. In the 2000 and 2002 surveys, however, keypad. the GSS completion rate was 70 percent. Their de- cline came primarily from refusals, because house- VR (voice recognition): Instead of asking the re- hold interviews produce higher rates of contact than spondent to use the telephone keypad, as in telephone surveys. TDE, this system accepts spoken responses. Most researchers believe that the widespread Nicholls and his colleagues report that such tech- growth of telemarketing has been a big part of the niques are more efficient than conventional tech- problems experienced by legitimate telephone sur- niques and that they do not appear to result in a veys, and there are hopes that the state and na- reduction of data quality. tional \"do not ca\\l\" lists may ease that problem. Further, we saw that other factors such as answer- Jeffery Walker (1994) has explored the possibil- ing machines also contribute to these problems ity of conducting surveys by fax machine. Ques- (Tuckel and O'Neill 2002). Response rate is likely to tionnaires are faxed to respondents, who are asked remain an issue of high concern in survey research. to fax their answers back. Of course, such surveys can only represent that part of the population New Technologies who have fax machines. Walker reports that fax and Survey Research surveys don't achieve as high a response rate as face-to-face interviews do, but, because of the per- As we have already seen in the case of computer- ceived urgency, they do produce higher response assisted telephone intervie'wing (CAT!), many of rates than mail or telephone surveys do. In one test the new technologies affecting people's lives also case, all those who had ignored a mail question- open new possibilities for survey research. For ex- naire were sent a fax follow-up, and 83 percent ample, recent innovations in self-administered responded. questionnaires make use of the computer. Among the techniques that are being tested are these I've already noted that, as a consumer of social (Nicholls, Baker. and Martin 1996): research, you should be wary of \"surveys\" whose apparent purpose is to raise money for the sponsor. CAPI (computer-assisted personal interview- This practice has already invaded the realm of \"fax ing): Similar to CAT! but used in face-to-face surveys,\" evidenced by a fax entitled \"Should Hand interviews rather than over the phone. Guns Be Outlawed?\" Two fax numbers were pro- vided for expressing either a \"Yes\" or \"No\" opinion. The smaller print noted, \"Calls to these numbers cost 52.95 per minute, a small price for greater

New Technologies and Survey Research 273 democracy. Calls take approx. 1 or 2 minutes.\" You Acknowledging the need for caution, they urged can imagine where the 52.95 went. that online polling be given a fair hearing: The new technology of survey research includes One test of the credibility of any new data col- the use of the Internet and the World Wide Web- lection method hinges on its ability to reliably two of the most far-reaching developments of the and accurately forecast voting behavior. For this late twentieth century. Some researchers feel that reason, last fall we attempted to estimate the the Internet can also be used to conduct meaningful 1998 election outcomes for governor and US survey research. Senate in 14 states on four separate occasions using internet surveys. Mick Couper (2001: 464) provides an excellent overview of the issues concerning the present and (1999: 20) prospective state of online surveys. The researchers compared their results with 52 tele- The rapid development of surveys on the World phone polls that addressed the same races. Online Wide Web (WWW) is leading some to argue polling correctly picked 21 of the 22 winners, or 95 that soon Internet (and, in particular, Web) percent. However. simply picking the vvinner is not surveys will replace traditional methods of sur- a sufficient test of effectiveness: How close did the vey data collection. Others are urging caution polls come to the actual percentages received by the or even voicing skepticism about the future various candidates? Taylor and Terhanian report role Web surveys will play. Clearly, we stand at their online polls missed the actual vote by an aver- the threshold of a new era for survey research, age of 6.8 percentage points. The 52 telephone polls but how this vvill play out is not yet clear. missed the same votes by an average of 6.2 percent- age points. As we have seen, one immediate objection that many social researchers make to online surveys Warren Mitofsky (1999) is a critic of online concerns representativeness: Will the people who polling. In addition to disagreeing with the way can be surveyed online be representative of mean- Taylor and Terhanian calculated the ranges of error ingful populations, such as all U.S. adults, all voters, just reported, he has called for a sounder, theoreti- and so on? This is the criticism raised with regard cal basis on which to ground the new technique. to surveys via fax and, earlier, with regard to tele- phone surveys. One key to online polling is the proper assess- ment and use of weights for different kinds of re- Camilo Wilson (1999), founder of Cogix spondents-as we discussed in the context of (http://vvww.cogix.com), points out that some quota sampling in Chapter 7. Taylor and Terhanian populations are ideally suited to online surveys: are aware of the criticisms of quota sampling, but specifically, those who visit a particular website. For their initial experiences with online polling suggest example, Wilson indicates that market research for to them that the technique should be pursued. In- online companies should be conducted online, and deed, they conclude by saying, \"This is an unstop- his firm has developed software called ViewsFlash pable train, and it is accelerating. Those who don't for precisely that purpose. Although website sur- get on board run the risk of being left far behind\" veys could easily collect data from all who visit a (1999: 23). particular site, Wilson suggests that survey sam- pling techniques can provide sufficient consumer Many of the cautions urged in relation to on- data without irritating thousands or millions of po- line surveys today are similar to those urged in re- tential customers. lation to telephone surveys in the first edition of this book, in 1975.lVlick Couper (2001: 466) But how about general population surveys? makes a similar observation: As I write this, a debate is brewing within the sur- vey research community. Humphrey Taylor and Several years ago, I predicted that the rapid George Terhanian (1999) prompted part of the de- spread of electronic data collection methods bate with an article, \"Heady Days Are Here Again.\" such as the Internet would produce a bifurcation

274 Chapter 9: Survey Research in the survey industry between high-quality afternoon, evening, or weekend, when respon- surveys based on probability samples and using dents are most likely to be reading mail at traditional data collection methods, on the one home, especially if the study requests respon- hand, and surveys focused more on low cost dents to check an item in the kitchen or other and rapid turnaround than on representative- area in the home. If a parent-child question- ness and accuracy on the otheL In hindsight I naire is planned, send the invitation late after- was wrong, and I underestimated the impact of noon when children are home, not early in the the Web on the survey industry. It has become day, when respondents can't complete the much more of a fragmentation than a bifurca- study because children are at school. tion (in terms of Web surveys at least), with vendors trying to find or create a niche for their Do be aware of technical limitations. For ex- particular approach or product. No longer is it ample, WebTV users currently cannot access just \"quick and dirty\" in one corner and \"ex- surveys using Java . If respondents' systems need pensive but high quality\" in the other; rather, to be Java-enabled or require access to stream- there is a wide array of approaches representing ing video, alert panelists at the beginning of the varying levels of quality and cost. study, not midway through. Whether online surveys will gain the respect Do test incentives, rewards, and prize drawings and extensive use enjoyed by telephone surveys to- to determine the optimal offer for best response. day remains to be seen. Students who consider us- Longer surveys usually require larger incentives. ing this technique should do so in full recognition of its potential shortcomings. Do limit studies to 15 minutes or less.* In the meantime, researchers are amassing a The web is already seeing extensive use as a body of experience with this new technique, yield- marketplace for surveys and other research tech- ing lessons for increasing success. For example, niques. As only a few illustrative examples, see the Survey Sampling, Inc suggests the following dos following: and don'ts for conducting online surveys: The Gallup Organization: Do use consistent wording between the invita- http://vvww. gallup. com/ tion and the survey. Don't use terms such as Harris Poll Online: \"unique ill number\" in the invitation, then ask http://wwvv.harrisinteractive.com/ respondents to type their \"password\" when SMS Research: http://www..smshawaiLcom/ they get to the survey. Changing terminology The Survey/Marketing Research e-Store: can be confusing. http://,vww.streamlinesurveys.com/ Streamline/estore/indexJ1tm Do use plain, simple language. Zogby International: http://www.zogby.com/ Don't force the respondent to scroll down the In another study of ways to improve response screen for the URi for the study location. rates in online surveys, Stephen Porter and Michael Whitcomb (2003) found that some of the tech- Do offer to share selected results from the niques effective in mail surveys, such as personaliz- study with everyone who completes the sur- ing the appeal or varying the apparent status of the vey. Respondents will often welcome informa- researcher. had little or no impact in the new me- tion as a reward for taking the study, especially dium. At the same time, specifying that the respon- when they are young adults and teens. dents had been specially selected for the survey Do plan the time of day and day of week to *Source: http://wwwworldopinion.com/the_frame/ maiL depending on the subject of the study and frame4 . htmL Reprinted with permission. type of respondent. Send the invitation late