DRAFT 11/24/99

Paying Respondents for Survey Participation

Eleanor Singer and Richard A. Kulka

1. The Problem: Surveying Welfare Leavers

In order to evaluate the effect of recent changes in welfare policy on the lives of people living at or below the poverty level, it is often necessary to survey a representative sample of them. As the workshop paper by Groves and Couper makes clear, achieving such a representative sample can be problematic both because members of low-income groups are hard to locate--they are more mobile, they are more likely to live in multi-family households, and they are less likely than the more affluent to have telephones--and because they may not be highly motivated to participate in surveys. Incentives--especially monetary incentives--are particularly useful in countering the second difficulty, as a supplement or complement to other efforts at persuasion. In this paper, we briefly consider why people participate in surveys (or fail to do so) and then review the use of incentives in counteracting certain kinds of nonresponse. We also review separately those findings that appear to be particularly relevant for low-income populations. Finally, we consider two special issues: The potential consequences of refusal conversion payments for respondents and interviewers, and the cost-effectiveness of prepaid incentives.

1.1 Why Do People Participate in Surveys?

Porst and von Briel (1995) point out that although a great deal is known about survey respondents--their demographic characteristics, as well as their answers to thousands of different survey questions--very little is known about why they choose to participate. Based on a content analysis of open-ended responses, their study of 140 participants in five waves of a German "Methods Panel" identifies three pure types of participants: (1) those who respond for altruistic reasons (e.g. the survey is useful for some purpose important to the respondent, or the respondent is fulfilling a social obligation--31% of respondents); (3) those who respond for survey-related reasons (e.g. they are interested in the survey topic, or find the interviewer appealing--38%); and (3) those who cite what the authors call "personal" reasons (e.g., they promised to do it--30%). In reality, of course, most people participate for a variety of reasons.

More recently, Groves, Singer, and Corning (1999) outlined a theory describing the decision to participate in a survey as resulting from a series of factors--some survey-specific, such as topic and sponsorship, others person-specific, such as concerns about privacy, still others specific to the respondent’s social and physical environment--each of which may move a particular person toward or away from cooperation with a specific survey request. Furthermore, these factors assume different weights for different persons, and they become salient for a specific individual--the potential respondent--when an interviewer calls to introduce the survey and request participation.

From this perspective, monetary as well as nonmonetary incentives are an inducement offered by the survey designer to compensate for the relative absence of factors that might otherwise stimulate cooperation--e.g., interest in the topic of the survey or a sense of civic obligation. Although other theoretical frameworks such as social exchange theory (cf. Dillman, 1978), the norm of reciprocity (Gouldner, 1960), and economic exchange (e.g. Biner and Kidd, 1994) can also be used to explain the effectiveness of incentives, the present perspective is able to account for the differential effects of incentives under different conditions (e.g., for respondents with differing interest in the survey topic or with different degrees of community activism) in a way that other theories cannot easily do.

1.2 Incentives and "Hard-to-Reach" Populations

As indicated above, members of a group may be hard to reach both because they are difficult to locate or to find at home, and because they have little motivation to participate in a survey. There is no empirical evidence that incentives are helpful in overcoming the first problem in a random digit dialed (RDD) survey, nor any theoretical justification for believing that they would or should be. Thus, if the primary problem is one of finding people at home for such a survey, incentives may not be very useful. However, an experiment by Kerachsky and Mallar (1981) with a sample of economically disadvantaged youths suggests that prepayment may be helpful in locating members of a list sample, especially in later waves of a longitudinal survey. One reason, apparently, is that prepayment (and perhaps promised incentives from a trusted source) may be useful in persuading friends or relatives to forward the survey organization’s advance letter or to provide interviewers with a current telephone number for the designated respondent.

The remainder of this paper is devoted to reviewing the evidence pertaining to the second reason for survey nonresponse--namely, the situation in which the respondent has little intrinsic motivation to respond to the survey request. This situation is likely to characterize many low-income respondents, especially those who no longer receive welfare payments as a result of changes in federal and state legislation. Hence, the findings reported in this paper about the effectiveness of prepaid monetary incentives are especially likely to apply to this population.

2. What Do We Know about the Effects of Incentives?

In this section we review what is known about the intended effects of incentives on response rates in mail as well as interviewer-mediated surveys, drawing on two existing meta analyses (Church, 1993; Singer et al., 1999) as well as subsequent work by the same and other authors. We specifically consider the usefulness of lotteries as an incentive and the use of incentives in panel studies. We also review what is known about such unintended consequences of incentives as effects on item nonresponse and response bias.

2.1 Effects on Response Rates

In an effort to counter increasing tendencies toward noncooperation, survey organizations are offering incentives to respondents with increasing frequency, some at the outset of the survey, as has traditionally been done in mail surveys, and some only after the person has refused, in an attempt to convert the refusal.

The use of incentives has a long history in mail surveys (for reviews, see Armstrong 1975; Church 1993; Cox 1976; Fox, Crask and Kim 1988; Heberlein and Baumgartner 1978; Kanuk and Berenson 1975; Levine and Gordon 1958; Linsky 1975; Yu and Cooper 1983). In such surveys, incentives are one of two factors, the other being number of contacts, that have consistently been found to increase response rates.

A meta analysis of the experimental literature on the effects of incentives in mail surveys by Church (1993) classifies incentives along two dimensions: whether the incentive is a monetary or nonmonetary reward; and whether it is offered with the initial mailing or made contingent on the return of the questionnaire. Analyzing 38 studies, Church concluded that:

Studies using prepaid monetary incentives yielded an average increase in response rates of 19.1 percentage points, representing a 65% average increase in response. Gifts, on the other hand, yielded an average increase of only 7.9 percentage points. The average value of the monetary incentive in the studies analyzed by Church was $1.38; the average value of the gift could not be computed, given the great diversity of gifts offered and the absence of information on their cost. Reports similar to those of Church are reported by Hopkins and Gullikson (1992).

Incentives are also increasingly used in telephone and face-to-face surveys, and the question arises whether their effects differ from those found consistently in mail surveys. A meta analysis of 39 experiments by Singer et al. (1999) indicates that they do not, although the percentage point gains per dollar expended are much smaller, on average, than those reported by Church. Their main findings are as follows:

2.2 Lotteries as Incentives

Some researchers, convinced of the value of incentives but reluctant to use prepaid incentives for all respondents, have advocated the use of lotteries as an incentive for stimulating response. The studies reported in the literature--all mail surveys or self-administered questionnaires distributed in person--have yielded inconsistent findings (e.g. positive effects by Balakrishnan et al., 1992; Hubbard and Little, 1988; Kim, Lee, and Whang, 1996; and McCool, 1991; no effects in four studies reviewed by Hubbard and Little, 1988 or in the experiment by Warriner et al., 1996). A reasonable hypothesis would seem to be that lotteries function as cash incentives with an expected value per respondent (e.g. a $500 prize divided by 10,000 respondents would amount to an incentive of 50 cents per respondent), and that their effect on response rates would be predicted by this value. Thus, the effect of lotteries would generally be small, both because the expected value per respondent is small, and because they are essentially promised, rather than prepaid, incentives.

2.3 Incentives in Panel Studies

Many studies of welfare leavers are panel studies--that is, they reinterview the same household, or the same respondent, more than once over a period of time. Assuring participation is especially important for panel studies, since participation at baseline usually sets a ceiling for the retention rate over the life of the panel. 1 For this reason, investigators often advocate using sizable incentives at the first wave of a panel study. An incentive experiment was carried out at Wave 1 of the 1996 Survey of Income and Program Participation (SIPP), a longitudinal survey carried out by the U.S. Census Bureau to provide national estimates of sources, amounts, and determinants of income for households, families, and persons. SIPP primary sample units were divided into three groups to receive 0$, $10, and $20. James (1997) found that the $20 incentive significantly lowered nonresponse rates in Waves 1-3 compared with both the $10 and the $0 conditions, but the $10 incentive showed no effect relative to the zero-incentive group. Mack et al. (1998) reported on the results through wave 6 using cumulative response rates, including an analysis of the effects of incentives on households differing by race, poverty status, and education in Wave 1. They found that an incentive of $20 reduced household, person, and item (gross wages) nonresponse rates in the initial interview and that household nonresponse rates remained significantly lower, with a cumulative 27.6% nonresponse rate in the $0 incentive group, 26.7% in the $10 group, and 24.8% in the $20 group at Wave 6, even though no further incentive payments were made. (SIPP does not attempt to reinterview households that do not respond in Wave 1 or that have two consecutive noninterviews.) Differences between the $10 incentive and the no-incentive group were not statistically significant. Interestingly enough, an experiment with paying incentives in Wave 8 of the 1996 SIPP to all Wave 7 nonrespondents (Abreu, Martin, and Winters 1999) found that a $40 prepayment significantly increased the response rate above that in both the $20 and the $0 groups, but there was no significant difference between the $20 group and the control group. Thus, not surprisingly, it costs more to convert a refusal than to persuade respondents to participate in the first place. (Differential responsiveness to incentives by respondents differing in economic status is discussed in the section on "Findings for Low-Income Populations," below.)

Research on the Health and Retirement Survey (HRS) suggests that respondents who are paid a refusal conversion incentive during one wave do not refuse at a higher rate than other converted refusers when reinterviewed during the next wave (Lengacher et al., 1995).

Unlike the SIPP, all respondents to the HRS receive an incentive at each wave. The SIPP results suggest that this may not be necessary, but it is possible that higher panel retention rates are obtained when that is done. It is also possible that in the SIPP, alternative factors--such as government authority--compensate for the absence of incentives in subsequent waves.

2.4 Effects on Respondents, or Effects on Interviewers?

Are the consistent effects of incentives in telephone and face-to-face interviews attributable to their effect on respondents, or are they, perhaps, mediated by their effect on interviewers? Clearly, this question does not arise with respect to mail surveys, where incentives have also been consistently effective; but it seems important to try to answer it with respect to interviewer-mediated surveys. It is possible, for example, that interviewers expect respondents who have received an incentive to be more cooperative, and that they behave in such a way as to fulfill their expectations. 2 Or, they may feel more confident about approaching a household that has received an incentive in the mail, and therefore be more effective in their interaction with the potential respondent.

In order to separate the effects of incentives on interviewers from their effects on respondents, Singer, Van Hoewyk, and Maher (1999) randomly divided all sample numbers in an RDD survey which could be linked to addresses into three groups. One third of the group was sent an advance letter and $5; interviewers were kept blind to this condition. Another third also received a letter plus $5, and still another third received the letter only. Interviewers were made aware of these last two conditions by information presented on their Computer-Assisted Telephone Interview (CATI) screens.

The results of this experiment are shown in Table 1. Large differences were observed between the letter-only and the letter-plus-incentive conditions, but there is no evidence that this is due to the effect of incentives on interviewers. Only one of the differences between the conditions in which interviewers are aware of the incentive and those in which they are not aware reaches statistical significance, and here the results are in a direction opposite to that hypothesized. Thus, prepayment of a $5 incentive substantially increases cooperation with an RDD survey, and the incentive appears to exert its effect directly on the respondent rather than being mediated through interviewer expectations. This conclusion is in accordance with research by Stanley Presser and Johnny Blair, at the University of Maryland, who also found substantial increases in response rates as a result of small prepayments to respondents to which interviewers were blind (personal communication).

3. Unintended Consequences of Incentives

3.1 Effects on Item Nonresponse

One question often raised about the use of incentives in surveys is whether they bring about an increase in response rate at the expense of response quality. This does not appear to be the case. On the contrary, what evidence there is suggests that the quality of responses given by respondents who receive a prepaid or a refusal conversion incentive does not differ from responses given by those who do not receive an incentive. They may, in fact, give better-quality answers, in the sense that they have less item-missing data and provide longer open-ended responses (Baumgartner et al., 1998; Singer, Van Hoewyk, and Maher, 1999; Shettle and Mooney, 1999; but cf. Wiese, 1998). Experiments reported by Singer, Van Hoewyk, and Maher (1999) indicate that promised and prepaid incentives reduce the tendency of older people and nonwhites to have more item missing data, resulting in a net reduction in item nonresponse.

Findings reported by Mason and Traugott (1999)suggest that persistent efforts to persuade reluctant respondents to participate may produce more respondents at the price of more missing data. But these authors did not use incentives, and motivational theory suggests that people who are rewarded for their participation would continue to give good information, whereas those who feel harassed into participation may well retaliate by not putting much effort into their answers.

Thus, there is no evidence that the offer of an initial incentive or a refusal conversion payment increases the unit response rate at the price of an increase in item nonresponse--if anything, such payments seem to reduce the amount of item missing data, especially among certain groups, although the effects are generally very small. There is no evidence about the effect of incentives on validity or reliability, and this is an important research question.

3.2 Effects on Response Bias

Even more troubling, potentially, than an effect on item missing data is the effect of incentives on the distribution of responses. Does offering or paying incentives to bring into the sample people who might otherwise refuse bias their answers to the survey questions?

It is useful to think about the reasons for which response bias might come about. One is that the use of incentives brings into the sample people whose characteristics differ from those who would otherwise be included, and their answers differ because of those differing characteristics. If that is the case, the apparent response bias is really due to a change in the composition of the sample, and should disappear once the appropriate characteristics are controlled. An example of the first process is presented Berlin et al. (1992), who demonstrate that the apparent effect of a monetary incentive on literacy scores can be accounted for by the disproportionate recruitment of respondents with higher educational levels into the zero-incentive group. There was no significant relationship between incentive level and the proportion of items attempted, indicating that the incentive influenced the decision to participate, but not performance on the test. Another example is presented by Merkle, Edelman, Dykeman, and Brogan (1998) in their report of an experimental effort to increase the response rate to exit polls by having interviewers in a random sample of precincts carry clipboards and folders clearly identifying them as associated with the major media and handing out pens with the same logo. Although the response rate was increased by these methods (not necessarily by the incentive alone), the response bias also increased because a greater number of Democratic voters were brought into the sample--apparently as a result of the clearer identification of the poll with the media. Effects of incentives on sample composition are discussed further in the following section.

A second reason incentives might bias responses is if they influence people’s opinions directly, or at any rate the expression of those opinions. A striking example of such influence (not, however, involving an incentive) is reported by Bischoping and Schuman (1992) in their analysis of discrepancies among Nicaraguan pre-election polls in the 1990 election and the failure of many to predict the outcome of the election accurately. Bishoping and Schuman speculate that suspicions that pre-election polls had partisan aims may have prevented many Nicaraguans from candidly expressing their voting intentions to interviewers, and tested this hypothesis by having interviewers alternate the use of three different pens to record responses: one carried the slogan of the Sandinista party; another, that of the opposition party; the third pen was neutral. The expected distortions of responses were observed in the two conditions that clearly identified the interviewers as partisan. Even in the third, neutral, condition, however, distortion occurred. The authors claim that "polls were apparently not perceived as neutral by many respondents. In the Nicaraguan setting, after a decade of Sandinista rule, a poll lacking partisan identification was evidently regarded as likely to have an FSLN [Sandinista] connection (p. 346)"; the result was to bias the reporting of vote intentions, and therefore the results of the pre-election polls, which predicted an overwhelming Sandinista victory when in fact the opposition candidate won by a large majority.

Still a third way in which incentives might bias responses is suggested by theory and experimental findings about the effects of mood (Schwarz and others: get citations). If incentives put respondents in a more optimistic mood, then some of their responses may be "biased" as a result. Using the same 17 key variables included in the Index of Nonresponse, Singer, Van Hoewyk and Maher (1999) looked at whether the response distributions varied significantly by (a) the initial incentive or (b) refusal conversion payments. They used the multinomial logit specification in CATMOD 3 and again controlled for the same set of demographic characteristics as in the analysis of item nonresponse.

The offer of an initial incentive was associated with significantly different response distributions (at the .05 level) on 4 of the 17 variables; a refusal conversion payment was also associated with significantly different response distributions on 4 of them. One variable was significantly affected by both types of incentives. 4 In five of these cases, the responses given with an incentive were more optimistic than those given without an incentive; in two cases, they were more pessimistic. In the remaining case, respondents who received an incentive were somewhat more likely to respond "good" and "bad," and somewhat less likely to give an equivocal reply. Thus, there is a suggestion that respondents to the Survey of Consumer Attitudes who receive an incentive may give somewhat more optimistic responses than those who do not. Similar findings have been reported by Brehm (1994) and by James and Bolstein (1990). However, no response bias was observed by Shettle and Mooney (1999) in their experimental investigation of incentives in a survey of college graduates, which found only 8 significant differences (at the .05 level) in response distributions to 148 questions--a number that does not differ from that expected on the basis of chance.

4. Effects on Low-Income Populations

The question of particular interest to this audience is how effective monetary and other incentives are in recruiting and retaining members of low-income populations. In a 1994 paper presented to a COPAFS workshop, Kulka reported some evidence suggesting that monetary incentives might be especially effective in recruiting into the sample low-income and minority respondents, groups that ordinarily would be underrepresented in a probability sample. Reviewing a number of experimental studies that provided evidence on the issue of sample composition, including the studies discussed by Kulka, Singer et al. (1999) found that in three studies, there was an indication that paying an incentive might be useful in obtaining higher numbers of respondents in demographic categories that otherwise tend to be underrepresented in sample surveys (e.g., low income or nonwhite race). Five other studies reported no significant effects of incentives on sample composition, and in one study the results were mixed.

Since then, additional evidence has accumulated suggesting that monetary incentives can be effective in recruiting and retaining minority respondents. Mack et al. (1998) found that the use of a $20 incentive in the first wave of a SIPP panel was much more effective in recruiting and retaining black households and households in poverty that it was in recruiting and retaining non-black and non-poverty households. 5 And a subsequent study by Abreu, Martin, and Winters (1999) found that an incentive of $20 significantly increased the recruitment of low-income nonrespondents to a previous wave of a SIPP panel, whereas $40 was required to produce a significant difference among nonrespondents with higher incomes. Both sets of results are in agreement with findings reported by Juster and Suzman (1995). They report that a special Nonresponse Study, in which a sample of people who refused "normal" refusal conversion efforts on the Health and Retirement Survey were offered $100 per individual or $200 per couple to participate, 6 brought into the sample a group of people distinctly different from other participants: they were more likely to be married, in better health, and, particularly, they had about 25% more net worth and a 16% higher income than other refusal conversion households or those who never refused. Finally, analyses by Singer, Van Hoewyk, and Maher (1999) indicate that a $5 incentive paid in advance to a random half of RDD households for which an address could be located brought a disproportionate number of low-education respondents into the sample; there were no significant differences on other demographic characteristics.

In other words, these studies suggest that, while monetary incentives are effective with all respondents, less money is required to recruit and retain low-income groups than those whose income is higher, and for whom the tradeoff between the time required for the survey and the incentive offered may be less attractive when the incentive is "small." It should be noted that few, if any, of these studies (Mack [1998] is a notable exception) have explicitly manipulated both the size of the incentive and the income level of the population; the findings reported above are based on ex post facto analyses for different subgroups, or on analyses of the composition of the sample following the use of incentives.

A number of other studies have also reported on the effects of incentives on sample composition. In some of these, it appears that incentives can be used to compensate for lack of salience of, or interest in, the survey by some groups in the sample. For example, the survey reported on by Shettle and Mooney (1999), the National Survey of College Graduates, is believed to be much more salient to scientists and engineers than to other college graduates, and in the 1980's the latter had a much lower response rate. Although this was also true in the 1992 pretest for the 1993 survey, the bias was less in the incentive than in the nonincentive group (7.1 percentage-point underreporting, compared with 9.8 percentage points), though not significantly so.7 Similar findings are reported by Baumgartner and Rathbun (1997), who found a significant impact of incentives on response rate in the group for which the survey topic had little salience, but virtually no impact in the high-salience group; and by Martinez-Ebers (1997), whose findings suggest that a $5 incentive, enclosed with a mail questionnaire, was successful in motivating less-satisfied parents to continue their participation in a school-sponsored panel survey. Berlin et al. (1992) found that people with higher scores on an assessment of adult literacy, as well as people with higher educational levels, were overrepresented in their zero-incentive group. Groves, Singer, and Corning (1999) also reported a similar result; in their study, the impact of incentives on response rates was significantly greater for people low on a measure of community involvement than for those high on community involvement, who tend to participate at a higher rate even without monetary incentives. In these studies, incentives function by raising the response rate of those with little interest, or low civic involvement; they do not reduce the level of participation of the highly interested or more altruistic groups.

In all of these studies, certain kinds of dependent variables would be seriously mismeasured if incentives had not been used. In the case of Groves, Singer, and Corning (1999), for example, the conclusions one would reach about the distribution of community involvement would be in error if drawn from a survey that did not use incentives. Nevertheless, questions remain about how representative of their group as a whole those brought into the sample by incentives are, and this is true for low-income and minority respondents, as well. In other words, low-income respondents brought into the sample by the lure of an incentive may well differ from those who participate for other reasons. But even if prepaid incentives simply add more respondents to the total number interviewed, without reducing the nonresponse bias of the survey, they may still prove to be cost-effective if they reduce the effort required to achieve a desired sample size. And the theory of survey participation outlined at the beginning of this paper (Groves, Singer, and Corning, 1999) suggests that the representativeness of the sample will be increased by using a variety of motivational techniques, rather than relying on a single one.

5. Issues in the Use of Differential Incentives

Some of the research reported above suggests that it may make economic sense to offer lower incentives to people with lower incomes and higher incentives to those who are economically better off. Another instance of differential incentives is the use of refusal conversion payments, in which respondents who have expressed reluctance, or who have actually refused, are offered payment for their participation whereas cooperative respondents are not. In both of these situations, the question arises how respondents who received lower, or no, rewards would feel if they learned of this practice, and how this might affect their future participation in this or another survey.

5.1 Effects of Disclosure of Differential Incentives on Perceptions of Fairness

From an economic perspective, the fact that some people refuse to be interviewed may be an indication that the survey is more burdensome for them and that therefore the payment of incentives to such respondents (but not others) is justified. Nevertheless, some researchers are concerned that using incentives in this way will be perceived as inequitable by cooperative respondents, and, that, if they learn of the practice, this will adversely affect their willingness to cooperate in future surveys (Kulka 1994).

These unintended consequences were the focus of two studies (Singer, Groves, and Corning, 1999; Groves, Singer, Corning, and Bowers, 1999), the first conducted as part of the Detroit Area Study, using face-to-face interviews, the second done in the laboratory with community volunteers, using self-administered responses to videotaped vignettes.

In the first of these studies, respondents were asked a series of questions concerning their beliefs about survey organizations’ practices with respect to incentives. Three quarters believed that such organizations offer monetary incentives to respondents to encourage participation (8.9% said they did not know). Those who received a prepaid $5 incentive (a random two thirds of the survey sample) were significantly more likely than those who received no such payment to say that at least some survey organizations use incentives. Indeed, beliefs about this practice appeared to increase with the total amount ($0, $5, $25, or $30) of the incentive the respondent received or was offered, with 94% of those who received $30 expressing the belief that at least some survey organizations use incentives. 8

All respondents were also asked the following question: "Some people do not want to be interviewed. However, to get accurate results, everyone chosen for the survey needs to be interviewed. Otherwise, the data may mislead people in the government who use the conclusions to plan important programs that affect everyone. Do you think it’s fair or unfair for people who refuse to be interviewed to receive money if other people don’t?" Despite the extensive justification for differential payment included here, 74% said they considered the practice unfair.

Near the end of the survey, in a more stringent test of whether or not the payment of differential incentives was perceived as fair or unfair, a random half of the respondents were informed that because of the importance of including everyone in the sample, some of those who had expressed reluctance to participate had been offered $25, while others had received nothing; and they were asked whether they considered this practice fair or unfair. Again, almost three quarters (72.4%) said they considered the practice unfair.

5.2 Effects of Disclosure of Differential Incentives on Willingness to Participate

Singer, Groves, and Corning (1999) hypothesized that those to whom the payment of differential incentives was disclosed would be less willing to participate in a future survey. In the laboratory study described above, subjects were significantly more likely to say they would not be willing to participate in a survey after watching a videotape in which the respondent discovered that a friend had received payment for participating whereas s/he had not, than those who watched a videotape in which no such disclosure occurred. However, the difference was reduced to insignificance when an explanation for the payment was offered by the interviewer.

In the field study, there were no differences in expressed willingness to participate between those to whom differential payments had been disclosed and those to whom they had not: About a quarter of each group said they would "definitely" be willing to participate in another survey by the same organization. Even those to whom differential incentive payments were disclosed and who perceived these payments as unfair did not differ significantly in their expressed willingness to participate in a subsequent survey by the same organization, although the trend in responses was as predicted: 25.8% vs. 32.8% expressed such willingness. 9 The investigators speculated that rapport with the interviewer might have mitigated the deleterious effects of disclosing differential incentives that had previously been observed in the laboratory experiment (Groves, Singer, Corning, and Bowers, 1999).

A little more than a year later, all the original DAS respondents for whom an address could be located were sent a mail questionnaire on the topic of assisted suicide, ostensibly from a different survey organization, but there were no significant differences in participation between those to whom differential payments had been disclosed a year earlier, and those to whom they had not. 10

Thus, the data indicate that most respondents believe survey organizations are currently using incentives to encourage survey participation; that these beliefs are affected by personal experience; that only half of those who are aware of the use of incentives believe that payments are distributed equally to all respondents; and that a large majority of respondents perceive the practice of paying differential incentives as unfair. However, disclosure of differential payments had no significant effect on expressed willingness to participate in a future survey, nor were respondents to whom differential incentives had been disclosed significantly less likely to respond to a new survey request, from an ostensibly different organization, a year later, although again the differences were in the hypothesized direction.

6. Are Prepaid Incentives Cost-Effective?

For a variety of reasons, including those discussed in the section immediately above, prepaid incentives to everyone in the sample may be preferable to refusal conversion or other differential payments.

One reason is that interviewers like them. Knowing the household is in receipt of an advance payment, modest though it may be, they feel entitled to ask the respondent to reciprocate with an interview. Furthermore, prepaid incentives are equitable--they reward equally everyone who happens to fall into the sample, and they reward them for the "right" behavior--i.e., for cooperation, rather than refusal. Both of these advantages are likely to make modest prepaid incentives an attractive alternative to refusal conversion payments in many types of surveys. There is also indirect evidence that the use of refusal conversion payments to persuade reluctant respondents leads to increasing reliance on such payments within an organization, in all likelihood because of their effects on interviewer expectations. 11

Still, the question arises whether such incentives are cost-effective. On the face of it, it would appear that paying a small number of refusal conversion payments to reluctant respondents would be cheaper than paying everyone, even if those initial payments are smaller.

Several studies have concluded that prepaid incentives are cost-effective in mail surveys. For such surveys, the comparison ordinarily has been among incentives varying in amount or in kind, or in comparison with no incentive at all, rather than with refusal conversion payments. Two recent investigations of cost-effectiveness, by James and Bolstein (1992) and by Warriner et al. (1996), have included information on the relative effectiveness of various incentives. James and Bolstein (1992) found that a prepaid incentive of $1 was the most cost-effective, yielding nearly as high a return as larger amounts for about one quarter of the cost. Warriner et al. (1996:9) conclude that for their study, a $5 prepaid incentive was the optimal amount, resulting in a saving of 40 cents per case (because the same response rate could be achieved as in a no-incentive, two-follow-up condition). The $2 incentive resulted in costs per case only a dollar less than the $5 incentive, while yielding a response rate 10 percentage points lower. Similar findings have been reported by Asch, Christalis, and Ubel (1998) in a mail survey of physicians.

For interviewer-mediated studies, as noted above, the comparison is much more likely to be with refusal conversion payments, and the answer is likely to depend on the nature of the study and the importance of a high response rate, on how interesting the study is to respondents (i.e., how many of them are willing to participate even without a prepaid incentive), on whether or not prepaid incentives reduce the effort required, and on a variety of other factors.

Several face-to-face surveys have reported that promised monetary incentives are cost-effective. Berlin et al. (1992), for example, reported that use of a $20 promised incentive in a field test experiment with the National Adult Literacy Survey, which entails completion of a test booklet by the respondent, resulted in a cost saving per interview over the $0 and $35 incentive conditions when all field costs were taken into account. Similarly, Chromy and Horvitz (1978) reported (in a study of the use of monetary incentives amoung young adults in the National Assessment of Educational Progress) that when the cost of screening for eligible respondents is high, the use of incentives to increase response rates may actually reduce the cost per unit of data collected.

Singer, Van Hoewyk, and Couper 12 investigated this problem in the Survey of Consumer Attitudes. They found that a $5 incentive included with an advance letter significantly reduced the number of calls required to close out a case (8.75 calls when an incentive was sent, compared with 10.22 when it was not; p=.05), and significantly reduced the number of interim refusals (.282 refusals when an incentive was sent, compared with .459 when it was not). As expected, there was no significant difference between the incentive and the no-incentive condition in calls to first contact. The outcome of the first call indicates that compared with the letter only, the addition of a $5 incentive results in more interviews, more appointments, and fewer contacts in which resistance is encountered.

Given the size of the incentive and the average cost per call aside from the incentive, sending a prepaid incentive to respondents for whom an address could be obtained was cost effective for the SCA. However, as we have tried to indicate, this conclusion depends on the size of the incentive as well as the structure of other costs associated with a study for a given organization, and should not be assumed to be invariant across organizations and incentives.

An argument that can be raised against the use of prepaid incentives is that they may undermine more altruistic motives for participating in surveys. Indeed, we have found that prepaid incentives have smaller effects on survey participation for people who score high on a measure of community activism (Groves, Singer, and Corning, 1999) than on people low on this characteristic. But this is because groups high in community activism already respond at a high rate. There is no evidence -- because we did not test this hypothesis -- that people high on community activism who are offered a prepaid incentive respond at a lower rate than they would have had they not been offered the incentive, nor do we know whether such an effect would appear on a later survey. Although there is anecdotal evidence that some people are offended by the offer of an incentive, going so far as to return the incentive to the survey organization, by all accounts such negative reactions are few.

Prepaid incentives have been common in mail surveys for many years, although the amounts used are ordinarily quite modest (see Church 1993). We suspect that the use of such incentives will increase in interviewer-mediated surveys as well. Such incentives are likely to be especially appropriate when other reasons that might move potential respondents to participate are weak or lacking, and when the names and addresses (or telephone numbers) or such potential respondents are known.

FOOTNOTES

References

Abreu, Denise A., Elizabeth Martin, and Franklin Winters. 1999. "Money and Motive: Results of an Incentive Experiment in the Survey of Income and Program Participation." Paper presented at the International Conference on Survey Nonresponse, Portland, Oregon.

Armstrong, J. S. 1975. "Monetary Incentives in Mail Surveys." Public Opinion Quarterly 39:111-16.

Asch, Christalis, and Ubel

Balakrishnan, P.V., S.K. Chawla, M.F. Smith, and B.P. Micholski. 1992. "Mail Survey Response Rates Using a Lottery Prize Giveaway Incentive." Journal of Direct Marketing 6:54-59.

Baumgartner, Robert and Pamela Rathbun. 1997. "Prepaid Monetary Incentives and Mail Survey Response Rates." Paper presented at the Annual Conference of the American Association of Public Opinion Research, Norfolk, Virginia.

Baumgartner, Robert, Pamela Rathbun, Kevin Boyle, Michael Welsh, and Drew Laughland. 1998. "The Effect of Prepaid Monetary Incentives on Mail Survey Response Rates and Response Quality." Paper presented at the Annual Conference of the American Association of Public Opinion Research, St. Louis, Missouri.

Biner, Paul M. and Heath J. Kidd. 1994. "The Interactive Effects of Monetary Incentive Justification and Questionnaire Length on Mail Survey Response Rates." Psychology and Marketing 11:483-92.

Bischoping, Katherine and Howard Schuman. 1992. "Pens and Polls in Nicaragua: An Analysis of the 1990 Preelection Surveys." American Journal of Political Science 36:331-50.

Berlin et al. 1992.

Chromy and Horvitz. 1978

Brehm, John. 1994.

Church, Allan H. 1993. "Estimating the Effect of Incentives on Mail Survey Response Rates: A Meta-Analysis." Public Opinion Quarterly 57:62-79.

Cox, Eli P. 1976. "A Cost/Benefit View of Prepaid Monetary Incentives in Mail Questionnaires." Public Opinion Quarterly 40:101-4.

Dillman, Don A. 1978. Mail and Telephone Surveys: The Total Design Method. New York: John Wiley and Sons.

Fox, Richard J., Melvin Crask, and Jonghoon Kim. 1988. "Mail Survey Response Rate: A Meta-Analysis of Selected Techniques for Inducing Response." Public Opinion Quarterly 52:467-91.

Gouldner, Alvin W. 1960. "The Norm of Reciprocity: A Preliminary Statement." American Journal of Sociology 25:161-78.

Groves, Robert M., Eleanor Singer, and Amy D. Corning. 1999. "Decision-Making in Survey Participation: Theory and a Test." Unpublished manuscript.

Groves, Robert M., Eleanor Singer, Amy D. Corning, and Ashley Bowers. 1999. "A Laboratory Approach to Measuring the Effects on Survey Participation of Interview Length, Incentives, Differential Incentives, and Refusal Conversion." Journal of Official Statistics 15: 251-68.

Heberlein, Thomas A. and Robert Baumgartner. 1978. "Factors Affecting Response Rates to Mailed Questionnaires: A Quantitative Analysis of the Published Literature." American Sociological Review 43:447-62.

Hopkins, K.D. and Gullickson, A.R. 1992. "Response Rates in Survey Research: A Meta-Analysis of Monetary Gratuities." Journal of Experimental Education 61:52-6

Hubbard, Raymond and Eldon L. Little. 1988. "Promised Contributions to Charity and Mail Survey Responses: Replication With Extension." Public Opinion Quarterly 52:223-30.

Hyman, Herbert H. 1954. Interviewing in Social Research. Chicago: University of Chicago Press.

James, Tracy. 1997. "Results of the Wave 1 Incentive Experiment in the 1996 Survey of Income and Program Participation." Proceedings of the Survey Research Section of the American Statistical Association.

James, Jeannine M. And Richard Bolstein. 1990. "The Effect of Monetary Incentives and Follow-up Mailings on the Response Rate and Response Quality in Mail Surveys." Public Opinion Quarterly 54:346-61.

__________. 1992. "Large Monetary Incentives and Their Effect on Mail Survey Response Rates." Public Opinion Quarterly 56:442-53.

Juster, F. Thomas and Richard Suzman. 1995. "An Overview of the Health and Retirement Study." The Journal of Human Resources 30 (Supplement 1995):49-.

Kanuk, L. and C. Berenson. 1975. "Mail Surveys and Response Rates: A Literature Review." Journal of Marketing Research 12:440-53.

Kerachsky, Stuart H. And Charles D. Mallar. 1981. "The Effects of Monetary Payments on Survey Responses: experimental evidence from a Longitudinal Study of Economically Disadvantaged Youths." In Proceedings of Survey Research Methods Section of American Statistical Association, pp. 258-63.

Kim, K., C. Lee, and Y. Whang. 1995. "The Effect of Respondent Involvement in Sweepstakes on Response Rates in Mail Surveys." Proceedings of the Section on Survey Research Methods, American Statistical Association, pp. 216-20.

Kulka, Richard A. 1994. "The Use of Incentives to Survey ‘Hard-to-Reach’ Respondents: A Brief Review of Empirical Research and Current Practice." Paper prepared for Seminar on New Directions in Statistical Methodology, Bethesda, MD.

Lengacher, Jennie E., Colleen M. Sullivan, Mick P. Couper, and Robert M. Groves. 1995. "Once Reluctant, Always Reluctant? Effects of Differential Incentives on Later Survey Participation in a Longitudinal Study." Paper presented at the Annual Conference of the American Association for Public Opinion Research, Fort Lauderdale, Florida.

Levine, S. and G. Gordon. 1958. "Maximizing Returns on Mail Questionnaires." Public Opinion Quarterly 22:568-75.

Linsky, Arnold S. 1975. "Stimulating Responses to Mailed Questionnaires: A Review." Public Opinion Quarterly 39:82-101.

Lynn, Peter. 1999. "Is the Impact of Respondent Incentives on Personal Interview Surveys Transmitted via the Interviewers?" Unpublished manuscript.

McCool, Steven F. 1991. "Using Probabilistic Incentives to Increase Response Rates to Mail Return Highway Intercept Diaries." Journal of Travel Research 30: 17-19.

Mack, Stephen, Vicki Huggins, Donald Keathley, and Mahdi Sundukchi. 1998. "Do Monetary Incentives Improve Response Rates in the Survey of Income and Program Participation?" Proceedings of the Section on Survey Methodology, American Statistical Association, pp. 529-34.

Martinez-Ebers, Valerie. 1997. "Using Monetary Incentives with Hard-to-Reach Populations in Panel Surveys. International Journal of Public Opinion Research 9:77-86.

Merkle, Daniel, Murray Edelman, Kathy Dykeman, and Chris Brogan. 1998. "An Experimental Study of Ways to Increase Exit Poll Response Rates and Reduce Survey Error." Paper presented at the Annual Conference of the American Association of Public Opinion Research, St. Louis, Missouri.

Porst, Rolf and Christa von Briel. 1995. "Waren Sie vielleicht bereit, sich gegenebenfalls noch einmal befragen zu lassen? Oder: Gründe für die Teilnahme an Panelbefragungen." ZUMA-Arbeitsbericht, Nr. 95/04. Mannheim, Germany.

Presser, Stanley. 1989. In Panel Surveys, ed. Kaspryzk, Duncan, Kalton, and Singh, New York, Wiley.

Shettle, Carolyn and Geraldine Mooney. 1999. "Monetary Incentives in Government Surveys." Journal of Official Statistics 15:231-50.

Singer, Eleanor, Martin R. Frankel, and Marc B. Glassman. 1983. "The Effect of Interviewers' Characteristics and Expectations on Response." Public Opinion Quarterly 47:68-83.

Singer, Eleanor, Nancy Gebler, Trivellore Raghunathan, John Van Hoewyk, and Katherine. McGonagle. 1999. "The Effect of Incentives in Interviewer-Mediated Surveys." Journal of Official Statistics 15:217-30.

Singer, Eleanor, Robert M. Groves, and Amy D. Corning. 1999. "Differential Incentives: Beliefs About Practices, Perceptions of Equity, and Effects on Survey Participation." Public Opinion Quarterly 63:251-60.

Singer, Eleanor and Luane Kohnke-Aguirre. 1979. "Interviewer Expectation Effects: A Replication and Extension." Public Opinion Quarterly 43:245-60.

Singer, Eleanor, John Van Hoewyk, and Mary P. Maher. 1998. "Does the Payment of Incentives Create Expectation Effects?" Public Opinion Quarterly 62:152-64.

____ 1999. "Experiments with Incentives in Telephone Surveys." Unpublished manuscript.

Sudman, Seymour, Norman M. Bradburn, Ed Blair, and Carol Stocking. 1977. "Modest Expectations: The Effects of Interviewers' Prior Expectations on Responses." Sociological Methods and Research 6:171-82.

Warriner, Keith, John Goyder, Heidi Gjertsen, Paula Hohner, and Kathleen McSpurren. 1996. "Charities, No, Lotteries, No, Cash, Yes: Main Effects and Interactions in a Canadian Incentives Experiment." Paper presented at the Survey Non-Response Session of the Fourth International Social Science Methodology Conference, University of Essex, Institute for the Social Sciences, Colchester, UK.

Wiese, Cheryl J. 1998. "Refusal Conversions: What Is Gained?" National Network of State Polls Newsletter:1-3.

Yu, Julie and Harris Cooper. 1983. "A Quantitative Review of Research Design Effects on Response Rates to Questionnaires." Journal of Marketing Research 2036-44.Table 1. Response and Cooperation Rates by Advance Letters and Letters Plus Prepaid Incentive, Controlling for Interviewer Expectations

	Response Rate (a,b)			Cooperation Rate (b,c)
	Interviewed %	Not Interviewed % (n)		Interviewed %	Not Interviewed % (n)
May 1998
Letter Only	62.9	37.1(62)		68.4	31.6(57)
Letter + $5 (interviewers blind)	75.4	24.6(69)		86.7	13.3(60)
Letter +$5 (interviewers not blind)	78.7	21.3(61)		82.8	17.2(58)
Ltr only vs. ltr +$5		X2=4.13, df=1, p<.05		X2=6.27, df=1, p<.05
Blind vs. not blind		n.s.		n.s.
June 1998
Letter only	58.2	41.8(55)		62.8	37.2(51)
Letter + $5 (interviewers blind)	73.8	26.2(61)		86.5	13.5(52)
Letter + $5 (interviewers not blind)	74.6	25.4(59)		83.0	17.0(53)
Ltr only vs. ltr +$5		X2=4.52, df=1, p<.05		X2=9.56, df=1, p<.01
Blind vs. not blind		n.s.		n.s.
July 1998
Letter only	61.8	38.2(55)		72.3	27.7(47)
Letter + $5 (interviewers blind)	81.3	18.6(59)		87.3	12.7(55)
Letter + $5 (interviewers not blind)	69.9	30.4(56)		72.2	27.8(54)
Ltr only vs. ltr +$5		X2=3.47, df=1, p=.06		n.s.
Blind vs. not blind		n.s.		X2=5.83, df=1, p<.10
August 1998
Letter only	63.8	36.2(58)		69.8	30.2(53)
Letter + $5 (interviewers blind)	75.0	25.0(68)		81.0	19.0(63)
Letter + $5 (interviewers not blind)	76.7	23.3(60)		85.2	14.8(54)
Ltr only vs. ltr +$5		X2=2.85, df=1, p=.09		X2=3.75, df=1, p=.05
Blind vs. not blind		n.s.		n.s.

a : Includes noncontacts in denominator.

b: After refusal conversion.

c: Excludes noncontacts from denominator.