Low_income-Cantor

David Cantor and Patricia Cunningham
Westat
1650 Research Bld.
Rockville, MD. 20850

Paper presented at the Workshop on Data Collection for Low Income Populations, December 16, 17, 1999. Washington D.C.

The purpose of this paper is to review methods used to conduct telephone surveys of low income populations. The primary motivation for this review is to provide information on "best practices" as it applies to studies that are currently being conducted to evaluate the Personal Responsibility and Work Opportunity Reconciliation Act of 1996 (PRWORA - hereafter referred to as "Welfare Reform"). The National Academy of Sciences (NAS) panel observed that many of the states are conducting telephone surveys for this purpose and that it would be useful to provide them with information on the best methods to use to maximize response rates. The information provided in this paper is intended to assist these individuals, as well as others, to either conduct these studies themselves or to evaluate/monitor contractors who are.

For purposes of this paper, we have divided the telephone surveys into two types. The first, primary, method is to sample welfare recipients or welfare leavers from agency lists. This can take the form of a randomized experiment, where recipients are randomly assigned to different groups at intake, with a longitudinal survey following these individuals over an extended period of time. More commonly, it takes the form of a survey of those leaving welfare during a particular period (e.g., first quarter of the year). These individuals are then followed up after "X" months to assess how they are coping with being off of welfare.

The second type of telephone survey is one completed using a sample generated by random digit dialing methods (RDD). In this type of study, the telephone numbers are randomly generated. The numbers are then called and interviews are completed with those numbers that represent residential households and that agree to participate in the interview. To effectively evaluate welfare reform, this type of survey would attempt to oversample persons that are eligible and/or who are participating on welfare programs.

The issues related to these two types of telephone surveys, one from a list of welfare clients and one using RDD, overlap to a large degree. The discussion below reviews both the common issues, as well as the unique aspects related to each type of survey. In the next section, we discuss methods to increase response rates on telephone surveys, placing somewhat more emphasis on issues related to conducting surveys from lists of welfare clients. We chose this emphasis because this is the predominant method being used by states using telephone surveys to evaluate welfare reform. The third section reviews a number of welfare studies that have been recently implemented. In this section we discuss how the methods that are being used match up with the "best practices" discussed in section 2 and how this may relate to response rates. The fourth section provides an overview of issues that are unique to RDD surveys when conducting a survey of low income populations. The final section summarizes and discusses the implications of current practices for significantly increasing response rates.

In this section we discuss the methods needed to obtain high response rates on a telephone survey. These methods include locating; contacting; and obtaining the cooperation of survey subjects. The review applies to all types of telephone surveys, but we have highlighted those methods that seem particularly important for conducting surveys from lists of welfare clients. Section 4 provides specific issues that should be considered in an RDD context.

Locating survey subjects begins with having sufficient information about the subject so that, if the subject moves, s/he can be located and an interview attempted. Low income households move at higher rates than the general population, and it seems reasonable to assume that within this group, "welfare leavers" will be the most mobile. Therefore, if surveys are going to become a routine part of the evaluation process, agencies should consider future evaluation needs in all of their procedures. This includes changing intake procedures to obtain additional information to help locate subjects in the future and designing systems to allow access to other state, administrative records. In the sections that follow these pre-survey procedures are discussed in more detail. This is followed by a description of the initial mail contact, which provides the first indication of whether a subject has moved. The section ends with a discussion of some tracing procedures that might be implemented if the subject is lost to the study.

As part of the intake process, or shortly thereafter (but perhaps separately from the eligibility process) detailed contact information should be obtained for at least two other people who are likely to know the subject’s whereabouts and who do not live in the same household as the subject. In addition to name, address, and telephone number, the relationship of the contact to the subject should be determined along with their place of employment. We believe this step is crucial if acceptable response rates are to be obtained. It may also be difficult to achieve because, in some situations, it may require a change in the intake system.

It is also useful to consider obtaining the subject’s informed consent to access data bases that require consent at the same time as contact information is obtained. It is hard to state when and how consent might be used given the differences in state laws, but we assume that, at a minimum, state income tax records fall into this category (if they are assessable at all, even with consent). This is a common procedure on studies that track and interview drug users and criminal populations (Anglin, et al., 1996).

In addition to the subject’s name, address, telephone number, social security number, and all contact information, consideration should be given to running the subject through other state administrative data bases (Medicaid, Food Stamps, etc.) prior to the survey. This may be particularly useful if the information on the sample file from which the sample is drawn is old or if the information on the files are different. Initial contacts should always start with the most recent address and telephone number. The older information is useful if it proves necessary to trace a subject. The advantage is to avoid unnecessary calls and tracing in as many situations as possible.

If the field period extends for a long period of time it might be necessary to update this information for some subjects during the course of the survey.

Sending letters to pre-notify the subject is accepted practice when conducting studies (Dillman, 1978). It serves the dual purpose of preparing the subject for the telephone interview and identifying those subjects whose address is no longer valid. It is always iterative in a survey of this type. That is, each time a new address is located for a subject (through tracing as discussed later) an advance letter is sent prior to telephone contact.

If an envelope is stamped "return service requested", for a small fee, the Postal Service will not forward the letter, but will instead affix the addressees new address to the envelope and return it to the sender. This only works if (1) the subject has left a forwarding address and if (2) the file is still active, which is usually just six months. If the letter is returned marked "undeliverable", "unknown", "insufficient address", etc. additional tracing steps must be initiated.

Since the Post Office updating procedure is only active for six months it is important to continue mail contacts with survey subjects if they are to be interviewed at different points in time. These mail contacts can be simple and include such thoughtful touches as a birthday card or perhaps a newsletter with interesting survey results.

Mailings should include multiple ways for the subject to contact the survey organization, e.g., an 800 number and a business reply post card with space to update name, address, and telephone numbers. Some small percentage will call and/or return the post card negating the need for further tracing.

One of the problems with first class letters is that the letters often do not reach the subject. The address may be out of date and not delivered to the correct household (e.g., Traugott, et al., 1997), the letter may be thrown out before anyone actually looks at it, or the subject may not have looked at the letter at all. To address these problems consideration should be given to using express delivery rather than first class mail. This initial idea comes from Dillman’s (1978) use of this method in mail surveys. It is based on the logic that express delivery will increase the likelihood that the package will be opened by potential respondents and the contents may be perceived to be of more importance. There is also more assurance with express delivery that the letter has actually reached the household and the subject, particularly if a signature is required. However, requiring a signature may not produce the desired result if it becomes burdensome for the subject, e.g., not being home during the initial delivery and having to make special arrangements to pick-it up. The annoyance may be lessened if, in addition to the letter, an incentive is enclosed.

Since express delivery is costly (but less than in-person contacts) it should be saved for those pre-notification situations in which other means of contact have not been fruitful. For example, if first class letters appear to be delivered, yet telephone contact has not been established, and tracing seems to indicate the address is correct, an express letter might be sent. It might also be used if the telephone number is unlisted or if the subject does not have a telephone. In these situations an express letter with a prepaid incentive might induce the subject to call an 800 number to complete the interview by telephone.

Tracing is costly and some tracing methods are more costly than others. As a general rule it is best to use the least costly methods first when the number of missing subjects is greatest, saving the costlier methods for later when fewer subjects are missing. Data base searches are generally the least costly at a few pennies a "hit" while in-person tracing can cost hundreds of dollars a "hit".

A comprehensive tracing plan that summarizes the steps to be taken in advance and a case management system to track progress are two key components to a successful locating operation. The case management system should maintain the date and result of each contact or attempt to contact each subject (and each lead). The system should provide reports by subject and by tracing source. The subject reports provide "tracers" with a history and allows the tracer to look for leads in the steps that had been taken to date. At the source level, the reports provide cost and "hit" data to help manage the data collection effort. In the end it helps determine those methods that were the most/least cost effective for searching for the population of interest, this data should be used for planning future surveys.

In the following paragraphs each of the tracing sources is briefly discussed. Names are not used, just a description of the type data available from each.

Directory Assistance (DA). There are now several DA services available by dialing a series of numbers. Even 411 is often contracted to outside vendors by some of the telephone companies. Accuracy of information can be "hit" or "miss". It is useful and quick however, in those situations in which just one or two numbers are needed. If the first DA attempt is not successful it may be appropriate to try again a few minutes latter (with a different operator) or try a different service. These calls are not free and the rates vary widely. Costs also include the labor charges of the interviewer/tracer making the calls.

Telephone Look-up Data Bases. There are several large telephone look-up services that maintain telephone directory and information from "other" sources in a data base. These data are available by name, by address, or by telephone number. The criteria for searching is a parameter determined by the submitter, e.g., match on full name and address, match on last name and address, match on address only, match on last name in zip code, etc. Early in the tracing process the criteria should be stricter with matches on address only and/or address with the last name being preferred, while latter in the process broader definitions may be incorporated. Charges for data base "look-ups" are generally based upon the number of matches not submissions, and the cost is usually a few cents. These look-ups are quick, generally requiring less than 48 hours with many claiming 24 hour turnaround. However, the match rate is likely to be low. In a general population survey the match rate might be as high as 60 percent, and of those, some proportion (up to half) will not be accurate. For a highly mobile, low income population, where only those whose numbers are known to have changed are submitted, the "hit" rate is likely to be much less. However, given the cost even a very low rate makes this type of source attractive.

There are several companies that provide this information and one might succeed where another might fail. There may also be regional differences with data in one area more complete than in others. In California for example, telephone numbers are often listed with a name and city, but no address. This limits the usefulness especially for persons with common last names.

Specialized Data Bases. These include credit bureaus and DMV checks where permitted. Checks with one or more of the credit bureaus require the subject’s social security number and are more costly than other data base searches. Charges are based upon each request not the outcome of the request. More up to date information will be returned if the subject has applied for credit recently, which is less likely with a low income population than the general population. DMV checks in many states, such as California, require advance planning to obtain the necessary clearances to search the records.

Other Data Bases. This is a controversial area as it relates to privacy but there are proprietary data bases available on the internet and elsewhere with detailed information on large numbers of people. Access to the data bases is often restricted but restrictions are often negotiable for limited searches for legitimate research purposes. Like credit bureaus, these files are often compiled for marketing purposes and low income populations may not make the purchases necessary to create a record on these files. Records on people are often initiated by such simple acts as ordering a pizza or a taxi.

Telephone Tracers. For the purpose of this discussion it is assumed that each telephone number and address that has been obtained for the subject has led to a dead end. This includes all the original contact information and the results from all the data base searches. It is at this point that tracing becomes expensive. Tracers receive special training on how to mine the files for leads. We have also found that people who have done similar work in the past, such as "skip tracing" for collection agencies tend to be adept at this task. Tracers need investigative instincts, curiosity, and bullheadedness that not all interviewers possess. Tracers are usually paid more than regular interviewers.

The tracers’ task is to review the subject’s tracing record looking for leads and to begin making telephone calls in an attempt to locate the subject. For example, on-line criss-cross directories and mapping programs might be used to locate and contact former neighbors; if children were in the household, neighborhood schools might be called; and employers, if known, might be contacted; etc. All contact of course must be carried out discreetly. Some of these techniques are more productive in areas where community members have some familiarity with one and other, generally places other than the inner cities of New York, Chicago, and LA. Nonetheless, even in urban areas, these techniques sometimes work.

Cost control is crucial in this process since much of the work is limited only by the imagination of the tracer (and tracers sometimes follow the wrong trail). Perhaps a time limit of 15 or 20 minutes might be imposed at which point the tracers work is reviewed by a supervisor to determine if further effort seems fruitful, if another approach might be tried, or the case seems to have hit a dead end.

In-Person Tracing. This is the most expensive method of tracing and is most cost effective if it is carried out in conjunction with interviewing. Like telephone tracing, in-person tracing requires special skills that an interviewer may not possess and vice versa. For this reason it might be prudent to equip tracers with cellular telephones so that the subject, when located, can be interviewed by telephone interviewers. The tracer can thus concentrate on tracing.

Tracing in the field is similar to telephone tracing except the tracer actually visits the former residence(s) of the subject and interviews neighbors, neighborhood businesses, etc. Cost control is more of a problem since supervisory review and consultation is more difficult, but just as important.

When a telephone number is available for either a subject (or a lead) the process of establishing contact becomes important. An ill-defined calling protocol can lead to significant non-response. In this section we discuss some of the issues related to the contact procedures.

Telephone calls need to be spread over different days of the week and different times of the day in order to establish contact with the household (not necessarily the subject). If contact with the household is established, it is possible to learn if the subject can be contacted through the telephone number, and if so, the best time to attempt to call. If the subject is no longer at the number, questions can be asked to determine if anyone in the household knows the subject’s location.

If the telephone is not answered on repeated attempts an assessment must be made of the utility of further attempts against the possibility that the number is no longer appropriate for the subject. In other words, how many times should a non answered telephone be dialed before checking to make sure it is the correct number for the respondent. It is important to remember that this is an iterative process applicable to the initial number on the subject’s record as well as to each number discovered through tracing, some of which will be "better" than others. It is an issue of assessing the trade-off between time and cost.

Many survey firms suggest that seven calls be made over a period of two weeks on different days (2), evenings (3), and weekends (2) before checking with Directory Assistance (DA), calling one or more of the contacts, or searching one of the data bases to determine if the number being dialed is correct. Other firms suggest doubling the number, theorizing that the cost of the additional calls is less than the cost of the searches. Unfortunately, there is no definitive answer, much depends upon the original source of the number being dialed, the time of the year, the age of the number, etc. Very "old" numbers are less likely to be good and perhaps fewer calls should be made before moving to a tracing mode (perhaps 7). If contact information is available, checking with the contact is perhaps cost effective earlier in the process (perhaps 5). In the summertime or around the Holidays more calls might be prudent (perhaps 10 to 12).

It should be obvious to the reader that call histories, by telephone number, for the subject (and lead) be thoroughly documented. This includes the date, time, outcome, as well as any comments that might prove useful as a lead should tracing be necessary.

It is estimated that message machines are now present in between 60 percent to 70 percent of the households in the US (Stores, 1995; Baumgartner, et al., 1997). As more households obtain machines, there has been a growing concern that subjects will use it to screen calls and thereby make them more difficult to contact. The empirical evidence to date has not shown message machines to be a major impediment to contacting respondents. Oldendick and Link (1993) estimate that a maximum of 2 percent to 3 percent of respondents may be using the machine in this way.

A related issue has been the proper procedure to use when an interviewer reaches an answering machine. Should a message be left? If so, when should it be left? Survey organizations differ on how they handle this situation. Some organizations leave a message only after repeated contacts fail to get a respondent on the phone (as reported by one of the experts interviewed). Other organizations leave a message at the first contact and do not leave one thereafter. The latter procedure has been found to be effective in RDD studies relative to not leaving any message at all (Tuckel, et al., 1997; Xu, et al., 1993) The authors favor leaving messages more often (perhaps with every other call with a maximum of 4 or 5) than either of these approaches. We believe, but can not empirically substantiate, that if the goal is to locate and interview a particular person, then the number of messages left might signal the importance of the call to the person hearing the message and might induce them to call the 800 number. Even if the caller says the subject does not live there is useful information. However, leaving too many messages may have a negative effect. This is an area requiring more research.

In this section we highlight some of the standard survey procedures for obtaining a high cooperation rate once contact with the subject has been established. These can be divided into issues of interviewer training, the questionnaire, and the treatment of refusals.

Interviewer experience has been found to be related to obtaining high respondent cooperation (Groves and Fultz, 1985; Dillman, et al., 1976). The theory is that experience gets interviewers familiar with many questions reluctant respondents may have about cooperating (Collins et al., 1988), and allows them to respond in a quick and confident manner. Showing any type of hesitation or lack of confidence is correlated with high refusal rates.

This finding suggests that intense training of interviewers on how to handle reluctant respondents may provide them with increased confidence, as well as the necessary skills, to handle difficult situations. Groves et al, (1997) present results from an experiment on an establishment survey that shows significant improvement in cooperation rates once interviewers are provided with a detailed training on how to handle reluctant respondents. This training consisted of drilling interviewers, through a series of role plays, on providing quick responses to respondent concerns about participating on the study. Since this study was done in an establishment survey, the applicability to a survey of low income respondents is not clear. Respondents to establishment surveys are more willing to converse with the interviewer, which allows for more time to present arguments on why the respondent should participate in the study.

Nevertheless, this suggests that interviewers must have the skills to answer the subject’s questions, to overcome objections, and to establish the necessary rapport to conduct the interview. Training in these areas is crucial if refusals are to be avoided. Answers to Frequently Asked Questions (FAQs) must be prepared and practiced so that the "answers" sound like the interviewer’s own words rather than a script that is being read. Interviewers must also be trained to know when to accept a refusal, leaving the "door open" for future conversion by a different interviewer who might have more success. This type of training is more difficult than training that is centered on the content of the questions, but it is also vital if refusals are to be avoided.

Several areas related to the design of the questionnaire could have an impact on response rates. These include: (1) the length of the questionnaire, (2) the introduction used, and (3) the type/placement of the questions. Each of these has been hypothesized to affect the ability of the interviewer to obtain a high response rate. Interestingly, for each of these characteristics, there is some belief that the effects are primarily on the interviewer’s perception of the task, rather than concerns the respondent may have with the procedure. If interviewers perceive the task to be particularly difficult to complete, their confidence levels may go down and their performance might be affected.

Pretests of the questionnaire should be conducted as part of any research design. Pretests, and accompanying debriefings of the interviewers often uncover problems that are easily corrected prior to interviewing the sample subjects. More elaborate pretesting methods should also be considered. These include, for example, "cognitive interviews", as well as review of the questionnaire by a survey research professional who has experience in conducting structured interviews.

Questionnaire Length. While it is commonly believed that the length of the questionnaire is related to response rates, there is very little empirical evidence that this, in fact, is true. Much of the evidence that does show a relationship between length and response rates are with mail surveys, where respondents get visual cues on how long the interview may be (Bogen, 1996). The length of a telephone interview may not be mentioned unless the respondent asks, so the respondent may not know how long it will take. This fact further confuses the relationship between interview length and response rates.

Two exceptions to this are studies by Collins, et al., (1988) and Sobal (1982). Both found a relationship between how long the interviewer told the respondent the interview would take and the response rate. Collins et al., (1988) found a modest effect of approximately 2 percent, while Sobal (1982) found a much larger reduction of 16 percent when comparing a 5 minute interview to a 20 minute interview. These studies, however, are difficult to generalize to other studies because they do not compare the effects of different descriptions of the length of the interview to one which does not state the length at all. This makes it unclear what the overall effect of interview length might be in the context of another survey, which does not state the length of the interview (unless asked).

This research does suggest, however, that significantly shortening the interview to 5 minutes may increase response rates to some degree. If the interview were shortened to this length, then it might be advantageous to state the length of the interview in the introduction to the survey. One would assume that cutting the interview to just 5 minutes is not an efficient way to increase the response rate. The loss of information needed for analyses will be much larger than anticipated gains in the response rate. For this reason, it might be useful to consider shortening the interview only for a special study of refusers. If shortening the interview significantly increases the number of persons who are converted after an initial refusal, more information might be obtained on how respondents differ from nonrespondents.

Survey Introduction. A natural place to start redesigning the questionnaire to improve response rates is the introduction. Many respondents refuse at this point in the interview. This is especially the case for an RDD survey, where interviewers do not have the name of the respondent and the respondent does not recognize the voice on the other end of the call. For this reason, it is important to mention anything that is seen as an advantage to getting the respondent to stay on the line. This is generally believed to be: (1) the sponsor of the study, (2) the organization conducting the interviews, (3) the topic of the survey and (4) why the study is important

Research in an RDD survey context has not found any general design parameters for the introduction that are particularly effective in increasing response rates. Dillman, et al., (1976), for example, find no effects from offering respondents results from the survey or statements about the social utility of the survey. Similarly, Groves, et al.,, (1976) find variations in the introduction do not change response rates. The exceptions to this are a few selected findings that: (1) government sponsorship seems to increase response rates (Goyder, 1987), (2) university sponsorship may be better than private sponsorship (Everett and Everett, 1990), and (3) making a "nonsolicitation statement" (e.g., "I am not asking for money") can help if the survey is not sponsored by the government.

The most widely agreed upon rule about introductions is that it needs to be as short as possible. Evidence that shorter is better is found in Dillman, et al., (1976), as well as our own experience. Since the interviewer may not have the full attention of the respondent at the initial outset of the call, it is better to simply state the best points of the survey and get the respondent to react to the first question. Interviewers also generally prefer short introductions, since it provides them with a greater opportunity to get the respondent involved in the conversation (less opportunity to hang up). By increasing interviewer confidence, the response rate should be positively affected. It is important to balance the informational requirements with the need to be brief and simple. Long explanations, going into a lot of detail about the survey, may turn respondents off more than motivate them to participate. It is best to provide the respondent with a very broad set of statements to get their attention at this point of the interview. Once rapport and trust have built up a bit, more details about the study can be presented.

Type/Placement of Questions. Sensitive questions have higher rates of non response and should be placed later in the questionnaire but still positioned logically so that the flow from one topic to the next is smooth. Sensitive information include such topics as income, detailed household composition (e.g., naming everyone in the household), participation in social programs and child care. Careful placement allows these questions to be asked after rapport has been established. This is especially true with initial contacts into the household. Asking sensitive questions within the first few minutes of the initial contact may turn respondents off unnecessarily.

If a respondent refuses to participate, it is important for the interviewer to indicate the level of hostility, if any. It may not be desirable (nor cost effective) to try and convert subjects who are extremely hostile (e.g., one in which the respondent is abusive). Other subjects might be recontacted in an attempt to have them reconsider their decision. This recontact should take place several days (7 to 21) after the initial contact to allow the respondent time to reconsider.

Prior to refusal conversion, a letter should be sent to try and convince the respondent to participate. This letter has been shown to be particularly effective if (1) an incentive is enclosed and (2) if express delivery is used for mailing (Cantor, et. al., 1999). Comparisons between the use of express delivery to a first class refusal conversion letter show a difference of ten percentage points in conversion rates on an RDD study and a difference of 15% - 20% if an incentive is enclosed. These results are not likely to be as dramatic for a survey of welfare leavers. However, this strategy has been applied in this context and is believed to be effective.

Based on work related to personal interviews (Couper, et al., 1992), it is possible to create specialized letters for refusal conversion based on what the respondent said at the time of the refusal. A procedure adopted for the National Survey of America’s Families (NSAF)was to have the interviewer provide a recommendation on the type of letter that would be sent to the respondent after the refusal occurred. Since most refusals fall into 2-3 categories (e.g., "no time", "not interested"), special letters could be developed that emphasized particular arguments why the respondent should cooperate (e.g., "no time"¾emphasize the length of the interview; can do the interview in several calls). The problem with this procedure is that for most refusals, there is very little information upon which the interviewer can base a good decision on the reason for refusal. A large number of respondents hang up before providing detailed feedback to the interviewer. As a result, a large majority of the mailouts for the refusal conversion are done using the "general" letter, which does not emphasize anything in particular.

However, in a survey of welfare leavers, where the interviewer may have more information about the reason for refusal, tailoring the letters to the respondent's concerns may be useful. This would depend on the amount of information the interviewer is able to collect on the reason for the nonresponse.

Refusal conversion calls are best handled by a select group of handpicked interviewers who are trained to carry out this type of work. They must be trained to analyze the reason for the refusal and be able to prepare answers for different situations.

In this section we discuss how these "best practices" have been applied in a number of surveys that have been conducted to evaluate welfare reform in different states. The purpose of this review is to provide a picture of the range of practices that have been used and how these practices relate to results.

To better understand the methods that have been implemented, we collected information on a small sample of state-surveys. The largest portion of our sample of studies are from the most recently funded group of ASPE grantees (9 of the 13 studies). The remaining studies were chosen by networking or referral by colleagues. Information was collected through interviews with the director of the research team and any reports that were available. These studies are meant to represent what the current practice is for welfare-leaver studies that are currently in the field and that resemble the conditions under which they are being implemented.

A summary of key characteristics for these 13 surveys is shown in Table 1. In 12 of the 13 surveys for which we collected information, a mixed-mode, two step, approach was used. First, as many telephone interviews were conducted as possible using information accessible to home office staff. Respondents were initially contacted using information available from the administrative records from the sample frame. Advance letters were sent out. For those persons for whom a phone number was not available, the letter asked for the subject to call into an 800 number to do the interview or set up an appointment.

If the telephone number did not lead to the subject, tracing was done from the home office. This typically included using directory assistance, reverse directories to find other addresses and free services on the Internet. Other methods implemented by most of the studies included:

Searches of credit-data bases-- This includes data-bases such as Transunion, CBI/Equifax and TRW. Stapulonis, Kovac and Fraker (1999) report the use of an unnamed data-base that seemed to add information above and beyond these, and
Searches of other data-bases across agencies -- These included food stamps, unemployment insurance, child support enforcement, motor vehicles, Medicaid, employment training, social security, vital records and state-id cards.

The ability to search the "other data-bases" was possible because in all cases the research organizations had the social security number of the respondent.

In discussions with different organizations, we got a clear sense that the original contact information was not of high quality. One study reported, for example, that 78% of the original phone numbers did not lead directly to subjects. This may be, in part, because there is very little need for agency representatives to maintain contact with the recipients over the telephone. In one state, for example, recipients are paid using a debit card that is continually re-valued at the beginning of a payment period. Thus, the address and telephone number information is not used on a frequent basis. In a study conducted by Westat several years ago, a similar result was found when trying to locate convicted felons (Cantor, 1995). Contact information provided by probation officers was found to be accurate about 50% of the time.

If the home office has not been able to find the subject, the case is sent out into the field. In some instances, the field interviewer is expected to both locate and interview the subject. In other instances the interviewer asked the subject to call into a central interviewing facility. If the subject did not have a telephone, the interviewer provided them with a cellular telephone to call into the facility. Several organizations reported that having the respondent call into the central office allowed for more specialization in the field tracing task. If field personnel were not required to administer the interview, then recruitment of personnel could concentrate more specifically on tracking skills.

These interviews provide some data on the possibilities and limitations related to conducting welfare-leaver studies. Many of these studies are implementing procedures that survey researchers advocate when trying to reduce nonresponse. This includes, for example, advance letters, incentives, tracking/tracing and refusal conversion. The resulting response rates ranged from a low of 30% to a high of 80%. Many of the studies are in the 40% - 50% range.

It is clear from these data, as well as from the authors' collective experience, that no single design feature guarantees a high response rate. The effectiveness of particular methods varies by situation and a number of methods are needed to maximize response rates. A useful illustration of this is a survey that was completed in Iowa of current and former TANF recipients (Stapulonis, Kovac and Fraker, 1999). This was a mixed mode survey that implemented literally all of the methods discussed above, including: 1) repeated mailings to respondents, 2) use of telephone interviewers experienced in tracking respondents over the phone, 3) incentive payments, 4) specialized data-base searches and 5) use of field staff to trace and interview respondents. As reported by Stapulonis, Kovac and Fraker (1999), no single method produced a high response. A response rate of around 25% - 30% was achieved through the use of the telephone. At the end of 16 weeks a 48% response rate was achieved by offering an incentive of $10 and sending cases into the field. The remainder of the 60 week field period was used to increase the rate to 72%. During this interim period, numerous methods were instituted, such as increasing the incentive payments, re-mailings (using express mail) to households, field tracing and use of more specialized tracing sources/methods. The latter included using highly experienced trackers in the telephone center and the field.

The data in Table 1 seems to indicate that the mixed mode approach, at least as currently implemented by most of these states, is necessary to achieve response rates of at least 50%. It also indicates that for many studies, use of only the telephone yields a response rate of approximately 30% - 40%. The clearest example of this is study numbers 1 and 2. These two studies were completed in the same state by the same organizations. In study #1, where a 30% response rate was obtained, only telephone and limited tracking was done from a central office. Study #2 instituted a number of additional tracing steps, but also added a field component. Similarly, Study #7 reported a 30% response rate before going into the field and study #8 reported a 40% response rate before releasing cases to the field. The major exceptions to these patterns are the few studies that report final response rates that are at least 70%. In these instances, the response rate obtained over the telephone is at least 50% and, in one case, 66%. To the extent that response rates of at least 50% are desired, it would seem important to use both telephone and field personnel to trace and locate respondents. Study #14 had a response rate of 72% and reported very poor experiences with their field tracers. Effectively, therefore, they completed most of their cases using the telephone as the primary method of contact.

This pattern is consistent with our general experience with working with low income populations. While it is possible, using proper procedures and preparation, to complete a significant number of interviews via mail and telephone, a proportion of this group simply does not respond to anything but in-person contacts. This may be related to this group’s mobility rate, the intermittent availability of the telephone or simply very busy work schedules. Whatever is the case, it is unlikely that achieving extremely high response rates (e.g., 70% or above) for welfare leavers can be achieved by simply the use of mail or telephone interviews.

As one might expect, the primary source of nonresponse in these studies is non-contact, rather than refusals. For example, of the 25% non-response in the study #6, approximately 17% is from not being able to locate respondents and 8% is from refusals. For surveys that have lower response rates (e.g., around 50%), the percent of non-locatables is even higher. This suggests that improving response rates has to do most with improving tracking.

Given this, an important component to pushing response rates above 50% is to improve the ability to find subjects. This relates to both the type of staff that are doing the locating, as well as the type of information that is available for finding the subjects. Study #6, which has a 78% response rate, is a good illustration of the importance of experienced staff. This study did not implement many of the procedures discussed in section 2 above, including pre-notification letters, refusal conversion or incentives. The staff doing the interviewing and tracing, however, were program quality assurance personnel. Since part of their job is to find and interview welfare recipients to conduct audits, they were highly experienced in finding this population. In addition, there seemed to be very strong oversight of the progress of the interviewers by the supervisor. Similarly, study #14 completed all interviews over the telephone and achieved a 72% response rate. They did not offer a monetary incentive and did not conduct refusal conversion. The success of this survey was attributed to the interviewers, who were also part of the quality assurance program.

Alternatively, a number of survey directors reported that the major barriers they encountered were related to inexperienced staff, either in the phone center or in the field, in tracking and tracing subjects. Stapulonis, Kovac and Fraker (1999) report failure of a field effort because of inexperienced field trackers, as did survey directors for study #8. In the latter case, the telephone interviewers were asked to conduct field interviewing.

Our experience has been very similar to this profile and it applies to in-person interviewing as well as tracking from a central telephone facility. The ability to look over case records, find leads and follow-up on those leads requires the ingenuity of a detective, as well as a personality that gains trust when calling neighbors or other community members.

Having solid information from which to trace subjects is also essential to eventually finding them. As noted above, most survey directors commented on the poor quality of the information that was contained in the original sample records. In many cases, the information is quite old (e.g., up to 6 - 9 months) and, in many cases, of questionable accuracy. Since this is a highly mobile population, the age of the records limits the utility of the information quickly. Study #2 attempted to minimize this problem by beginning the tracking process as soon as subjects are off the welfare records. While this may lead to tracking too many people[<a name="endback3">][<a href="#end3">]3[</a>], it provided a way to maintain contact with subjects until the field interviewing started 6 months later. The success of this process is yet to be evaluated, but this method may provide a way to keep the information about sampled persons as up to date as possible.

All of the studies have social security numbers and access to data-bases in other parts of the government (e.g., motor vehicle registrations, food stamps, child support enforcement, Medicaid, unemployment insurance). These provide a powerful set of tools to finding respondents. However, only two of the studies have tracing contact information, containing the names and phone numbers of at least one person, preferably someone who the subject does not live with, who is likely to know where the person is at any point in the future. These two studies both achieved response rates above 75%. Both of these studies were experiments, set up in advance to sample clients at intake and collect this information at that time.

The availability of tracing contacts would not only improve the tracking rates for these studies, but it would reduce the amount of time devoted to tracing. It has been our experience, in fact, that with good tracing contacts, as well as occasional interim contacts with subjects (e.g., every 6 months), very little in-person tracking has to be done. Respondents can be located by interviewers making update phone calls. This is, in fact, what many longitudinal surveys do as part of their routine activities for staying in touch with respondents. As a point of illustration, Westat recently completed a feasibility study that located 85% of subjects three years after their last contact with the study. These subjects were high risk youth who had been diverted into a family counseling program in 1993 and were last contacted in 1996. At that time, tracing contact information had been collected. This population lived in highly urbanized, poor, neighborhoods and could be considered comparable to those being traced in the welfare-leaver studies discussed above. Approximately 67% of the population was found through the use of mail and telephone contacts. The remaining 18% were found by field tracing.

Pushing response rates higher can also be done through selective adoption of other methods related to making the response task easier. Some percentage of the persons that are classified as non-locatable are really tacit refusers. That is, some of those individuals that "can’t be located" are explicitly avoiding contact with the interviewer or field tracer because of reluctance to participate in the survey. This reluctance may be due to not wanting to take the time to do the survey or it may be more deep-seated and related to a general fear of being found by anyone who happens to be looking for them.

Several of the studies found that repeated mailings to the same addresses over time did result in completed interviews. This seemed to be especially effective when these mailings were tied to increased incentives. This would tend to support the idea that at least some portion of the "non contacts" are actually persons that are tacitly refusing to do the interview, at least the first few times around. Express mail was also used for selected follow-up mailings, although it is unclear whether this method of delivery was particularly effective.

As noted in Table 1, a number of the studies do not implement any type of refusal conversion. The reluctance stems from fear that this would be viewed as coercive, since the agency conducting the research is the same agency that is responsible for providing benefits on a number of support programs. Other survey groups, however, reported confidence in conducting refusal conversion activities, as long as they were convinced the interviewers were well trained and understood the line between trying to directly address respondents concerns and coercion. In fact, many initial refusals are highly situational. For example, it may be that the interviewer calls at a time when the kids happen to be giving the parent an especially difficult time or at a time when the subject just came home from an exhausting day at work. It may also be that the respondent simply did not understand the nature of the survey request. In all of these cases, calling back at another time, with an elaborated explanation of the survey, could prove useful. One study director reported, in fact, that about 50% of the initial refusers in the study were eventually converted to final completed interviews. This is not out of line with refusal conversion rates found on other studies, either of the general or low income populations.

In many ways, RDD surveys pose a much different set of challenges than those for list-based samples, especially issues related to nonresponse. For surveys of welfare clients, the target population is identified very clearly and quality issues have to do with finding sample members to conduct the interview. For RDD surveys, the primary issues have to do with efficiently identifying low income subjects and, once identified, convincing them to participate on a survey.

To provide some perspective on the level of response achieved on RDD surveys, Massey, et al (1998) presented results of a study that reviewed the response rates of a large number of RDD surveys conducted for government agencies or as part of a large survey effort. Their results found a median response rate of between 60% - 64%, with about 20% of the surveys exceeding 70%. The overall perception among survey analysts is that the trend is for this rate to be decreasing over time. That is, it is increasingly getting harder to achieve high response rates for RDD surveys.

An RDD survey of low income populations faces several hurdles relative to achieving a high response rate. The first is the need to screen all households on the basis of income. This leads to two types of problems. The first is the fact that it adds an additional opportunity for someone to refuse to do the survey. A screener written to find low income households has to include a number of questions that respondents are sensitive to, including information on who lives in the households, as well as some type of income measure. Much of the non-response on RDD surveys occur at this point in the process. For example, on the National Survey of America's Families (NSAF), a national RDD survey that oversamples low income groups, the screener response rate was in the high 70's. Once a respondent within the household was selected, the response rate to do the extended interview was in the 80's. Nonetheless, the combination of the two rates, which form the final response rate, results in a rate in the mid-60's (Brick, et al, 1999).

Low response rates on RDD surveys are partly an issue of credibility. Relative to a survey of welfare leavers, the issue of credibility places more emphasis on design features that motivate respondents to participate on the survey (vis-à-vis trying to locate respondents). For example, research on methods to increase RDD response rates have found that pre-notification prior to the call, methods of delivery of pre-note letters and use of incentives can provide important boosts above those normally achieved when implementing many of the other important design features reviewed in section 2 above. All three of these increase response rates in the context of an RDD survey (Camburn, et al., 1995; Brick, et al., 1997; Cantor, et al, 1999).

In addition, refusal conversion is particularly important for an RDD survey, since such a large proportion of the nonresponse is due to refusals.. Refusal to the screener could be from almost any member of the households, since most screeners accept responses from any adult in the household. Calling into the household a second time provides an opportunity to get another person in the household (who may be more willing to participate) or get the same respondent who may not be as difficult to convince to participate in a short screening instrument. Refusal to the extended interview may be more difficult to turn around. Refusal conversion strategies at this level are amenable to more traditional "tailoring" methods (e.g., Groves and Couper, 1999), since respondents at this stage of the process may be more willing to listen to the interviewer.

A second issue related to conducting RDD surveys of low income populations is the ability to actually find and oversample this group. Screening for person of low income has been found to have considerable error. This has been assessed when comparing the poverty status reported on the initial screener and the income reported when using more extensive questions on the longer, extended interview. For example, on the NSAF approximately 10% - 15% of those that report being below 200% of poverty on the longer interview initially answer the screener they are above this mark. Alternatively, 20% - 30% of those reporting themselves as above 200% of poverty on the extended interview, initially screen in as above this mark (Cantor and Wang, 1998). Similar patterns have been observed for in-person surveys, although the rates do not seem to be as extreme. This reduces the overall efficiency of the sample design. This, in turn, requires increasing sample sizes to achieve the desired level of precision.

To date, there has not been a clear solution to this problem. In-person surveys have developed more extensive screening interviews to allow predicting income status at the point of the screener (Moeller and Mathiowetz, 1994). This approach might also be taken for RDD screeners, although there is less opportunity to ask the types of questions that are needed to predict income. For example, asking detailed household rosters, collecting information on jobs or material possessions all would likely reduce the screener response rate.

A second issue related to sample design on an RDD survey is the coverage of low income households. While only 6% of the national population is estimated as not have a telephone (Thornberry and Massey, 1987), about 30% of those under poverty are estimated to be in this state. For an RDD survey of a low income populations, therefore, it is important to decide how coverage issues will be approached. One, very expensive, approach would be to introduce an area frame into the design. This would include screening for non-telephone households in-person and then conducting the extended interviews either in-person or over the telephone.

Over the last few years, a new method, based on an imputation method, has been tested that does not require doing in-person interviews (Keeter, 1995).. The premise of the method is based on the idea that for a certain segment of the population, having telephone service is a dynamic, rather than stable, characteristic. Consequently, many of the people that do not have service at one point in time may have service shortly thereafter. This implies that one might be able to use persons that have a telephone, but report interrupted service as proxies for those who do not have telephones at the time the survey is being conducted. Based on this idea, telephone surveys are increasingly including a question that asks respondents if they have had any interruptions in their telephone service over an extended period of time (e.g., last 12 months). If there was an interruption, they are then asked how long did they not have service .This information is then used in the development of the survey weights. Those reporting significant interruptions of service are used as proxies for persons without a telephone.

Recent evaluations of this method as a complete substitute for actually conducting in-person interviews has shown some promise (Flores-Cervantes, et. al., 1999; Frankel, et al., 1999). Initial analysis has shown the use of these questions significantly reduces the bias for key income and other well-being measures when compared to estimates that use in-person interviewing. This is not always the case, however. For certain statistics and certain low income subgroups, the properties of the estimator are unstable. This may be due, in part, to developing better weighting strategies than currently employed. Nonetheless, the use of these questions seem to offer a solution that, given the huge expense involved with doing in-person interviews, may offer significant advantages.

The use of this method may also be of interest to those conducting telephone surveys with persons from a list of welfare clients. Rather than viewed as a way to reduce coverage error, however, they could be used when trying to impute missing data for high nonresponse rates.

This paper has attempted to provide information on methods to achieve high response rates on telephone surveys of low income populations. We have concentrated much of the review on studies that start with a list of welfare recipients, but have also provided information for persons conducting RDD surveys as well. In practice, most of the welfare leaver studies have been mixed modes, using both telephone and in-person interviews to complete the studies. The first section provided a list of "best practices" that should be considered when conducting any telephone survey. The second section provided examples of what is currently being practiced in recently completed welfare-leaver studies and how these practices relate to the types of results that are being achieved. The fourth section provided special issues related to RDD surveys. In this section we draw out some of the implications of these discussions for conducting telephone surveys with low income populations and provide some recommendations on how response rates on studies such as those reviewed in section 3 above might achieve higher response rates.

Clearly one primary theme taken from our review is the need to improve the ability of studies to find subjects. Most studies were faced with very similar situations -- the information used to track respondents was both relatively old (6 -- 9 months) and limited. The age of the information could be partly addressed through methods such as those mentioned above -- start contacting the respondent immediately after they leave the program. Maintain this contact until the time to conduct the interview (e.g., 6 months after leaving). We suspect that the age of the information will be difficult to overcome, however. Much of the information in the records could may have been collected when the subject entered the program and may not have been updated since that time. Short of requiring continual updates of this information while the subject is in the program, therefore, this is a constraint that will be difficult to get around.

A more effective way to improve the information is by collecting tracing contacts when the subjects initially enters the program. This type of information does not go out of date nearly as fast as a single name and address. Even if these go out of date, the names and addresses can provide additional leads that can be followed up by trackers. When collecting this information it is important that the names be of persons that do not live with the subject. This decreases the possibility that if the subject moves, the contact person would have moved as well.

If it is not possible to collect this information at the time the person enters, another source for this information might be any case-history documentation that is available on the subject. Many of the studies reviewed above reported using information from other government data-bases, such as motor vehicles or other recipiency programs, to collect updated addresses and phone numbers. Examination of hardcopy case folders, if they exist, would be one way to supplement this information. One study reported doing this and found it a good source for tracing contact information. Subjects, at some point, could have provided information on references, employers and friends as part of the application process. This information, if confidentiality issues can be addressed, can be examined to find further leads to find and track those people that cannot be found.

The other major point on tracking is to use supervisory and interviewing staff who are experienced at conducting a study like this. Strong supervisors are especially important, since they can teach new interviewers methods to finding particular cases. They can also provide guidance and new ideas for experienced interviewers. The supervision has to be done on an interviewer by interviewer basis. Supervisors should review each case with interviewers on a frequent basis (e.g., every week) and provide feedback/advice on how to proceed with each one. This would include making sure the interviewer is following up with the leads that are in hand, as well as ideas on how to generate more leads.

Effective locating and management of a survey like this cannot be learned on the job. Given this, it is important that sponsoring agencies gather evidence that the personnel involved have the appropriate experience and successful track record to successfully complete the work. This advice applies if using personnel within the sponsoring agency or with a contractor. When considering a contractor, the sponsoring agency should ask for hard evidence that a study like this has been conducted, references checked and success rates evaluated. Questions should be asked about the availability of experienced staff to complete the work. If the work is to be done by telephone, then some information on the track record of telephone tracers should be requested. For in-person contacts, information on the experience of personnel who reside in the local area where the study is to be conducted should be collected.

To provide some perspective on the impact tracing might have on the cost of a survey, we developed estimates of cost under a scenario where tracing contact information is available. Costs for surveys of this type are difficult to estimate because so much is dependent upon the ability of the data collector to monitor the process; the availability of skilled staff to carry-out the tracing; and the nature and quality of information that is available at the start. The first two factors rest with the data collector while the latter is dependent upon information obtained about each subject (and its accessibility) by the agency. If the data are not current or not complete, tracing is very difficult and costly regardless of the controls the data collector has in place.

Under this scenario, we estimate that that approximately 600 fewer hours are required to trace 1000 subjects if tracing contact information is available. Contact information, for this example, would have been obtained during the intake process and been delivered to the data collector with the sample. The table may be somewhat deceptive because, for purposes of illustration, we have forced the two samples to have approximately the same found rate in order to compare the level of effort. In reality, the found rate (and consequently the response rate) for those with contact data would likely be much higher than those without.

In creating Table 1 we assumed the following times for each level of tracing, with each column corresponding to an item below:

While these estimated times are reasonable they can also mislead. For example, if several data bases are used (agency, Credit Bureau, DMV, commercial, etc.) each can produce a "hit" and require follow-up, so it is likely that more than one follow-up call might be carried out for some sample members, and none for others. In organizing "hits", care must be taken to make sure it is genuine and not a duplicate of an earlier "hit" that has already been invalidated. This adds time though the process can be aided by a case management system.

The level of interviewer/tracer effort is only one dimension of cost. Supervisory hours will range between 20 to 40 percent of interviewer hours, depending upon activity, with the highest percentage needed for the intense tracing. Other variable labor costs include all clerical functions related to mailing and maintaining the case management system, and all direct non labor costs. These include, but are not limited to: charges from data base management companies to run files, Directory Assistance charges, telephone line charges, field travel expenses, and postage/express delivery charges.

Fixed costs include the management costs to coordinate the activities and the programming functions to develop a case management system; preparing files for data searches; updating files with results of data searches; and preparing labels for mailing.

First and foremost in any survey operation is the need to develop an infrastructure that maintains control over cases as they move from the initial pre-notification letter, to call scheduling and case documentation. Understanding what stage each case is in and what has been already tried is critical to making sure each case goes through all possibilities. These basics are not particularly expensive to implement and can yield a large payoff in terms of completed interviews. For example, supervisory staff should be reviewing telephone cases as they move to different dispositions, such as "ring, no answer", "initial refusal", "subject not at this number". As with tracing, supervisors should review cases and make case-by-case determinations on the most logical next step.

Some monitoring of the call scheduling should also make sure that different times of the day and different days are used when trying to contact respondents. This is one big advantage of a centralized computerized telephone interview (CATI). The computer "deals" cases at the appropriate times and pretty much ensures that the desired calling algorithms are followed. However, if the study is being done on paper and pencil, then a system to document and monitor call history should be in place to ensure that this occurs.

Pre-notification is being used extensively for the studies reviewed in section 3 above. Low income populations are particularly difficult to reach by mail. For this reason, some attention to the form and content of this correspondence is likely worth a small investment of professional time. This includes, for example, the way the letters are addressed (e.g., labels, computer generated, hand written), the method of delivery (express delivery vs. first class mail) and the clarity of the message. The contents of the letter should be structured to be as clear and as simple as possible. One study reviewed above noted an improvement (although not experimentally tested) when formatting letters with large subheadings and minimal text. The details surrounding the study were relegated to a question and answer sheet. We have also found this to be an improvement over the standard letter-type format. Similarly, use of express delivery, at least when there is some confidence in the validity of the respondent's address, may also be a cost-effective way to provide respondents with information about the survey that would eventually increase their motivation to participate.

Incentives are also being used in the studies mentioned above. We have not elaborated on this much, partly because another paper will be presented on just this topic. One pattern we did notice was that all incentives are "promised" for completion of the interview. These amounts generally ranged from $15 to $50, with the largest amounts being paid at the end of field periods to motivate the most reluctant respondents. Research has found that pre-paid incentives are more effective than promised incentives. Research we have done in an RDD context has shown, in fact, that not only is it more effective, but the amount of money that is needed to convince people to participate is much smaller. It may be worth experimenting with pre-payments that are considerably smaller than the current promised incentives (e.g., $5) to see if there is a significant improvement in the ability to locate and interview respondents.

In conclusion, conducting a telephone survey of low income populations is a task that requires careful preparation and monitoring. The surveys implemented by states to this point have been discovering this, as they attempt to locate and interview respondents. Improving on response rates will require attention to increasing the information used to locate respondents, as well as making it as easy as possible for respondents to participate. This paper has provided a thumbnail sketch on some of the important procedures to consider to achieve this goal. It will be interesting to see how future surveys adapt or innovate on these procedures to overcome the barriers they are currently encountering.

Table 1. Results of Interviews with 14 Selected Telephone Surveys of Welfare or Ex-Welfare Recipients

Study Number	Advance Letter/ Incentives	Telephone Tracking Sources	Field Tracking	Refusal Conversion	Response Rate	Field Period
1	Yes/Yes	Directory Assistance, Reverse Directory	No	No	30%	3 months
2	Yes/Yes	Directory Assistance, Reverse Directory, Specialized tracking firm, Motor Vehicale/ID Records	yes Exp. Staff	Yes	51%	9 months+
3	Yes/Yes	Director Assistance, Credita data-bases, other welfare offices	Yes Exp. staff	yes	Total: >70% Tel: 50%	5 months
4	Yes/Yes	Directory Assistance, Credita data-bases, Other welfare offices	Yes Exp. staff	Yes	Total: >70%	5 months
5	Yes/Yes	DK	Yes Unknown Exp.	DK	52%	DK
6	No/No	Directory Assistance, Other Agency Data-base	Yes Exp. Staff	No	Total: 78% Tel: 66%	4 months
7	Yes/Yes	Directory Assistance, Credit data-bases, Motor vehicle, other agency data-bases	Yes Exp. Staff No Exp.	Yes	Total: 72% Tel: 25% - 30%	60 months
8	Yes/Yes	Reverse Director, Other Agencies	Yes No Exp.	No	Total: 46% Tel: 40%	DK
9	DK	Directory Assistance, Tracing Contact, Credit Data-bases, Other Agencies	Yes Exp. STaff	DK	Total: 80% Tel: 50%	DK
10	Yes/Yes	Directory Assistance, Credit Data-bases, Other Agencies	Yes Exp. Staff	DK	Total: 75%	DK
11	Yes/Yes	Directory Assistance, Credit Data-bases, Other Agencies	Yes Exp. Staff	Yes	Total: 81%	DK
12	Yes/Yes	Directory Assistance, Credit Data-bases, Other Agencies	No	DK	Total: 40%	4 months
13	Yes/Yes	Directory Assistance, Other Agencies	Yes Exp. Staff	No	Total: 72%	DK
14++	Yes/Yes**	Directory Assistance, Other Agencies	Yes No Exp	No	Total: 72% Tel: >65%	2 months

+ Interviewing period was 3 months. Used 6 months before interviewing period to establish contact information and find respondents.

Table 2: Comparison of tracing hours and found rates by contact information.

	No Tracing (a)	Calling Contacts (b)	Data Base Search (c)	Intense Telephone Follow-up (d)	In-person Tracing (e)	Total (f)

Tracing time per sample unit (minutes)	0	20	20	60	420

With contact Information
Sample size	1000	700	490	343	240	1000
Found rate	0.30	0.30	0.30	0.30	0.15	0.80
Number Found	300	210	147	103	36	796

Estimated number of hours	0	233	163	343	1681

Without Contact Information
Sample Size	1000	n/a	700	469	314.23	1000
Found rate	0.30		0.33	0.33	0.33	0.79
Number Found	300		231	154.77	104	789

Estimated number of hours	0		233	469	2200	2902

1. A related concern is whether respondents are using caller ID in a similar way.

2. This rate is surprisingly low, given that probation officers are supposed to be in regular contact with probationers.

3. Most studies had, as an eligibility criteria, that leavers had to stay off the welfare program for at least two months. Sampling within a month of leaving the program, therefore, eventually results in having to drop subjects because they return to the program within two months.

4. Telephone interviews would be conducted by having the respondent call into a central facility using a cellular telephone.

Anglin, D. A., Danila, B., Ryan, T. and K. Mantius (1996) "Staying in Touch: A Fieldwork Manual of Tracking Procedures for Locating Substance Abusers for Follow-Up Studies" National Evaluation Data and Technical Assistance Center, Center for Substance Abuse Treatment, SAMSHA, January, 1966.

Baumgartner, R.M., Tathbun, P.R., Grabener and K. Bailly (1997) "Telephone Answering Machine Messages and Completion Rates." Paper presented at the Annual meeting of the American Association for Public Opinion Research, Norfolk, VA, May 15-17, 1997

Bogen, K. (1996) "The Effect of Questionnaire Length on Response Rates--A Review of the Literature." American Statistical Association 1996 Proceedings of the Section on Survey Research Methods, pp. 1020-1025.

Brick, J.M., Collins, M., and Chandler, K. (1997) "An Experiment in Random Digit Dial Screening." National Household Education Survey. Technical Report.

Brick, J.M., Flores-Cervantes, I. and D. Cantor (1999) "1997 NSAF Response Rates and Methods Evaluation, Report No. 8" NSAF Methodology Reports, Urban Institute.

Camburn, D.P., Lavrakas, P.J., Battaglia, M.P., Massey, J.T., and Wright, R.A. (1995) "Using Advance Respondent Letters in Random-Digit-Dialing Telephone Surveys." American Statistical Association - Proceedings of the Section on Survey Research Methods, pp. 969-974.

Cantor, D. (1995). Prevalence of drug use in the DC metropolitan area juvenile and adult offender populations: 1991. Washington, DC: National Institute on Drug Abuse.

Cantor, D. and K. Wang (1998) "Correlates of Measurement Error when Screening on Poverty Status for a Random Digit Dial Survey" Paper presented at the American Statistical Association Meetings, Dallas, TX, August, 1998.

Cantor, D., Cunningham, P. and P. Giambo (1999) "The use of pre-paid incentives and express delivery on a Random Digit Dial Survey" Paper presented at the International Conference on Survey Nonresponse" Portland, Oregon, October, 1999.

Collins, M., Sykes, W., Wilson, P., and Blackshaw, N. (1987) "Non-Response: The UK Experience." Telephone Survey Methodology, Wiley, chapter 13.

Couper, M., Groves, R.M. and Cialdini, R.B. (1992) "Understanding the Decision to Participate in a Survey." Public Opinion Quarterly vol. 56: 475-495.

Dillman, D. Mail and Telephone Surveys: The Total Design Method. (1978) New York: Wiley.

Dillman, D., Gallegos, J.G., and Frey, J.H. (1976) "Reducing Refusal Rates for Telephone Interviews." Public Opinion Quarterly vol. 40: 66-78.

Flores-Cervantes, I., Brick, J.M., Hankins, T. and K. Wang (1999) "Evaluation of the Use of Data on Interruption in Telephone Service" Paper presented at the American Statistical Association meetings in Baltimore, MD, 1999.

Goyder, J. (1987) The Silent Minority, Nonrespondents on Sample Surveys. Polity Press: Cambridge, UK.

Groves, R. and N. Fultz (1985) "Gender Effects Among Telephone Interviewers in a Survey of Economic Attributes." Sociological Methods and Research, 14: 31-52.

Groves, R., Cantor, D., McGonagle and K., Van Hoewyk (1997) "Research Investigations in Gaining Participation From Sample Firms in the Current Employment Statistics Program." Paper presented at the Annual Meetings of the American Statistical Association, Anaheim, California, August 10-14.

Groves, R.M., Cannell, C., and O’Neil, M. (1979) "Telephone Interview Introductions and Refusal Rates: Experiments in Increasing Respondent Cooperation." American Statistical Association - Proceedings of the Section on Survey Research Methods, pp. 252-255.

Keeter, S. (1995) "Estimating Telephone Noncoverage Bias with A Telephone Survey." Public Opinion Quarterly vol. 59: 196-217.

Massey, J., O’Connor, D., Krotki, K., and Chandler, K. (1998) "An Investigation of Response Rates in Random Digit Dialing (RDD) Telephone Surveys." American Statistical Association 1997 Proceedings of the Section on Survey Research Methods\

Moeller, J.F. and N. A. Mathiowetz (1994). "Problems of Screening for Poverty Status" Journal of Official Statistics, 10: 327-337.

Oldendick, R.W., and Link, M.W. (1994) "The Answering Machine Generation: Who are they and what problem do they pose for Survey Research." Public Opinion Quarterly vol. 58: 264-273.

Sobal, J. (1982) "Disclosing Information in Interview Introductions: Methodological Consequences of Informed Consent,." Sociology and Social Research, 66:349-61.

Stapulonis, R.A., Kovac, M. and T.M. Fraker (1999) "Surveying Current and Former TANF Recipients in Iowa" Paper presented at the 21st Annual Research Conference of the Association for Public Policy Analysis and Management, Washington, D.C.: Mathematica Policy Research, Inc.

Stores (1995) "Answering Machines Hold Off Voice Mail Challenge." STORES vol. 77 no. 11: 38-39.

Thornberry, O.T. and J.T. Massey (1988) "Trends in United States Telephone Coverage Across Time and Subgroups" pp. 25-50 in Groves, R.M., Biemer, P.P., Lyberg, L.E., Massey, J.T., Nicholls, W.L. and Waksberg, J. (eds.), Telephone Survey Methodology. New York: Wiley.

Traugott, M., Lepkowski, J. and P. Weiss (1997) "An Investigation of Methods for Matching RDD Respondents with Contact Information for Validation Studies." paper presented at the Annual meeting of the American Association for Public Opinion Research, Norfolk, VA, May 15-17, 1997.

Tuckel, P.S. and Feinberg, B.M. (1991) "The Answering Machine Poses Many Questions for Telephone Survey Researchers." Public Opinion Quarterly vol. 55: 200-217.

Xu, M., Bates, B., and Schweitzer, J.C. (1993) "The Impact of Messages on Survey Participation in Answering Machine Households." Public Opinion Quarterly vol. 57: 232-237.