Determining Which Questions Are Best: Methodologies for Evaluating Survey Questions

James L. Esposito, P.C. Campanelli, J.M. Rothgeb, and Anne E. Polivka

Abstract

Various techniques have been developed over the years to pretest new survey questions or to evaluate the effectiveness of preexisting questions. As part of the current effort to redesign the Current Population Survey (CPS), researchers from Census and BLS are using a variety of methods to evaluate questions designed to elicit accurate labor force data and to assess the quality of interviewer-respondent interactions. These methods include: (a) the coding of interviewer and respondent behavior, (b) debriefing respondents to obtain additional information about how they view CPS questions/concepts, and (c) debriefing interviewers on how respondents react to and understand CPS questions/concepts, and (d) comparing item nonresponse rates and response distributions for different versions of the same survey questions. This paper describes the advantages and disadvantages of each of these techniques. The discussion is illustrated with selected results from a CATI/RDD test.