The development of questions on disability for the Current Population Survey
After a decade of development, questions aimed at identifying people with disabilities were added to the Current Population Survey.
In June 2008, six questions designed to identify people with disabilities were added to the Current Population Survey (CPS). In February 2009, monthly labor force estimates of people with disabilities were released to the public via a website that was updated each month with the newest data. One year later, a table with data on the labor force status of people with disabilities was added to the monthly Employment Situation news release.1
The release of the first CPS data on people with disabilities was preceded by a 10-year development period that began in the late 1990s. In the early stages of the process, and in accordance with President Clinton’s Executive Order 13078,2 the Bureau of Labor Statistics (BLS), along with several other federal agencies, identified the goal of placing a small set of questions within the CPS to measure the employment status of people with disabilities as a means of tracking their employment progress after the adoption of the Executive order. Extensive research efforts in support of this goal resulted in the identification of a set of seven questions, which were then tested in the February 2006 CPS. Analysis of the February 2006 CPS data revealed that the additional questions did not adversely affect the response rates for the survey, but the questions did function in a different way than was anticipated on the basis of previous testing. As a result of these unanticipated consequences, in lieu of further testing of the same questions, the decision was made to adopt a set of questions that had been developed for use in the American Community Survey (ACS).3
The purpose of this article is to document the process by which BLS and other federal agencies developed questions to identify the disability population in the CPS.
Formation of interagency working group
In early 1998, Executive Order 13078 established the Presidential Task Force on Employment of Adults with Disabilities, an overarching organization that provided an operating base for several committees and workgroups with mandates specified in the order. The Executive order directed BLS to work with several other federal agencies to develop a measure of the employment rate of people with disabilities that was to be published on as frequent a basis as possible. Pursuant to the mandate, the task force established the Employment Rate Measurement Methodology (ERMM) Work Group, to include members from 17 federal agencies. The effort received financial support primarily from the Department of Labor’s Office of Disability Employment Policy (ODEP) and the National Institute on Disability and Rehabilitative Research (NIDRR).
The task of developing questions to identify the disability population was challenging because there is considerable disagreement on how to define disability. In fact, there are numerous definitions of the term that are used throughout the federal government. The definition of disability given in Executive Order 13078 states, “An adult with a disability is a person with a physical or mental impairment that substantially limits at least one major life activity.” This definition views disability as a function of the interaction between an individual with an impairment and his or her environment.
The CPS, a monthly survey of about 60,000 households that is conducted by the Census Bureau for BLS, collects information on labor force status by a variety of demographic characteristics. The CPS was chosen as the vehicle for including the disability questions for two main reasons. First, because the CPS is a monthly survey, it would satisfy the requirement in the Executive order to present the data on disability in a timely fashion. Second, because the CPS is already the official source of employment and unemployment data for various demographic groups, it was a logical decision that data on individuals with disabilities be included within its scope.
The main purpose of the CPS is to ascertain the labor force status of the U.S. population. In addition, the CPS collects a wide range of demographic and labor force information, and answering all of its questions places a time burden on respondents, who are volunteer participants. Thus, any new question would increase the respondent burden and might cause some respondents not to participate in the survey. Moreover, because the same households may be interviewed more than once, future responses also could be affected. Complicating matters further, testing established that the measurement of disability status would require multiple questions, and asking about disabilities might be particularly sensitive for some demographic groups. In light of these difficulties, the ERMM Work Group decided that the questions on disability should be few while still meeting the requirements set forth in the Executive order.
Using the National Comorbidity Survey to test the questions
The ERMM Work Group undertook a summary review of existing disability questions and found that little testing had been done on those questions. The limited evidence available showed that the questions did not work well and yielded results that were difficult to interpret. Given the absence of a consensus on which questions could best identify the disability status of individuals, BLS worked with several academic researchers to identify surveys from which to draw questions. These surveys were selected because they were nationally known, their disability questions had face validity, or the data collected from their respondents were widely reported. The surveys ultimately chosen for disability questions were the Survey of Income and Program Participation, the National Health Interview Survey on Disability, the World Health Organization Disability Assessment Scale, the Census 2000, the National Organization on Disability’s Harris poll, and the Behavioral Risk Factor Surveillance System Quality of Life Module. The investigation resulted in a large number of potential questions, which BLS survey methodologists and subject-matter specialists evaluated to identify the ones that might work well in the CPS and to eliminate those which were unsuitable.
While the process of narrowing the list of candidate questions continued, BLS identified a test vehicle. The National Comorbidity Survey (NCS), a nationally representative survey fielded by the Department of Health Care Policy of Harvard Medical School, is funded primarily by the National Institute of Mental Health and contains extensive questions on mental health. The survey also has many questions on physical well-being and contains the Global Assessment Schedule from the fourth edition of the Diagnostic and Statistical Manual of Mental Disorders. The questions on mental health in particular were considered important because research conducted for the ERMM Work Group showed that it is difficult to identify individuals with mental health difficulties in a survey. These individuals may appear outwardly to have no disability despite the fact that their condition affects many areas of their daily living.
Because the NCS collected such extensive information, it enabled BLS to compare a respondent’s answers to the test questions with his or her answers to the more detailed NCS questions. Using the expanded set of data from the NCS, BLS could identify a respondent’s disability status and then compare the results obtained from the respondent’s answers to a smaller set of questions with the conclusion drawn from his or her answers to the larger NCS set of questions. In this way, the larger NCS set of questions could be used as a standard to evaluate the effectiveness of a small set of questions at correctly identifying a respondent’s disability status.
The candidate questions were identified, modified to conform to the testing requirements, and tested by means of cognitive interviews. The purpose of the cognitive testing was twofold: to determine whether respondents understood the questions as intended and could provide accurate answers based on that understanding; and, more broadly, to discover what information respondents considered in forming their answers.
The NCS was fielded from 2001 through 2002; analysis of the NCS dataset began while the survey was still underway in 2001. The results from the NCS interviews were used to classify respondents according to the likelihood that they had one or more disabilities. In several cases, the respondent’s disability status was difficult to identify, and these respondents were recontacted to gain further information in order to create a more complete profile for each respondent.4 From the profiles that were created, analysts were successfully able to ascertain the disability status of the majority of respondents. Then, statistical analysis was used to identify seven questions that accurately identified most people with disabilities.5
Using the CPS to test the questions
Whenever a change to the CPS is considered, one of the main concerns of the agencies involved is ensuring that there are as few adverse effects as possible on the CPS labor force estimates and response rate. In order to identify what effect the addition of a set of disability questions might have on labor force estimates, the disability question set was tested as a supplement to the February 2006 CPS in a joint effort between BLS, the Census Bureau, ODEP, and NIDRR. The set of seven disability questions was asked directly after the end of the regular monthly CPS questions. There were two primary goals: to compare the CPS disability rate with that obtained from the NCS and to evaluate the effect on CPS response rates in the next month.
The test disability questions appeared in the February 2006 CPS in the following format:
This month we want to learn about the well being of people in this household.
- Does anyone in this household who is 15 years old or over have…A hearing problem that prevents them from hearing what is said in normal conversation even with a hearing aid?
- Does anyone in this household who is 15 years old or over have…A vision problem that prevents them from reading a newspaper even when wearing glasses or contacts?
- Does anyone in this household who is 15 years old or over have…Any condition that substantially limits one or more basic physical activities such as walking, climbing stairs, reaching, lifting, or carrying?
- Does anyone in this household who is 15 years old or over have…Any other physical disability?
- Does anyone in this household who is 15 years old or over have…Any emotional or mental disability?
- Because of a physical, mental, or emotional condition lasting 3 months or longer, does anyone in this household who is 15 years old or over have…Difficulty learning, remembering or concentrating?
- Because of a physical, mental, or emotional condition lasting 3 months or longer, does anyone in this household who is 15 years old or over have…Difficulty participating fully in school, housework, or other daily activities?
The test used a split-panel design that included households in their first through third or fifth through seventh monthly interviews.6 Under this design, half of the households were asked the disability questions while the other half were not asked any additional questions. Such a design ensured that enough households received the disability questions to provide meaningful data. In addition, differences in the February–March response rate between households that were asked the disability questions and those not asked any additional questions could be examined to determine whether asking the disability questions adversely affected the likelihood that households would continue to participate in the CPS.
The test of the disability questions within the February 2006 CPS revealed that there was no adverse effect on the response rates of households that had received the questions. However, the disability rate obtained from the CPS was much lower than that produced by the NCS. Disability researchers have long been aware that there is a sizable context effect on disability questions that are asked in different surveys. However, the inability to explain or quantify this effect caused BLS researchers to require further investigation before accepting the seven disability questions for use in the CPS.7
Adoption of ACS questions
About 5 years after BLS began conducting its research into disability questions, another interagency group was formed with the goal of revising the disability questions contained in the ACS. This survey had the important role of replacing the Census Bureau long form. The ACS is conducted continuously, sampling communities across the United States. In 2005, the survey sampled more than 3 million households, a sample much larger than the CPS annual sample of about 720,000 households. The interagency group identified a set of six questions that were to be placed into the 2008 ACS, and BLS considered the feasibility of adopting these questions into the CPS.8
At that time, the CPS questions had been in development for quite a while and the field testing had raised questions that required further examination before they could be used. However, with the creation of the ACS questions, there was now another short set of questions, thus creating another option. The Office of Management and Budget (OMB) had been encouraging the use of the same methods of measuring demographic characteristics (including disability) when possible and recommended adopting the ACS questions rather than continuing with the development of other questions for the CPS.
BLS consulted with the ERMM Work Group and OMB to determine whether adopting the ACS questions would be supported by those who had been involved in the development effort. Concerns were raised, but the benefit of using the same questions for two surveys, along with the possibility of speeding up the development process, outweighed those concerns, and the decision was made to adopt the ACS questions for placement into the CPS.
Some changes were made to the questions that were adopted from the ACS in order for them to work properly in the CPS. The questions are asked in person or over the phone in the CPS, while the ACS is administered primarily through the mail, so certain modifications had to be made to accommodate this difference. Also, the ACS asks the question of each individual separately. For the CPS, however, the disability questions were changed in order to reduce respondent burden. In the CPS, each disability question would be posed to the entire household and would ask about the existence of some household member with a disability; any positive response would be followed up with an inquiry to clarify who the person was and whether there were any other household members with that condition.
In conjunction with Census Bureau staff working on the CPS, BLS developed training materials to prepare interviewers to administer the new questions. These materials included information obtained from Census Bureau interviewers in a focus group organized by BLS in order to take the interviewers’ experiences into account.
An individual’s disability status is identified in the same way in the ACS and the CPS. In both surveys, a response of “yes” to any one of the six questions indicates that the person has a disability. The disability questions appear in the CPS in the following format:
This month we want to learn about people who have physical, mental, or emotional conditions that cause serious difficulty with their daily activities. Please answer for household members who are 16 years old or over.
- Is anyone deaf or does anyone have serious difficulty hearing?
- Is anyone blind or does anyone have serious difficulty seeing even when wearing glasses?
- Because of a physical, mental, or emotional condition, does anyone have serious difficulty concentrating, remembering, or making decisions?
- Does anyone have serious difficulty walking or climbing stairs?
- Does anyone have difficulty dressing or bathing?
- Because of a physical, mental, or emotional condition, does anyone have difficulty doing errands alone such as visiting a doctor’s office or shopping?
In June 2008, the disability questions were added to the CPS. In the first month, questions were asked of everyone; from July 2008 forward, only those new to the CPS were asked the questions on disability.9
Publication of CPS data on people with disabilities
Despite the fact that the two surveys use the same questions, there are differences between ACS and CPS data on disability, as might be expected.10 CPS data are published on a monthly basis, providing a timely measure of the labor force status of individuals with disabilities. By contrast, the ACS collects data throughout the year but releases data annually. Also, because of the smaller size of the CPS, data on a subnational level are limited, whereas ACS data are available for many detailed geographic areas.
The summer and fall of 2008 were used to assess CPS data collection and the estimates themselves. In addition, BLS used this time to develop production plans for the release of monthly CPS data on individuals with disabilities. The first time BLS publicly released disability data was on February 5, 2009, together with the release of the January 2009 employment and unemployment estimates. A list of "Frequently Asked Questions" about the CPS disability data was posted on the BLS website in early 2009.11 The public-use microdata are available on the Census Bureau’s website,12 and unpublished tables are available from BLS upon request. On February 5, 2010, with the release of the January 2010 employment and unemployment estimates, a table containing data on the labor force status of people with disabilities was added to the monthly Employment Situation news release. In addition, on August 25, 2010, the first news release focusing specifically on the labor force status of people with disabilities was published; this release, now issued annually, presents more extensive information than that published monthly.13
CPS disability supplement
In 2009, ODEP sponsored an effort to develop a group of questions that could be asked as a supplement to the CPS. Together with BLS, ODEP hosted several meetings with interested parties in order to collect input about the topics to be covered in the supplement. The goals for the supplement that came from these outreach efforts were as follows:
- Learn more about the low labor force participation rates of people with disabilities
- Better understand how many people with disabilities use current programs designed to prepare them for employment and learn whether they are satisfied with such programs
- Learn more about the work history of people with disabilities
- Identify the different types of barriers to employment that people with disabilities experience
- Determine the types of workplace accommodations that assist people with disabilities
- Measure the use of financial assistance programs among those with disabilities and whether these programs affect their likelihood of working
BLS survey methodologists reviewed ODEP’s proposed questionnaire and conducted two rounds of cognitive interviews. Prior to fielding the questionnaire, other agencies with an interest in disability data were given the opportunity to comment on it, and after their comments were received BLS worked closely with OMB to incorporate recommended changes to the wording of the questions.
In May 2012, BLS fielded the disability supplement. The data collected were included in a news release14 that was published on the BLS website on April 24, 2013. The public-use microdata file and technical documentation were posted on the Census Bureau's website.15
The availability of monthly labor force data by disability status is a great improvement in the effort to include those with disabilities in national statistics. The analysis of these data has revealed labor market difficulties among people in this group. Efforts such as the addition of supplementary questions in the May 2012 CPS may lead to a further expansion upon what is known about the labor market challenges of those with disabilities. The attempt to improve both the set of questions used to identify those with disabilities and the depth of knowledge about this group is ongoing and sure to yield even better data with time.
ACKNOWLEDGMENT: The authors thank Scott Fricker and Anne E. Polivka for their comments on an earlier version of this article.
1 Labor force estimates of people with disabilities are listed in table 6 of the Employment Situation (U.S. Bureau of Labor Statistics, updated monthly), http://www.bls.gov/news.release/empsit.toc.htm; click on “Table A-6. Employment status of the civilian population by sex, age, and disability status, not seasonally adjusted.”
2 See “Increasing employment of adults with disabilities,” presidential documents, Executive Order 13078 of March 13, 1998 (Federal Register, March 18, 1998), http://www.gpo.gov/fdsys/pkg/FR-1998-03-18/pdf/98-7139.pdf.
4 To avoid people being classified as having a disability when they did not (“false positives”), and on the basis of the more extensive assessment derived in other parts of the NCS, it was determined that respondents needed to say “yes” to at least two of the seven questions asked in order to be classified as having a disability.
5 For a complete description of the NCS test, see Terence M. McMenamin, Thomas W. Hale, Douglas Kruse, and Haejin Kim, “Designing questions to identify people with disabilities in labor force surveys: the effort to measure the employment level of adults with disabilities in the CPS” (U.S. Bureau of Labor Statistics, Social Security Administration, Rutgers University, and California State University–Dominguez Hills, August 2005), http://www.bls.gov/osmr/pdf/st050190.pdf.
6 Each monthly sample is divided into eight representative subsamples, or rotation groups. A given rotation group is interviewed for a total of 8 months, divided into two equal periods. The rotation group is in the sample for 4 consecutive months, leaves the sample during the next 8 months, and then returns for another 4 consecutive months.
7 For a complete discussion of the CPS test, see Terence McMenamin, Stephen M. Miller, and Anne E. Polivka, “Discussion and presentation of the disability test results from the Current Population Survey,” Working Paper 396 (U.S. Bureau of Labor Statistics, August 2006), http://www.bls.gov/osmr/pdf/ec060080.pdf.
8 For a complete discussion of the ACS questions, see Matthew Brault, Sharon Stern, and David Raglin, “Evaluation report covering disability,” 2006 American Community Survey Content Test Report P.4 (U.S. Census Bureau, January 3, 2007), http://www.census.gov/acs/www/Downloads/methodology/content_test/P4_Disability.pdf.
9 Households in the CPS are divided into eight representative subsamples, as described in note 6. Households that are “new” to the CPS include those in their first and fifth months in the survey; in addition, replacement households are considered to be new to the survey.
10 Disability data from the CPS cannot be directly compared with data from other surveys. The disability questions in the CPS were originally developed for use in the ACS. Disability data from the ACS collected with the use of these questions were released for the first time in the fall of 2009 and annually thereafter. Although data users may wish to compare the estimates from the CPS with those from the ACS or other surveys that include disability questions, for several reasons it is unlikely that data from these surveys will yield identical estimates of the size and composition of the U.S. disability population. For example, the scope of the CPS is more restrictive than that of the ACS: the population covered by the CPS is confined to the civilian noninstitutional population ages 16 and older, while the ACS includes those in institutions. There are additional differences as well, including the mode of survey collection and the context of the survey. All of the differences need to be accounted for in any attempt to compare data from two different surveys.
11 See “Frequently asked questions about disability data,” Labor force statistics from the Current Population Survey (U.S. Bureau of Labor Statistics, November 20, 2012), http://www.bls.gov/cps/cpsdisability_faq.htm.
12 Disability microdata from the CPS are available from the U.S. Census Bureau. Beginning with data for 2009, disability status variables are part of the basic monthly files at http://thedataweb.rm.census.gov/ftp/cps_ftp.html. The 2008 disability data are in a separate extract file at http://thedataweb.rm.census.gov/ftp/cps_ftp.html#cpsbasic_extract. As with all CPS microdata, personally identifiable information has been removed.
15 As mentioned in note 12, the disability supplement microdata file is available on the Census Bureau’s website at http://thedataweb.rm.census.gov/ftp/cps_ftp.html#cpssupps. For technical documentation, see Current Population Survey, May 2012, disability supplement file: technical documentation CPS—12, http://www.census.gov/prod/techdoc/cps/cpsmay12.pdf.