Adding questions on certifications and licenses to the Current Population Survey
While sources of federal data on educational attainment are plentiful, information on nondegree credentials, such as professional certifications and state or industry licenses, is much more limited. Including a small set of questions about certifications and licenses on the Current Population Survey (CPS)—the monthly source of labor force statistics published by the U.S Bureau of Labor Statistics—would add valuable information about the labor market and would enable researchers and policymakers to examine how these types of credentials affect labor market success. After a lengthy development process, three questions on certifications and licenses were added to the monthly CPS in January 2015. This article describes the process of developing the CPS questions, starting with the early work on question development done by the Interagency Working Group on Expanded Measures of Enrollment and Attainment (GEMEnA), which was formed late in 2009. It then describes the work done by the Bureau of Labor Statistics to adapt the questions developed by GEMEnA to the CPS, the development and acceptance of a proposal to add the questions, the fielding of the questions, and the first publication of results.
There are many sources of federal data on educational attainment. These measures generally use the traditional educational attainment categories, which range from less than a high school diploma through a bachelor’s degree or higher. However, many other credentials appear to be valuable in the labor market, such as professional certifications and state or industry licenses. Until recently, there were few federal sources of data on these types of credentials.
In response to the increased interest in data about alternative credentials, the Interagency Working Group on Expanded Measures of Enrollment and Attainment (GEMEnA) was formed late in 2009 and continued into 2016. GEMEnA was chaired by the Department of Education’s National Center for Education Statistics (NCES). Over the years, members included senior staff from the U.S. Bureau of Labor Statistics (BLS), the Census Bureau, the Council of Economic Advisors, the U.S. Department of Education’s Office of the Under Secretary, the National Center for Science and Engineering Statistics, and the OMB. GEMEnA worked on a variety of education- and credential-related subjects, including certifications and licenses.1
Two of GEMEnA’s main goals were
- to develop a limited set of questions to measure the prevalence and key characteristics of nondegree credentials, such as certifications and licenses—these questions could then be placed on existing federal household surveys;2 and
- to develop a common vocabulary for nondegree credentials among federal statistical agencies, federal programs, and state data systems—this would allow people to better communicate about the data.
In 2009, President Obama called for Americans to commit to achieving at least 1 year of higher education or career training,3 and as a result, GEMEnA concentrated its initial research efforts on subbaccalaureate certifications and licenses. Over time, however, GEMEnA expanded its scope to include certifications and licenses at all educational levels.
This article describes the process of developing the questions on certifications and licenses for the Current Population Survey (CPS),4 starting with the early work on question development done by GEMEnA. It then describes the work done by BLS to adapt the questions to the CPS, the development and acceptance of a proposal to add the questions, the fielding of the questions, and the first publication of results.
Defining certifications and licenses
GEMEnA began its work on certifications and licenses by conducting a literature review, consulting with subject matter experts, and reviewing the credential-related questions that existed on a variety of nonfederal surveys. After these efforts, the group identified two distinct types of credentials that are commonly referred to as certifications or licenses: (1) those that are awarded by a certification body and reflect the acquisition of a level of skill in an occupation but do not convey a legal authority to work in that occupation and (2) those awarded by a governmental body that convey a legal authority to work in an occupation. Because there was some confusion about how the two terms were used, GEMEnA created working definitions for certifications and licenses. These definitions were a first step in developing the common language to better discuss these credentials. (See table 1.)
|Awarded by whom||A credential awarded by a nongovernmental certification body on the basis of individuals demonstrating, through an examination process, that they have acquired the designated knowledge, skills, and abilities to perform a specific job; the examination can be written, oral, or performance based||A credential awarded by a governmental licensing agency on the basis of predetermined criteria; the criteria may include some combination of degree attainment, certifications, educational certificates, assessment, apprenticeship programs, or work experience|
|Legal authority||Does not convey a legal authority to work in an occupation||Conveys a legal authority to work in an occupation|
|Renewal||Time limited, renewed through a recertification process(1)||Time limited, must be renewed periodically|
|Examples||Information technology certification and project management professional certification||Cosmetology license and teacher’s license|
(1) Some certifications are time limited but do not have a recertification process. For example, many information technology certifications are for a specific software that becomes obsolete when it is replaced with a newer version.
Source: Interagency Working Group on Expanded Measures of Enrollment and Attainment and U.S. Bureau of Labor Statistics.
GEMEnA also investigated a third type of nondegree credential—educational certificates. According to GEMEnA definitions, educational certificates, which are awarded by an educational institution upon the recipient’s completion of all requirements for a program of study, are not the same as certifications and licenses. GEMEnA’s background research showed that certifications and licenses are quite distinct from educational certificates. Certifications and licenses are job or occupational requirements which demonstrate that the holder has the authority or skills that are needed to perform specific job duties. By contrast, educational certificates are awarded to show completion of a program of study; while this training may help in the performance of a specific job, educational certificates are not necessarily required or considered proof of qualification.5 Like educational certificates, certificates of attendance or participation in short-term training are also considered to be different from certifications and licenses.
GEMEnA focus groups and cognitive testing
After developing definitions of certifications and licenses, GEMEnA decided that some exploratory qualitative research would be helpful. In December 2009, NCES funded three focus groups in the Washington, DC, area on nondegree credentials. Reflecting GEMEnA’s early interest in subbaccalaureate certifications and licenses, the focus group participants were ages 21 to 40 and had at least a high school diploma or GED (a high school equivalency diploma) but did not have a bachelor’s degree. Two of the focus groups were composed of people who said they were certified, registered, or licensed in a health care, information technology, or business field. A third focus group was made up of individuals who said they had a certification in another field or who were interested in obtaining a certification.6
In 2010, NCES also funded three rounds of cognitive testing (60 interviews total) to determine whether people could answer questions about nondegree credentials. Participants included individuals with certifications or licenses as well as those without these credentials. Because of the concern that there might be regional differences, the interviews were conducted with people in three different areas: Washington, DC; Charlotte, NC; and Minneapolis, MN. Respondents were ages 21 to 40. The first round of testing concentrated on certifications and licenses, the second on educational certificates, and the third on both types of credentials.7
In general, the focus groups and cognitive testing confirmed that respondents were aware of their certifications and licenses and were able to answer questions about them. For many, these types of credentials were necessary for their jobs, either because the law or their employer required them. Particularly in some occupations, respondents often had more than one credential. Also, respondents did not confuse these credentials with the traditional educational attainment categories, and often closely associated certifications and licenses with their work.
Because of the interest in distinguishing between certifications and licenses, GEMEnA investigated the possibility of asking respondents to identify their credentials as either a certification or a license. However, in both the focus groups and the cognitive interviews, some participants found it difficult to distinguish between the two types of credentials, with a few saying that they thought the two terms were synonymous. Many factors contribute to this confusion. In some fields, certification is a prerequisite for applying for a license—such as a financial planner who must first become a certified financial planner before applying for a license through a state board of standards. The reverse is also true in some fields—that is, the individual must first obtain a license before becoming certified. For example, a real estate agent may need to get a license before being certified in a specialization area. In addition, in some fields, both “certification” and “license” may be used to refer to the same type of credential. For example, all states have credentialing requirements for public school teachers. These requirements are variously referred to as teaching licenses, certifications, credentials, or certificates. However, although state requirements vary, all of these credentials are issued by a governmental agency and convey a legal authority to teach school. Thus, according to GEMEnA definitions, these credentials are considered licenses.
Rather than asking about certifications and licenses separately, GEMEnA decided to test whether the two types of credentials could be asked about together. In the third round of cognitive testing, both the terms “certification” and “license” were used in the same question—that is, people were not asked to distinguish between the two but simply to indicate yes or no as to whether they had a “certification or license.” This approach was found to work well.8
Adult Training and Education Survey pilot study
The 2010 Adult Training and Education Survey (ATES) pilot study was an NCES-sponsored 15-minute mail and telephone survey that was fielded from September 2010 to January 2011. A major goal of this study was to evaluate proposed questions on certifications and licenses.9 The pilot study was designed to evaluate estimates generated from the questions, investigate the extent of measurement error, and determine whether proxy respondents—that is, respondents who provide information about other members of their households—could accurately answer the questions.10
Evaluating the estimates
The pilot contained a nationally representative sample of 3,730 adults age 18 and over. Although the pilot was never intended to provide official population estimates, weights were developed so that nationally representative estimates could be generated. Although no federal data on certifications and licenses existed at the time, there had been periodic attempts, using nonfederal surveys, to measure the numbers of people with certifications or licenses. Pilot-study estimates were compared with estimates from the Princeton Data Improvement Initiative (PDII), a survey that had been conducted in 2008 and included questions on occupational licensing. To create comparable estimates, the ATES and PDII samples were restricted to employed adults. According to the ATES, 38 percent of employed adults had a certification or license, compared with 34 percent in the PDII. Although these estimates are not identical, the difference is relatively small, especially given that the two surveys were conducted 2 years apart and had a number of differences in their questionnaires. GEMEnA concluded that estimates from the ATES questions were reasonable as compared with previous data collection efforts.
In addition to including the nationally representative sample, the ATES included a nonrandom sample of 340 adults who were known to hold nondegree credentials. Some of the adults in this seeded sample had been reported by a credentialing body to have received a certification or license.11 The seeded sample contained people in the following certified or licensed occupations: alcohol and drug counselor or aide; health technician, specialist, or sonographer; pharmacy technician; electrical aide; plumber; heating, ventilation, and air conditioning technician; and fuel technician. The seeded sample was included specifically to evaluate how well respondent answers lined up with the administrative data, allowing for the calculation of underreporting rates. Here are the underreporting rates (in percent) of certifications and licenses by age and employment status:
|18 to 34 years||19.0|
|35 to 64 years||11.6|
|65 years and over||29.2|
|Employed by a private company||15.4|
|Employed by government or nonprofit, or self-employed||6.9|
Although the underreporting rates vary for different groups, there are few statistically significant differences across different subgroups. A multivariate analysis showed that differences in underreporting rates are driven, in part, by employment status, with employed people being more likely than the not employed to accurately report that they have a certification or license. This difference may reflect the fact that a certification or license could be more meaningful for people who are employed and who, therefore, have these credentials for work-related reasons. Those who have credentials but are not employed—the majority of whom are not looking for work—may not view them as relevant and thus might be less likely to report them. Also, while considerable effort was made to find current and accurate administrative records for the seeded sample, the measurement error associated with these records is unknown. It is possible that some reporting differences could be due to errors in the seeded sample frame. After taking all of these factors into account, GEMEnA determined that the underreporting rates in the pilot were within acceptable ranges.12
Many federal household surveys use proxy respondents. Thus, a key factor in developing questions to include on existing surveys was to determine whether respondents could answer the questions about others in their households. To evaluate the questions for proxy respondents, the 2010 ATES pilot study, in a subsample of about 1,000 households with 2 or more adults, asked adults to report about other adults in their household. The study did not include multiple respondents from the same households, so direct comparisons between self-responses and proxy responses about individuals could not be done. Instead, GEMEnA compared the frequencies of self-responses and proxy responses to determine whether there were large differences in the incidence of “don’t know” responses. High rates of nonresponse among proxies might lead to bias in the survey estimates and would mean the questions were unsuitable for surveys relying on proxy respondents. However, nonresponse rates for proxy respondents were fairly low for the main certification and license questions, and GEMEnA determined that proxy reporting was acceptable for these questions.13
After all the ATES results had been examined, GEMEnA made several recommendations.14 Because of cost constraints and concerns about increasing respondent burden, most federal household surveys—particularly large surveys like the CPS—can add no more than a few questions, and so GEMEnA recommended only a small number of questions. The group suggested that surveys include the following:
- The broad question about certifications and licenses used in ATES, which GEMEnA determined worked well.
- A question to distinguish between certifications and licenses. Certifications are issued by industry or professional organizations, while licenses are issued by government agencies.
- A question to determine the field of the certification or license. However, GEMEnA recognized that surveys with little available space for new questions might be unable to collect this information, particularly in the case of individuals with multiple certifications or licenses.
- A question to determine whether the certification or license is related to an individual’s current work.15
Fielding the GEMEnA questions
A number of federal surveys have added GEMEnA questions on certifications and licenses. The first federal survey to include questions developed by GEMEnA was the Survey of Income and Program Participation (SIPP). In the autumn of 2012, nine questions on certifications and licenses were added to the 13th wave of the 2008 SIPP. These included the main questions recommended by GEMEnA as well as questions about whether coursework was required to earn the credential, the main reason for obtaining the credential, and whether the credential could be used to get a job with any employer in the field.16
After the SIPP had been fielded, the questions were proposed for a number of other federal household surveys, such as the National Science Foundation’s National Survey of College Graduates and several NCES surveys. The number of questions each survey could include varied according to the goals of the survey and how much of the questionnaire could be devoted to the topic.17
Because of the successful use of the questions in other surveys, BLS, the Census Bureau, NCES, and GEMEnA members began to consider whether the questions would be a valuable addition to the CPS.
Current Population Survey
The CPS is a survey of approximately 60,000 households that is jointly sponsored by BLS and the Census Bureau. The survey is probably best known for the national unemployment rate, which is published every month by BLS. Statistics from the CPS are among the country’s most timely economic indicators and are widely used by policymakers and researchers. The CPS provides extensive information about the employment and unemployment status of the population, and the survey data can be broken out by a variety of demographic characteristics, such as race and ethnicity, educational attainment, disability status, age, and gender. In addition, the CPS is a primary source of socioeconomic data about the labor force, including industry and occupation, hours of work, and earnings.
Including a small set of questions about certifications and licenses would add valuable information about the labor market and would enable researchers and policymakers to examine how these types of credentials affect labor market success. Unemployment rates and other key labor market indicators could be compared for those with and without certifications and licenses and could be broken out by the many demographic and labor force characteristics already collected in the CPS.
The core part of the survey is referred to as the monthly CPS (or the basic CPS). Households are in the survey for 8 months total on what is referred to as a 4-8-4 rotation—that is, they are in the sample for 4 months, out of the sample for the next 8 months, and back in the sample for the following 4 months. For example, a household that was scheduled to be interviewed for the first time in April would also be contacted in May, June, and July. The household would then be out of the sample for the next 8 months and would return to the sample the following April, May, June, and July.
In addition to the monthly CPS, there are periodic CPS supplements, which are typically included in a particular month of the year. Supplements are sets of questions on a specific topic and generally take no longer than 10 minutes to administer. The best known of the supplements is the Annual Social and Economic Survey (ASEC), also sometimes referred to as the “March supplement”; this supplement is the source of the poverty rate as well as other income and health insurance statistics published annually by the Census Bureau.
An immediate question of the working group was where to add questions to the CPS. Likely CPS supplement candidates were quickly narrowed down to two: the ASEC and the October school enrollment supplement.18 Another alternative was to add the questions to the monthly CPS. Each of the three candidates had advantages and disadvantages, as shown in table 2.
|Annual Social and Economic Survey (ASEC)||The ASEC contains detailed income data, and one key research question is how certifications and licenses affect income.||The ASEC is by far the longest of all CPS supplements. Adding questions would make it even longer and further increase the burden on respondents. This would severely limit the number of questions that could be included. Data users might not find a small set of questions sufficient for analytical needs.|
|Analysis of detailed subgroups would be limited because of the relatively small sample size.|
|School enrollment supplement||Because the supplement is fairly short, a more extensive set of questions about certifications and licenses could be included.||The supplement does not contain information about income and thus could not be used to answer key research questions on that topic.|
|Questions about certifications and licenses might not work well in the supplement because the cognitive task of reporting about enrollment might be too different from that of reporting about nondegree credentials.|
|Analysis of detailed subgroups would be limited because of the relatively small sample size.|
|Monthly Current Population Survey||Adding questions to the monthly CPS would create a monthly time series, allowing for an examination of cyclical trends, as well as long-term trends.||Since the survey is given every month, questions are added very rarely because of the increase in respondent burden. Only a very small set of questions could be added, and data users might not find such a small set sufficient for analytical needs.|
|Information on certifications and licenses could be linked to data collected in any CPS supplement, including income data from the ASEC and information from the school enrollment supplement.|
|Because the CPS is asked each month, the data can be pooled across months to increase sample size. This would allow for more detailed examination of subgroups.|
|Source: U.S. Bureau of Labor Statistics.|
All agreed that the monthly CPS was the most desirable option for the questions. This choice was largely because the monthly CPS provided the greatest flexibility: it would provide monthly data suitable for cyclical analysis; would allow for analysis of the income data on the ASEC; and could be linked to any other supplement, including the school enrollment supplement.
However, there were considerable challenges to adding questions to the monthly CPS. Changes are made occasionally to CPS supplements but only rarely to the monthly CPS. In general, both BLS and the Census Bureau are very reluctant to increase the length of the monthly survey. Doing so not only increases respondent burden but also could make breakoffs in a given month or refusals in future months more common. In addition, there is always the possibility that a change in the monthly survey could adversely affect responses to other questions, including those related to unemployment. While such an effect may be unlikely, BLS and Census take great care to ensure that changes to the monthly CPS do not affect the unemployment rate and other key statistics. Also, BLS did not have funding available to pay the ongoing costs of permanently increasing the overall length of the survey. Therefore, both BLS and Census agreed that only a limited set questions could be added and that if questions were added, some existing questions would have to be removed to limit the increase in cost and respondent burden.
Identifying questions to add to the CPS
BLS, Census, and GEMEnA held a series of discussions about the questions. Three questions were proposed for addition:
- A question to identify certifications or licenses. This is the main question recommended by GEMEnA and has been shown to reliably identify individuals who have these credentials. Because the monthly CPS is primarily a labor force survey, capturing this information for both the employed and the unemployed is important. Doing so would enable BLS to produce a variety of labor force statistics regarding certifications and licenses, including unemployment rates.
- A question to identify whether the credential was issued by any level of government. The intent of this question is to determine the issuing authority of a credential and, thus, to distinguish between certifications and licenses. According to GEMEnA definitions, licenses are issued by a governmental body.
- A question to determine whether the certification or license was needed for the current job. In this question, “required” can mean either legally required (such as a license to practice medicine) or required by the employer (such as an information technology certification). People may have credentials that they do not use for their current job, such as a computer technician who obtained a real estate license for personal interest or for a second or previous job. Also, people sometimes have certifications or licenses that may be helpful for their jobs but are not required by law or by their employer.
Two other questions were briefly considered:
- A question about how long the individual had held a credential. GEMEnA had not yet tested and approved the wording of such a question. BLS survey methodologists were concerned about the possibility of recall bias, particularly among proxy respondents. Therefore, considerable work would have to be done to develop an acceptable question and would likely have delayed implementation.
- A question about the field of the certification or license. Although GEMEnA had tested versions of this question, all were too long for the monthly CPS. Additionally, Census staff would have to review respondents’ answers and code them into a set of predetermined categories. This would be time consuming and labor intensive, making such a question prohibitively expensive. Also, for those certifications or licenses needed for individuals’ jobs, the broad topic of the credential could be inferred from the information on occupation and industry already collected in the monthly CPS.
Because of these concerns, as well as concerns about increasing the respondent burden and survey costs, BLS and Census determined that adding either of these questions was not practical.
Identifying questions to remove from the CPS
As previously mentioned, BLS and Census wished to limit the increase in the length of the survey and the corresponding increase in respondent burden. Therefore, both agencies agreed that if questions on certifications and licenses were added, some questions would have to be removed.
After an evaluation of the CPS questionnaire, three educational attainment questions were identified for possible removal. These educational attainment questions had been added to the CPS in 1996 and were designed to allow researchers to construct a measure of continuous years of education as an alternative to degree-based measures, but the information was little used. Data from these questions had never been published by BLS, and a literature review found very few publications using the questions, even though variables derived from the questions had been included on the CPS public-use microdata files for many years.
The educational attainment questions proposed for removal were as follows:
- Since completing your bachelor’s degree, have you ever taken any graduate or professional school courses for credit? [Asked of people whose highest educational attainment was a bachelor’s degree]
- Did you complete six or more graduate or professional school courses? [Asked of people who responded “yes” to the previous question]
- Was your master’s degree program a 1-year, 2-year, or 3-year program? [Asked of people whose highest educational attainment was a master’s degree]
Developing the proposal
BLS formed a committee of CPS subject matter experts, survey methodologists, and research economists to develop a proposal to add questions on certifications and licenses to the monthly CPS. In addition to relying on internal expertise, BLS also consulted with GEMEnA and the GEMEnA expert panel. This proposal was to add three questions on certifications and licenses and remove the three little-used questions on educational attainment, but many details needed to be worked out, such as question wording, placement, and frequency.
For all three questions, BLS started with the question wording recommended by GEMEnA. After the wording was reviewed by BLS survey methodologists, subject matter experts, and GEMEnA members, a few minor revisions were made to make the questions work in the context of the CPS. Specifically, because the CPS is collected through personal and telephone interviews, respondents answer for themselves and for other members of their households. Hence, a number of wording changes needed to be made to accommodate proxy respondents.
On the first question, the most notable change was that a sentence was added to the first question about business licenses. This was done because in the SIPP, people whose businesses required licenses (such as liquor licenses) sometimes answered that they had a license. Also, some clarifications were made to the text, which the interviewer could read if necessary. In addition, GEMEnA suggested some minor refinements to the question text. Final wording of the first question is as follows:
1. Do (you/name) have a currently active professional certification or a state or industry license? Do not include business licenses, such as a liquor license or vending license.
(Read if necessary: A professional certification or license shows you are qualified to perform a specific job. Examples include a real estate license, a medical assistant certification, a teacher’s license, or an IT certification. Only include certifications or licenses obtained by an individual.) [Asked of people age 16 and over]
For the second question, a change was made to accommodate respondents with more than one certification or license. The original question wording was to ask “was it issued by the federal, state, or local government?” For people with multiple certifications or licenses, such a question would be confusing, especially if the answer were different for different credentials. GEMEnA and the GEMEnA expert panel decided it was most important to find out if the respondent had at least one license. As they recommended, the second question was modified to allow for the situation in which a person has more than one certification or license. Instead of “was it issued by . . . ” the question wording was modified to “were any of your certifications or licenses issued by . . . ” to allow for the possibility of multiple credentials:
2. Were any of (your/his/her) certifications or licenses issued by the federal, state, or local government? [Asked of people who responded “yes” to the previous question]
For the third question, modifications to the text were made for two reasons. First, like the second question, the third question required some modifications to wording for people who had multiple credentials. Second, CPS subject matter experts decided that it was important to ask the third question of the unemployed about their past jobs. This is consistent with other labor force questions, such as industry and occupation, and allows for an examination of the unemployed who have previously worked. The wording had to be modified for these individuals to clarify that it was their previous job that was being asked about. The final wording of the third question is as follows:
3. Earlier you told me (you/name) had a currently active professional certification or license. Is (your/his/her) certification or license required for (your/his/her) (job/main job/job from which [you are/he is/she is] on layoff/job at which [you/he/she] last worked)? [Asked of people who were employed and those who were unemployed but had previously worked]
Note that the wording differs by individuals’ labor force status. For example, people who are employed at only one job are asked about their job, while those with multiple jobs are asked about their main job. Likewise, unemployed individuals who are on layoff are asked about the job from which they are on layoff, and unemployed individuals who previously worked are asked about their last job.
The CPS has a long and complex questionnaire, and so another task for the BLS working group was to determine where best to place the questions. It was important that the questions not confuse respondents or interfere with responses to other questions. Because results from GEMEnA focus groups had shown that people did not necessarily regard certifications and licenses as related to educational attainment but rather as credentials for their jobs, the team evaluated whether they would be most appropriate next to other job-related questions.
After consideration, BLS decided that placing the first two questions after the educational attainment questions made the most sense. The primary reason behind this decision was the importance of collecting information from these two questions for all individuals, not just for those who were employed. Unlike the job-related questions, the educational attainment questions are asked of everyone. The third question, which asked specifically about whether the credential was for the respondent’s job, did belong among other job-related questions.
BLS also needed to determine how often the questions should be asked. Some questions on the CPS are asked only once because the requested information is unlikely to change, such as questions about race, ethnicity, and foreign- or native-born status. By contrast, questions on labor force status are asked every month because the primary purpose of the monthly CPS is to track the nation’s employment and unemployment characteristics, and it is vital that this information is as current as possible. Other questions are asked with varying periodicity; for example, educational attainment is asked of people in their first and fifth interviews and also in July, October, and February. Questions about disability are asked in the first and fifth interviews.
After consultation with GEMEnA and the GEMEnA expert panel, BLS determined that whether or not a person has a certification or license is not likely to change much from month to month. In order to limit the addition to respondent burden, BLS proposed that the questions be asked only in the first and fifth interviews. A further reason for not asking the questions more often was the desire to limit the increase in the length of the survey.
Evaluating the proposal
Evaluation of the proposal took several months and included cognitive testing by BLS survey methodologists. It also involved considerable stakeholder outreach by both GEMEnA and BLS.
The questions on certifications and licenses had been thoroughly tested through the GEMEnA work. However, BLS staff felt that it was important to test the questions within the CPS to ensure that they worked as expected. Specifically, the cognitive testing sought to
- test the placement of the questions
- test the minor wording changes that had been made to the questions
- determine whether the new questions would likely have an impact on existing CPS questions
- determine if introductory or transition language is necessary before the questions
- determine if interviewer instructions or help screens are necessary to explain the key concepts.
About 25 cognitive interviews were conducted. Respondents included individuals who had certifications or licenses as well as those who did not. The testing did not reveal any problems with the proposed placement or wording of the questions. Also, the cognitive testing did not find evidence that the new questions had an impact on responses to existing questions. Since adding transition language would lengthen the survey and respondents did not appear to have any difficulty understanding the questions without such language, the cognitive testing report recommended against adding transition language. The report did recommend that interviewers be trained on the objectives of the new questions.19
A great deal of outreach on the CPS proposal was done through GEMEnA. The proposal was presented to the GEMEnA expert panel at their annual meeting, and GEMEnA members presented the proposal to a variety of governmental and nongovernmental groups. As an agency particularly concerned with measures of educational attainment, NCES was also consulted.
In addition, BLS reached out to key stakeholders. Specifically, BLS sought to learn whether stakeholders thought the questions would be a valuable addition to the survey, whether the removal of the three educational attainment questions was an acceptable price to pay, and whether they thought BLS should proceed with the proposal.
The proposal was also presented to the BLS Technical Advisory Committee. Informally, BLS contacted key stakeholders who might have an interest in the educational attainment questions proposed for removal. In addition, the new questions were reviewed and cleared by the Office of Management and Budget, and the clearance process included two periods of public comment. Feedback to the proposal was almost exclusively favorable.
Collection of data
Because stakeholder feedback was positive and the cognitive testing revealed no issues, BLS decided to accept the proposal to permanently add the three questions about certifications and licenses to the monthly CPS. In accordance with the proposal, the first two questions would be asked after the educational attainment questions, and the third would be asked in the labor force section. The three questions would be asked only in the first and fifth interviews, though the questions would be asked in other months of new household members. Also, the third question would be reasked in other months in certain circumstances, such as when an individual changed jobs or an unemployed person became employed.
As recommended in the cognitive testing report of the proposed CPS questions, the BLS working group developed training materials that were given to interviewers prior to the first collection of data. These materials covered common issues that had come up in cognitive testing at BLS and through the GEMEnA work. In addition to being reviewed by BLS staff, the training materials were reviewed by GEMEnA members. These materials were also incorporated into a section in the CPS interviewers’ manual.20
The training materials were provided to interviewers prior to the introduction of the new questions with the collection of data for January 2015. Had the questions been asked starting in January only in the first and fifth interviews, the full sample would not have received the questions until May 2016. Thus, annual averages would not have been available until 2017, and ASEC data about certification and licenses would likewise not have been available until 2017. To allow for the possibility of 2015 annual averages and 2015 ASEC data, the questions were asked in all interviews (not just the first and fifth) in January 2015. While responses to the questions asked in other than the first and fifth interviews might be different, the risk was thought to be minimal.
During the first year of data collection, BLS staff listened to many interviews in order to ensure that the questions were working as expected. BLS staff also evaluated the data each month to identify any possible issues. One significant data collection problem was encountered: because of a programming error, the third question was not asked in the first and fifth interviews in May and June. The lack of data in these 2 months resulted in BLS not including 2015 annual averages for the third question in the first publication of data on certifications and licenses. Moreover, a public-use variable derived from the third question was not produced for 2015.21
Nonresponse on the questions is quite low. However, as with most survey questions, there are respondents who either refuse to answer or do not know the answers to the questions. As is customary for nonresponse to questions on the CPS, BLS economists devised an imputation method to fill in missing data. This is often done with survey data because this type of nonresponse is unlikely to be random; analyses using only reported values implicitly assume that all respondents are equally likely or unlikely to respond and that the estimate is approximately unbiased. The imputation method used for the questions on certifications and licenses is similar to that used for other CPS variables.22
BLS published the first CPS data from the questions about certifications and licenses on its website in April 2016.23 As mentioned earlier, only data from the first two questions were tabulated for this first publication. An extract file containing public-use variables for 2015 for the first two questions is available to researchers on the Census Bureau website. These variables can be linked to the 2015 CPS public-use microdata files, enabling researchers to conduct their own analyses. A similar extract file that includes variables for all three questions will be made available early in 2017. Also, starting with the release of data for January 2017, variables for all three questions will be included on the CPS public-use microdata files.24
Mary Dorinda Allard, "Adding questions on certifications and licenses to the Current Population Survey," Monthly Labor Review, U.S. Bureau of Labor Statistics, November 2016, https://doi.org/10.21916/mlr.2016.52.
2 Realizing that adding a limited set of questions to large national household surveys could not address all the many questions about certifications and licenses, GEMEnA also worked to develop a more extensive survey for more indepth analysis to be administered by NCES.
3 U.S. President Barack Obama, Address to Joint Session of Congress, February 24, 2009. See https://obamawhitehouse.archives.gov/the-press-office/remarks-president-barack-obama-address-joint-session-congress/.
5 Educational certificates are defined by GEMEnA as credentials awarded by an educational institution—such as a community college, another type of college or university, or a trade school—on the basis of completion of all requirements for a program of study, including coursework and test or other performance evaluations, and are not time limited. Examples of these types of credentials include a digital arts certificate from an online university or a motorcycle mechanics diploma from a community college. Most educational certificates are awarded at the subbaccalaureate level, and a small number are awarded after the completion of a postsecondary degree. Individuals may enroll in an educational certificate program to gain the knowledge needed to attain a certification, which may be required prior to applying for a license. For more information about educational certificates, see the Adult Training and Education Survey (ATES) pilot study technical report, U.S. Department of Education, April 2013, http://nces.ed.gov/pubs2013/2013190.pdf.
6 For detailed information about the focus groups, see appendix F, ATES pilot study technical report.
7 For detailed information about the cognitive testing, see appendix G, ATES pilot study technical report.
8 Over the years, a number of additional focus groups and cognitive testing efforts have included questions about certifications and licenses. These have supported the findings of the original GEMEnA focus group and cognitive testing reports.
9 The ATES pilot study also included questions about educational certificates. For more information, see ATES pilot study technical report.
10 For more information about all of these analyses, see the ATES pilot study technical report.
11 All those in the seeded sample were known to have a nondegree credential. However, some of those in the seeded sample had educational certificates rather than certifications or licenses.
12 Further GEMEnA studies have also included seeded samples. Underreporting rates of the number of certifications and licenses from the later studies are consistent with the ATES.
13 More detailed questions about certifications and licenses, such as the credential name and the year the credential was awarded, were not found to work for proxy respondents.
14 More detailed discussion of these recommendations can be found in the ATES pilot study technical report.
15 Two other pilot studies were conducted that included certification and license questions: the 2013 National Training and Education Survey pilot study and the National Household Education Survey feasibility study in the 2014 ATES. These pilot studies had similar findings.
18 The school enrollment supplement has been sponsored by NCES, BLS, and the Census Bureau since the late 1960s.
19 For more information on the cognitive testing, see appendix H of the 2014 Office of Management and Budget clearance for the CPS.
20 Interviewer instructions on the questions about certifications and licenses can be found on pages C3-22 and C3-23 of the Current Population Survey interviewing manual, U.S. Census Bureau, April 2015, http://www2.census.gov/programs-surveys/cps/methodology/intman/CPS_Manual_April2015.pdf.
21 Three other minor data collection issues occurred. In January 2015, the first two questions were not asked of people who were known to be over age 15 but who had not provided an exact age. Also, for the first half of the year, the questions were not asked of people who had responses of “don’t know” or “refused” for the educational attainment questions. BLS investigated both of these issues and found that they affected a small number of respondents. After examining later months, BLS determined that responses for these groups do not appear to be systematically biased. A third issue was that the third question on certifications and licenses was not asked of certain people who should have been asked; for example, some people who were unemployed and had previously been employed were not asked the question.
22 Typically, CPS imputation methods rely on information about basic demographic characteristics, such as age and educational attainment. Unlike other CPS imputation methods, the imputation for certifications and licenses also uses information on labor force status and occupation. More information about imputation methods can be found in chapter 9 of Design and methodology: Current Population Survey, technical paper 66, U.S. Bureau of Labor Statistics and U.S. Census Bureau, October 2006, https://www.census.gov/prod/2006pubs/tp-66.pdf.
23 Results are available on the BLS website at https://www.bls.gov/cps/certifications-and-licenses.htm.