Bureau of Labor Statistics > Occupational Requirements Survey > Home

Preproduction Collection Report

Executive Summary

In fiscal year (FY) 2015, the Bureau of Labor Statistics (BLS) completed data collection for the Occupational Requirements Survey (ORS) pre-production test. The pre-production test might better be described as a “dress rehearsal” as the sample design, collection procedures, data capture systems, and review were structured to be as close as possible to those that will be used in production. While the feasibility tests in FY 2014 and earlier were intended to gauge the viability of collecting occupational data elements and to test modes of collection and procedures, in FY 2015 BLS integrated the prior work into a large-scale nationally representative pre-production test.

The intent of this report is to assess the pre-production collection, drawing some preliminary conclusions regarding whether BLS is able to successfully capture data on the requirements of occupations that will meet the needs of the Social Security Administration (SSA). BLS notes, however, that though the collection and review of pre-production ORS data is complete as of May 22, 2015, the estimation processes are just beginning. The report on ORS pre-production estimation results is anticipated to be released before the end of FY 2015. BLS also notes that because estimation processes have not been applied, the numbers presented in this report are unweighted and should be expected to differ from the final, weighted numbers that will be available in the estimation report.

Preliminary indications show that BLS is able to successfully collect data on occupational requirements that will meet the needs of the SSA. In particular, the overall and item response rates, quality assurance program, and feedback from data collection debriefs provide support that survey procedures are working well and data on job requirements can be collected by BLS field economists. As BLS moves into production, however, there will be continued refinements and testing, particularly for the mental and cognitive elements, as described in this report.

Background and pre-production test overview

In the summer of 2012, the SSA and BLS signed an interagency agreement, which has been updated annually, to begin the process of testing the collection of data on occupational requirements. As a result, ORS was established as a test survey in late 2012. The goal of ORS is to collect and publish occupational information that meets the needs of SSA at the level of the eight-digit standard occupational classification (SOC) that is used by the Occupational Information Network (O*NET).(1)

The ORS data are collected under the umbrella of the National Compensation Survey (NCS), which uses Field Economists (FEs) to collect data. FEs are well-suited for ORS data collection as their training (described in the “Quality Assurance” section below) focuses on identifying the appropriate respondent, probing the respondent to clarify apparent inconsistencies in responses, and following up with respondents to ensure data are complete and accurate. FEs generally collect data elements through either a personal visit to the establishment or remotely via telephone, email, mail, or a combination of modes.

For ORS, FEs are collecting occupationally-specific data elements to meet SSA’s needs including:

Physical demands
Educational requirements
Mental and cognitive demands
Environmental conditions in which the work is performed.

In fiscal years 2013 and 2014, several feasibility tests were performed to assess the viability of collecting data on occupational requirements using the platform currently used by the NCS. These tests provided evidence that the NCS platform could be adapted to ORS data collection, which led to the pre-production test in FY 2015.(2) Unlike the earlier tests, which were small-scale and tested a subset of data elements or the viability of different collection methods, the pre-production test was designed as a relatively large-scale, nationally representative test of ORS data collection. The sampling, data collection, procedures, and review were designed to mimic what will occur during ORS production beginning in September 2015.

This report summarizes the pre-production test from sampling through review, describes future testing, and discusses next steps for ORS data collection as BLS moves into production. It is important to note that the end of pre-production data collection does not complete the test – estimation, validation, and dissemination processes are on-going and a report that covers these topics will be available at the end of FY 2015.

Sampling

The ORS pre-production sample was drawn from the same frame as the NCS – the Quarterly Census of Employment and Wages, which includes all establishments covered by state unemployment insurance laws, and a supplementary file of railroads. The frame contains virtually all establishments in the 50 United States and the District of Columbia in the private sector (excluding agriculture, forestry and fishing and private households) and in state and local governments. The pre-production ORS sample contained 2,549 establishments.

Roughly one-third of the ORS pre-production sample consisted of establishments that are also in the NCS sample (“NCS-ORS overlap”) and the remainder are ORS-only.(3) Across all establishments (NCS-ORS overlap and ORS-only), approximately 15% are government owned and 85% privately owned.

For each establishment in the ORS sample (establishments are also referred to as “schedules”), jobs are selected for data collection through probability selection of occupations; these jobs are referred to as “quotes”.(4) The number of jobs selected within a private establishment varies from 4 to 8, based on establishment size, and in government, the number of jobs varies from 4 to 20. It is common for multiple individuals within an establishment to have the same job (e.g. elementary school teachers within a school/school district), which can result in fewer individual quotes for that establishment. Because the quote-level information is tied to the job, not the individual, sampling a certain number of jobs within an establishment is not equivalent to sampling a certain number of workers within an establishment.

Sample weights will be assigned to each quote and establishment to represent the entire frame. Sample weights were unavailable at the time this report was written, therefore, all numbers presented are unweighted.

Data collection

ORS data collection began in October 2014 and continued until May 2015. At the close of the data review process, there were 7,116 quotes collected from 1,851 establishments, slightly more than 4 jobs per establishment. These jobs spanned 22 of the 23 unique 2-digit SOCs and 704 unique 8-digit SOCs. Though there are 23 2-digit SOCs, one of these, military-specific occupations (55-0000), is out of scope for ORS. The 704 8-digit SOCs represent 63.4% of the 1,100 unique 8-digit SOCs.

A few caveats should be noted about the scope of occupational coverage. First, collection of a unique SOC means only that there is at least one job in the data that falls under that SOC; it should not be interpreted to mean that an estimate suitable for publication is available for the SOC using pre-production data. A second thing to note is that the employment covered by an individual SOC varies considerably by 8-digit code. Therefore, a count of unique SOCs (or the share of total SOCs) is a poor proxy for employment coverage. A better measure of scope would be the share of total employment covered by the unique SOCs collected in pre-production; however, this requires weighted estimates.

BLS measures collection response rate as the total number of usable schedules divided by the sum of refusals and usable schedules.(5) The collection response rate for the ORS pre-production test was 79.3%. This is comparable to the 78% response rate for establishments initiated in the most recent NCS sample.(6) Most ORS data elements have both a measure of presence and one of duration. For example, FEs ask whether a job involves kneeling as well as how much time is spent kneeling. For data elements that measure the presence of a job requirement, the average item response rate was 85%. When we include measures of duration as well as presence, the average item response rate was 80%. Generally, respondents were able to provide presence and duration of environmental conditions. For certain physical requirements, such as gross and fine manipulation, the response rates for presence were high, but respondents had difficulty providing data on duration, resulting in a relatively lower response rate for those items.

Interview duration was, on average, 63 minutes. Roughly half (49.4%) of schedules were conducted by personal visit and the balance by some other mode (telephone, email, mail, or a combination of approaches).

Duration tended to be longer, on average, for personal visit appointments (75.6 minutes) than for other modes (50 minutes), which is expected since personal visit appointments were more likely to occur for establishments with a higher number of sampled jobs.

The establishment respondents in the ORS pre-production test included human resource professionals/workforce planners, workers compensation staff, safety/occupational health/risk management staff, and subject matter experts (supervisors, incumbents, department heads). These respondents were similar to those who provided data during previous feasibility tests. The feedback from FEs in pre-production is that finding the right respondent is key to data collection and that respondents with knowledge of job descriptions tended to be most helpful.

The HR departments were often the starting point for finding respondents, for two reasons. First, for establishments that are currently part of the NCS sample, the FEs already have established contacts who are often in HR. Second, in order to perform the job sampling within an establishment, FEs first need information that is typically maintained in the HR department. The FEs reported that often the primary respondent would consult with others on staff for responses to data elements.

Procedures

There are a number of resources provided for FEs to ensure consistent procedures, collection, and coding of data elements. FEs are provided with a collection manual (and training on the contents of the manual) which covers such topics as securing cooperation from respondents, fundamentals/definitions of ORS pre-production data elements (along with practical examples), guidance on how elements might be related, and instructions on coding in the data capture system.

During collection, FEs may submit inquiries to resolve ambiguities that arise. Often these are one-off topics, such as whether walkie-talkies would fall under the category of phone communication. The procedures staff provides a response to the inquiry, which is then distributed to all FEs. There were 23 such responses provided during pre-production.

If there is evidence that additional clarification is needed on data elements (i.e. there exists a more general source of confusion), then the procedures staff highlights the correct procedure through a “procedures alert,” that is distributed to all FEs. An example of this during pre-production was an alert that provided additional information on “deviations from the norm” for mental and cognitive elements. During pre-production there were five procedures alerts issued.

Debriefs with the FEs were also used during the ORS pre-production test. These were an opportunity to share best practices, get feedback on respondent comprehension of data elements, and for FEs to highlight any problems they were having with collection. Feedback from these debriefs will inform FE training and the materials provided in the collection manual as preparations are made for ORS data collection for FY 2016.

ORS review program

The ORS pre-production data were subject to a review program modeled on the NCS that will also be used in production. The data review program has components that occur in the field as well as components that occur after the data are submitted to BLS headquarters.

The quality assurance process starts with the classroom and on-the-job training of newly hired FEs, which takes place over 9-12 months. The classroom training includes survey procedures, collection protocols, data capture and review systems, and interviewing techniques. Once basic training is completed, the certification process begins, which pairs a new FE with a certified FE who will observe data collection and review collected data.

Additional review of collected data includes: full schedule review, where all data elements for an establishment (also referred to as a “schedule”) are reviewed by experienced field staff; technical re-interview, where respondents are re-contacted and a subset of data elements are re-collected; and targeted schedule review, where a subset of elements are reviewed by staff at BLS headquarters. Roughly 15% of schedules are subject to full schedule review, 5% to technical re-interview, and 20% to targeted review.

The remaining 60% of schedules are subject only to secondary review, in which data elements flagged outside of the primary data capture system are reviewed. This secondary review focuses on verifying data elements that are unexpected or inconsistent with other data elements (e.g. physical elements that are unexpected given the occupation). This also encompasses review across schedules to identify schedules and elements with anomalies.

The review staff work with the FEs to ensure that all data are correctly coded and that any flagged elements have documentation verifying that the elements are correctly coded. Reviewers questioned approximately 1% of the data during ORS pre-production. When a field economist receives a question from a reviewer, the FE reviews the values coded in the data capture system, reviews their notes from the interview with the respondent, may review the procedures and collection manual, and may call back the respondent to clarify any information from the interview. Roughly 61% of the time these questions resulted in data being changed.

Changes in collection anticipated for production

Implementing a large-scale test of the ORS data elements has resulted in changes and refinements to several of the elements as the survey moves into production. These changes were made in consultation with the Social Security Administration and reflect the desire to collect data that meet the needs of SSA. What is most notable is that relatively few data elements are changing; the pre-production test has demonstrated that the definitions and wording of the majority of the data elements are working well in data collection.

There was only one change to the physical requirements elements. This involved rewording the question on climbing ramps or stairs to ensure that the climbing was work-related and not simply the result of someone, say, choosing to walk up stairs to a third floor office.

Among the environmental elements, there were two changes. The first consolidated the presence of hazardous contaminants into one data element rather than two. The second added an indicator for the presence of personal protective equipment for hazardous contaminants, moving mechanical parts, high exposed places, and noise intensity level.

All of the mental and cognitive elements are being revised for production. Highlights are provided below:

The question on task complexity has been removed. A question has been added, “what type of decision-making is required to perform the tasks of the occupation?”
The question on work controls has been removed. A question has been added, “what type of supervision does this occupation have?”
A question has been added on pace of the work performed.
A question has been added regarding what controls the pace of work (the work or the worker).
The question on work deviating from the norm has been removed. A question has been added, “how often do (work/schedule/location) change in this occupation?”
The questions on contacts have been revised to rescale the options for frequency of contacts.

Future testing plans

Two additional tests of data collection are planned for June-August 2015. The first is the ORS job observation pre-test. The job observation pre-test is intended to assess whether the data collected through ORS interview collection methods are systematically different than data collected through direct observation. This pre-test involves re-contact of a subset of establishments that were interviewed as part of the pre-production test. FEs will observe select jobs within the establishment and record data on physical and environmental data elements. This pre-test is in response to both Federal Register Notice public comments and an external subject matter expert’s recommendations for testing and validation of ORS survey design.(7) A written report of this test will be available in fall 2015.

The second test is a test of the revised mental and cognitive elements, to understand respondents’ perception of the wording of these questions. Data will be collected from establishments who participated in ORS feasibility tests. If necessary, the results from this test will be used to further refine the mental and cognitive elements after the first year of ORS data is collected.

Lessons learned and overall assessment

At this point, the results from the ORS pre-production test demonstrate that data on occupational requirements can be collected using the processes established by BLS, but final conclusions about the feasibility of these methods will be made when the analysis of pre-production data is complete. Training and procedures are designed for consistent data collection by field economists. The feedback channels between FEs and the procedures team ensure that questions are answered quickly, information is disseminated to all ORS staff, and any needed modifications to procedures are made and communicated effectively. The data review process will evolve as new procedures are implemented and the indicators used to identify inconsistencies in collected schedules are refined.

The response rate for ORS is comparable to that for the NCS and item level response rates provide preliminary evidence that usable data on job requirements are being collected. The elements that address the physical, environmental, and vocational preparation requirements (tested through feasibility tests in prior years) appear to be working well in pre-production collection and few changes are being made to those elements as BLS moves into production.

There are some aspects to collection that need to be addressed going forward. Training and procedures for ORS production should address best practices for identifying the proper respondent as well as strategies for dealing with establishments for which FEs are collecting both ORS and NCS data elements. Item level response rates indicate that some respondents are having difficulty providing duration data for some job requirements and tools and procedures to assist field economists in obtaining measures for those data elements are needed.

The mental and cognitive elements have largely been rewritten for production, which will require additional training and procedures that assist FEs in collecting these new elements from respondents. The test of mental and cognitive elements scheduled for summer 2015 should help shape this. Minor refinements may be made during the first year of production collection to mental and cognitive elements as part of the OMB clearance process for production collection. However, major changes cannot be made until the second year of collection. If the summer test and initial collection indicate further major changes to the mental and cognitive elements are needed, tests of the mental and cognitive elements will occur in FY2016.

1 The occupational classification system most typically used by BLS is the six-digit SOC (www.bls.gov/soc/), generally referred to as “detailed occupations”. O*NET uses a more detailed occupational taxonomy (www.onetcenter.org/taxonomy.html), classifying occupations at eight-digits and referring to these as “O*NET-SOC 2010 occupations”. There are 840 six-digit SOCs and 1,100 eight-digit SOCs.
2 Copies of earlier test reports can be found at: www.bls.gov/ors/research/collection/pdf/fy14-feasibility-test.pdf
3 Additional information on the survey design can be found as part of the Federal Register Notice, www.reginfo.gov/public/do/PRAViewDocument?ref_nbr=201403-1220-002.
4 Probability selection of occupations (PSO) is also used to select jobs in the NCS. Additional information on probability selection of occupations can be found in the BLS Handbook of Methods, www.bls.gov/opub/hom/ncs/design.htm.
5 The sum of refusals and usable schedules will not equal the size of the sample initially drawn. Establishments may have gone out of business or be classified as out of scope for the survey.
6 A “usable” quote for ORS is defined as a job for which at least one data element has been reported. For the Employment Cost Index component of the NCS, a usable quote is one for which information on wages is present. For both ORS and ECI, one usable quote makes a schedule usable.
7 The subject matter expert’s report, “Occupational Requirements Reliability and Validity Literature Research,” is available here: https://www.bls.gov/ors/research/collection/pdf/handel-methodological-issues-data-collection-full-report-feb15.pdf.

Last Modified Date: June 17, 2015