Handbook of Methods Occupational Requirements Survey History

Timeline Events:

Hover over the red dot to see historical information.

October 2012: Occupational Requirements Survey (ORS) established as a test survey

November 2012: Phase one test: Initial proof of concept

January 2013: Phase two test: Collection protocol testing

April 2013: Phase three test: Broad scale testing

November–December 2013: Observations test conducted concurrently with other fiscal year 2014 tests:^⁠1

• ORS-only efficiency innovations test
• Central Office Collection elements test
• National Compensation Survey (NCS)/ORS joint collection test
• New data element tests

February 2014: Alternative modes test

October 2014–September 2015: Preproduction testing (collection, review, estimation, and validation)

June 2015: Job observations pilot test

September 2015–August 2016: First sample group in the first wave collected

May 2016–July 2017: Second sample group in first wave collected

December 2016: Published 2016 reference year estimates from one sample group in the first wave

June 2017–September 2017: Job observations test

September 2017–July 2018: Third sample group in the first wave collected

• Narrowed the scope of collection. Initially field economists collected job requirements based on how workers generally performed job tasks. The survey scope was narrowed to include job requirements based on workers performing critical tasks in support of critical job functions
• Tested the revised cognitive and mental requirements

November 2017: Published 2017 reference year estimates which combined data from two sample groups in the first wave

February 2018–August 2018: Job observation test

September 2018–August 2019: Collected first sample group in the second wave

• Second wave includes five sample groups
• Updated occupational selection process from modelled occupational frame for private industry
• Survey measurement changes to include cognitive and mental requirements as well as low posture (kneeling, crouching, crawling, stooping) requirements

February 2019: Published 2018 reference year estimates which combined data from three sample groups in the first wave

May 2020: Published 2019 reference year estimates to include cognitive and mental requirements

August 2019–July 2020: Second sample group in the second wave collection

Background

The Social Security Administration (SSA) and Bureau of Labor Statistics (BLS) sign annual interagency agreements for BLS to produce estimates on occupational requirements. These estimates will assist the SSA in making accurate disability determinations and decisions. SSA also intends to use ORS estimates, along with information from other occupational sources, to create the Occupational Information System (OIS).

During the developmental stages of the ORS, BLS identified its existing infrastructure was already available to coordinate with the ORS. That framework had the capability to manage and implement a new survey to meet data needs as well as systems and processes to support all the steps of the survey. In addition, field economists who work on the NCS were already familiar with collecting data elements similar to those captured by the ORS. For example, the NCS program classifies each job selected using the Standard Occupational Classification System (SOC), collects worker characteristics (such as bargaining status and part-time or full-time workers), and determines industry classification using the North American Industry Classification System (NAICS) for sampled establishments. In addition, the BLS is collecting and publishing information on the knowledge required to perform the job, job controls provided, the complexity of tasks, the contacts made by workers, and the physical environment where the work is performed.^⁠2 After the initial assessment of whether BLS could collect job requirements, BLS began testing the collection of these requirements.

Preproduction testing

BLS established ORS as a test survey in FY 2013. In FY 2013 and 2014, several feasibility tests were performed to assess the viability of collecting data on occupational requirements using the platform used by the NCS.

In FY 2013, testing was conducted in three phases:

• The main objective of phase 1 was to ensure that BLS field economists knew how to describe the survey and find respondents for the ORS data elements. BLS also created and tested an initial set of data collection protocols and collection aides.
• In phase 2, BLS expanded the number of field economists that could describe and collect ORS data while obtaining additional information not included in phase 1. Phase 2 testing also evaluated the effectiveness of collection tools.
• The primary goal of phase 3 was to test whether field economists could collect ORS data elements and relevant information across the country in a uniform and efficient manner. Also during phase 3, BLS tested the feasibility of collecting both ORS and NCS elements, adding more ways to conduct ORS interviews, including new data capture systems and review procedures, and establishing the Central Office Collection (COC). Some companies have special arrangements with BLS regarding the manner in which data should be collected for their individual establishments. Therefore, a COC may require permission and coordination from headquarters in order to proceed with collecting data.

Test objectives were successfully met in these phases, and the findings from these tests suggested that the collection of the ORS data was viable.

As a result of FY 2013 testing, areas were identified where further testing was needed before moving to full- scale production. In FY 2014, five feasibility tests were completed to refine ORS methodology tested in previous phases:

1. The ORS-only efficiency innovations test refined the methods to develop more efficient approaches for data collection as identified during FY 2013testing.
2. The NCS/ORS joint collection test determined how to best collect occupational requirements data elements and NCS data elements from the same establishment.
3. The new data element tests determined the new cognitive and mental requirements work data elements and evaluated the use of occupational task lists as developed by the Department of Labor, Employment and Training Administration (ETA), and Occupational Information Network (O*NET) program during data collection.
4. The Central Office Collection (COC) test determined how best to collect occupational requirements data elements from large firms and state governments.
5. The alternative modes test determined how to collect occupational requirements data elements efficiently (such as via phone, email, or fax) when a personal visit is not possible.

These tests provided evidence that the NCS platform could be adapted to ORS data collection and demonstrated the effectiveness of the revised materials and procedures.

Testing activities in FY 2013 and 2014 laid the foundation for the preproduction test conducted in FY 2015. Unlike the earlier tests, which were small-scale, testing a subset of data elements or the viability of different collection methods, the preproduction test was designed as a relatively large-scale, nationally representative test of ORS data collection. The sampling, data collection, procedures, and review were designed to mimic what would occur during ORS production. The results from the ORS preproduction test demonstrated that data on occupational requirements could be collected using the processes established by BLS. As a result of the preproduction test, some changes and refinements to several of the elements were made before the implementation of a move to production.

Detailed information on completed tests and other testing activities can be found on ORS research page.

Change in scope of collection

Initially, BLS and SSA agreed to define the scope of collection as how work is "generally performed" in each establishment. This meant BLS collected requirements related to all aspects of work, including job functions that were incidental or not specific to one job and were unrelated to the primary hiring and pay factors of jobs.

Historically, SSA relied upon information from the Department of Labor's Dictionary of Occupational Titles (DOT) and its companion the Selected Characteristics of Occupations (SCO). The data from these publications appear to show a narrower scope for data collection. DOT data appears to show that analysts only rated work requirements that pertained to the hiring and pay factors of jobs.

Beginning with collection of the third sample group in the first wave, BLS took steps to revise current procedures to align more closely with a narrower scope of work that pertains to the hiring and pay factors of the job. The revised scope of work is limited to only tasks related to the "critical job function" (i.e., the reason the job exists). These tasks must be expected and usual, now defined as "critical."

In addition, the ORS program reviewed the measurements of duration for applicable job requirements, which included point measurements, respondent-determined ranges, and SSA-defined ranges. BLS determined that measurement of duration only in the SSA-defined ranges would more accurately reflect duration of job demands and incorporated this change during the first wave.

By adjusting the parameters of what work is included to only that of the critical functions of jobs and specifying duration ranges, the BLS expects to more accurately capture job requirements while still identifying changes in the way work is performed in the modern economy.

Cognitive and mental requirements

Estimates for the second wave, beginning with preliminary 2019 reference year estimates, include cognitive and mental requirements. Cognitive and mental requirements were initially included in estimates for the first wave (2016 and 2017 reference year estimates) but were discontinued while the ORS program refined concepts, conducted testing, developed procedures, trained staff, and updated the estimation system. The ORS program conducted feasibility testing to assess measures of cognitive and mental requirements that are understandable to employers and can be used to generate useful estimates, resulting in better defined estimate categories that can be used to measure work pace, ability to pause work, presence of supervisor, and many additional categories. (See the calculation section for a full listing of cognitive and mental estimate types.)