An official website of the United States government
The Bureau of Labor Statistics (BLS) is working with the Social Security Administration (SSA) to establish the Occupational Requirements Survey (ORS), an establishment survey that collects information on the vocational preparation and the cognitive and physical requirements of occupations in the U.S. economy, as well as the environmental conditions in which those occupations are performed. In preparation for the first ORS production sample, we studied several potential sample designs and assessed each for practicality and coverage of the population. In addition, we considered how and if the ORS sample might be coordinated with the sample of BLS’s National Compensation Survey (NCS). In this paper, we evaluate each sample design option, describe the sample design that was selected for the first ORS production sample, and outline plans for monitoring the effectiveness of the selected sample design.
Keywords: survey design, sample allocation, sample selection
In the summer of 2012, the Social Security Administration (SSA) and the Bureau of Labor Statistics (BLS) signed an interagency agreement, which has been updated annually, to begin the process of testing the collection of data on occupations. As a result, the Occupational Requirements Survey (ORS) was established as a test survey in late 2012. The goal of ORS is to collect and publish occupational information that will replace the outdated data currently used by SSA. More information on the background of ORS can be found in the next section. All ORS products will be made public for use by non-profits, employment agencies, state or federal agencies, the disability community, and other stakeholders.
An ORS interviewer attempts to collect close to 70 data elements related to the occupational requirements of a job. The following four groups of information will be collected:
Section 2 of this paper provides additional background information on the ORS. Prior to selecting the first ORS sample, which will go into collection in fall 2015, several sample design options were studied and compared. Section 3 discusses the major sample design options that were considered, including the results of simulations of each of the options. Section 4 presents the sample design that will be used for the first ORS production sample and explains why it was chosen. Section 5 describes several approaches taken to estimate the number of occupations for which ORS data will be published under the selected sample design. Section 6 describes a modification to the design for the first production sample. Finally, section 7 outlines plans for using data collected from the first few ORS samples to evaluate the selected sample design.
In addition to providing Social Security benefits to retirees and survivors, the Social Security Administration (SSA) administers two large disability programs which provide benefit payments to millions of beneficiaries each year. Determinations for adult disability applicants are based on a five-step process that evaluates the capabilities of the worker, the requirements of their past work, and their ability to perform other work in the U.S. economy. In some cases, if an applicant is denied disability benefits, SSA policy requires adjudicators to document the decision by citing examples of jobs the claimant can still perform despite restrictions (such as limited ability to balance, stand, or carry objects) [1].
For over 50 years, the Social Security Administration has turned to the Department of Labor's Dictionary of Occupational Titles (DOT) [2]as its primary source of occupational information to process the disability claims. SSA has incorporated many DOT conventions into their disability regulations. However, the DOT was last updated in its entirety in the late 1970’s, and a partial update was completed in 1991. Consequently, the SSA adjudicators who make the disability decisions must continue to refer to an increasingly outdated resource because it remains the most compatible with their statutory mandate and is the best source of data at this time.
When an applicant is denied SSA benefits, SSA must sometimes document the decision by citing examples of jobs that the claimant can still perform, despite their functional limitations. However, since the DOT has not been updated for so long, there are some jobs in the American economy that are not even represented in the DOT, and other jobs, in fact many often-cited jobs, no longer exist in large numbers in the American economy.
SSA has investigated numerous alternative data sources for the DOT such as adapting the Employment and Training Administration’s Occupational Information Network (O*NET) [3], using the BLS Occupational Employment Statistics program (OES) [4], and developing their own survey. But they were not successful with any of those potential data sources and turned to the National Compensation Survey program at the Bureau of Labor Statistics [5].
The NCS has a fully-developed survey platform available for potential coordination with the ORS. This platform includes its management structure, collection staff, and survey procedures, as well as its sample of establishments and occupations. Coordination with the NCS sample, in particular, might provide efficient data collection opportunities for the ORS. At the same time, the NCS is an established survey that includes the Employer Cost Index, a principal federal economic indicator, so the ORS sample design cannot be allowed to harm the NCS sample or its outputs. Therefore, we have studied two main categories of sample designs: designs in which the ORS and NCS samples are coordinated for sampling, and designs in which the ORS sample is selected independently from the NCS sample.
In a coordinated sample design, the NCS sample would be selected as a subsample of the larger ORS sample. All establishments selected for the NCS sample would also be included in the ORS. The evaluation of coordinated ORS and NCS sample designs is described in detail by Ferguson et al. in “Occupational Requirements Survey Sample Design Evaluation” [6].
The research described in Ferguson et al. showed that coordinated sampling was possible but might be difficult to implement. Multiple sample simulations were carried out with the NCS sample selected as a subsample of the ORS sample. It was a challenge to maintain NCS sample design with this approach because the preferred sample distribution for each survey differs. The NCS sample is allocated among 120 cells that are defined by 5 aggregate industries and 24 geographic areas. The measure of size for each frame unit is adjusted so that sample units are distributed among 24 major private sector industries in certain proportions. In other words, there are effectively 576 cells for which NCS needs sufficient frame units. NCS’s industry proportions, however, differ from the industry proportions chosen for the ORS sample. Since the ORS sample is projected to be almost three times larger than the NCS sample, it was assumed that the NCS sample should be a subsample of the ORS sample in a coordinated design. But it was found to be difficult to apportion the ORS sample in a way that maintains its own targeted industry proportions while providing an adequate frame for the NCS subsample.
The most promising coordinated sample design required a change to the NCS sample cells. With 24 detailed industry sample cells, implicitly stratified by geographic area, the ORS sample and NCS subsample were both able to achieve overall target employment and sample size goals. Slight shifts to NCS sample by geographic area were noted but not considered a serious problem. The sample design also has the advantage of being straightforward and easy to explain. However, changing the NCS sample design to this degree would require resources that are not currently available, making this sample design difficult to implement before the first production sample is needed.
Having studied a range of coordinated sample designs, we investigate independent sample designs in more detail in this paper. With an independent sample design, the effect of the ORS sample design on the NCS sample design is not a factor, allowing the ORS more freedom to tailor its sample design to its needs. To test the effectiveness and utility of each design, we followed procedures similar to those used for coordinated designs in Ferguson et al. [6].
To test each sample design in private industry and to allow comparability, we took a recent sample frame – the same frame used to test the coordinated designs – and selected 150 ORS samples from that frame. We then used those samples to calculate average sample sizes and weighted employment totals in various categories of interest. For private industry, we also selected 150 NCS samples from the same frame, according to the NCS sample design. We calculated the average amount of overlap between each ORS sample and its corresponding NCS sample.
For the state and local government sectors of the economy, the NCS sample is selected less frequently, so we already have access to the NCS government sample that will be in collection during the first few years of ORS production, as well as the frame from which it was selected. Therefore, we selected the ORS simulation samples from this frame and compared the ORS government simulation samples to the actual NCS sample.
Since we were not limited by the NCS sample cells, we used 23 detailed industry strata as the ORS sample cells in private industry. We used a proportional to employment allocation method to distribute the total sample size among the 23 private industry ORS sample cells. Note that the NCS adjusts its measures of size to achieve targeted sample sizes in these 23 industries, but the sample sizes are not guaranteed and can vary by a few units more or less. See Appendix A for a list of the industry sampling cells for private industry samples.
In government we used 10 detailed industry strata as the ORS sample cells. NCS sample cells use 5 aggregate industries and 24 geographic areas for a total of 120 sampling cells. For NCS, the 5 aggregate industries are implicitly stratified by the 10 detailed industries used for ORS. NCS and ORS both allocate the government sample proportional to employment. The ORS industry sampling cells for state and local industry samples are also shown in Appendix A.
3.2.1 Maximum Overlap Independent Design
We tested two types of independent sample designs. The first type was a maximum overlap independent design, in which the ORS sample is selected separately from the NCS sample, but ORS data is collected from all NCS units. The total ORS sample comprises the independently selected ORS-only sample and the independently selected NCS sample.
To allocate the ORS-only sample using the maximum overlap design, the composition of the NCS sample must be taken into account even though it is independent from the ORS sample. This is not a concern in the government sample because both ORS and NCS are allocated proportional to employment in similar cells. In private industry, however, NCS sample sizes for the ORS sample cells are not guaranteed. The NCS sample must be selected, not just allocated, prior to allocating the ORS sample, so that the exact number of NCS units in each ORS sample cell is known. Then the total ORS sample size, including the NCS units, is allocated proportional to employment across the 24 sample cells. The sample cell allocations are reconciled with the NCS sample counts; if the NCS sample size is greater than the allocation for a sample cell, the ORS allocation is increased to meet the NCS sample size and the remaining sample is reallocated among the other cells. The reconciliation process continues until each total ORS cell allocation is at least as large as the NCS sample size for the cell. The NCS sample sizes are then subtracted from the total ORS allocations to create the ORS-only sample allocations.
Two ORS sample cells, Finance (Rest of) and Insurance, required increases to their original allocations due to the size of the NCS sample in the cells, as shown in Table 1. Because the NCS sample sizes in these cells were so large, all of their ORS sample units came from the NCS sample, and there were no ORS-only units. On the other end of the range, only 9% of the Health Care, Social Assistance (Rest of) cell came from NCS. Overall, NCS units accounted for 38% of the total private industry ORS sample.
ORS Sample Cell (Detailed Industry) | Percent Change from Original to Final Allocation | Percent of Cell from NCS |
---|---|---|
Health Care, Social Assistance (Rest of) | -1% | 9% |
Accommodation and Food Services | -1% | 15% |
Finance (Rest of) | 13% | 100% |
Insurance | 14% | 100% |
... | ... | ... |
Total ORS Sample | 0% | 38% |
After the ORS-only sample allocations were determined, the ORS-only sample was selected using systematic probability proportional to employment selection. The total ORS sample clearly overlapped the NCS sample for the 38% that comes from the NCS private industry sample (the proportion was nearly the same in government). However, because the ORS and NCS samples were independently selected from the same frame, there was the possibility of additional, unintentional overlap between the samples. These units would be collected in any case, since ORS data is collected from all NCS units under the maximum overlap design. However, the unintentional overlap reduces the yield of unique ORS data that can be collected.
Table 2 shows that approximately 1.4% of a single year’s ORS-only sample overlapped the NCS private industry sample. ORS estimates will ideally be based upon data collected from several years of sample, and Table 2 shows that when a three-year span of samples is considered, the percentage of overlapping units rose to 3.3%. The three-year overlap percentage was not three times the single year’s overlap percentage because there was overlap from year to year within a survey. For the ORS-only portion of the private industry sample, about 6.9% of units overlapped with another ORS-only unit in one (or both) of the other years in the three-year span of samples.
The government sample size for the ORS is about 15% of the total planned ORS sample size, which is proportional to the share of government employment out of all employment (government and private industry). However, government establishments have an average of 80 employees per establishment, while private industry establishments have an average of 14 employees per establishment. It follows that there were proportionally fewer establishments from which to select the government sample, compared to the private industry sample. Some establishments had particularly large employment, making it likely that they would be selected for any sample, ORS or NCS. As a result, the amount of overlap was higher for the government sample. Close to a quarter of ORS-only units overlapped with an NCS government unit in any single sample group, and about a third overlapped with NCS after three years of sample were combined. The amount of overlap between ORS samples within a three year span was also relatively high at nearly 30%.
Span | Type of Overlap | Private Industry | State & Local Government |
---|---|---|---|
One year (one sample group) | ORS to NCS | 1.4% | 23.1% |
Three years (one estimation group) | ORS to NCS | 3.3% | 33.1% |
Three years (one estimation group) | ORS to NCS | 6.9% | 29.2% |
3.2.2 Minimum Overlap Independent Design
The second type of independent sample design that we studied was a minimum overlap independent design. In such a design, the ORS sample is selected separately from the NCS sample, and ORS data is collected only from units selected in the ORS sample. For this design, the ORS simulation sample size was allocated across the 24 private industry sample cells and 10 government sample cells strictly proportional to employment. There were no adjustments or other complications, so the resulting simulation sample sizes and weighted employment totals met all expectations.
Under the minimum overlap design, the entire ORS sample is an ORS-only sample. Since the ORS would ideally obtain data from the same number of observations regardless of design, the ORS-only sample under the minimum overlap design was about 62% larger than the ORS-only sample under the maximum overlap design. As a result, though there was no intentional overlap between the ORS and the NCS, there was more unintentional overlap between ORS-only and NCS units under the minimum overlap design. Table 3 shows that 1.8% of a single year’s private industry ORS sample overlapped the NCS sample, and 3.7% overlapped over a three-year span. These overlaps do not decrease the amount of unique ORS data that is collected since the NCS sample is not part of the total ORS sample. However, these overlaps do increase respondent burden for the overlapping NCS units.
The amount of unintentional overlap between ORS-only units across years was slightly lower under the minimum overlap design than under the maximum overlap design. In private industry, it was 0.2 percentage points lower, and in government it was 5.3 percentage points lower. This means that there would be a somewhat larger amount of unique ORS data collected.
Span | Type of Overlap | Private Industry | State & Local Government |
---|---|---|---|
One year (one sample group) | ORS to NCS | 1.8% | 17.2% |
Three years (one estimation group) | ORS to NCS | 3.7% | 29% |
Three years (one estimation group) | ORS to NCS | 6.2% | 24.2% |
There was one coordinated sample design that met the sample-related needs of both the ORS and the NCS. Also, there were maximum and minimum overlap independent sample designs that resulted in satisfactory samples. Any of the three approaches could be used to meet the sample-related needs of both the ORS and the NCS. Therefore, the choice of a final sample design hinged upon other considerations: flexibility, staffing, cost, respondent burden, and response.
Despite two years of survey field testing, it remains difficult to fully predict the experience of a full production collection cycle. Many features of the ORS sample – such as total size, industry distribution, and collection timing – require modifications during the first few years of ORS production. Flexibility, therefore, is essential. On the other hand, changing the sample selection process for the NCS, a stable and well-established survey, must be undertaken with care and deliberation. To balance these competing requirements, it was first decided that the ORS would implement an independent, rather than coordinated, sample design.
The next decision was to determine the desired amount of overlap between the collection of the ORS and NCS samples. As discussed in section 3, there would be some overlap between the two samples regardless of whether a maximum or minimum overlap design was chosen, but the amount of intentional overlap can be controlled. Past field testing showed that collecting ORS data from NCS units produced some efficiencies. Established relationships with NCS respondents could be used to encourage ORS response. Overall staffing needs and associated costs were lowered because the collection of administrative data and certain data elements common to both surveys could be combined. For the same reasons, the aggregate collection time of a given amount of ORS and NCS data was lower than if no units had overlapped, potentially allowing a shorter collection cycle. Aggregate respondent burden was also lower, as fewer respondents were contacted to collect the same amount of data across the two surveys.
On the other hand, it was found that the data collection process for a combined ORS and NCS unit was lengthy, especially for units with a large number of occupational observations. Large establishments can have eight or more occupations. The collection of ORS items alone for four occupations has typically taken about an hour [7]. A combined ORS/NCS sample unit would also involve the collection of all NCS elements: wages, hours, and the availability, cost, and provisions of various employee benefits. As a result, individual respondent burden is increased, which could discourage existing NCS respondents from responding.
The potential danger posed by additional individual respondent burden is the largest concern because it could affect the amount of response and quality of collected data for both the ORS and the NCS. It was determined that the best way to mitigate this concern is to limit overlap between the two surveys and implement the minimal overlap independent sample design. This design will be implemented initially for both private industry and government, but it was noted that the independent sample design would allow one or the other ownership group to modify the amount of NCS overlap in the future. In particular, response rates for government units are historically high for the NCS. Given the smaller number of available units, unintentional overlap will be higher for the government sector than for private industry, so it might make sense to increase intentional government overlap at some point.
A key feature of ORS estimates is the focus on detailed occupational information at the 8-digit Standard Occupational Classification (SOC) level. Because of this, it is important to collect a large number of unique observations for as many detailed occupations as possible. The sample design affects how many occupations will have enough data for publishable estimates.
The sample frame available to ORS contains only establishment-level information and does not have occupational data. Therefore, in our selected sample design, industry distribution is being considered as a proxy for occupational distribution, and we are assuming that occupational variety is roughly proportional to the number of employees in an industry. It is difficult to gauge the accuracy of this assumption before collecting occupational data under the design, but we have done three studies to get an idea of how many occupations ORS will be able to publish. We have studied the former NCS National Wage publication, the current NCS estimation sample, and the Occupational Employment Statistics (OES) sample.
Until 2011, the NCS program produced a publication of national wage estimates covering private industry and state and local government. The estimates were based on a probability proportional to employment sample of more than 35,000 establishments, making it similar in industry composition and size to the planned ORS sample. Occupations were selected using the same probability selection method (PSO) that will be used for the ORS and were classified using the Standard Occupational Classification (SOC) codes [9]. For this study, NCS wage establishment response rates are assumed to be comparable to response rates in the ORS; while wages are more sensitive than most ORS data, a larger quantity of data will be requested from an ORS respondent than from an NCS wage respondent. Unlike the ORS sample, the NCS wage sample was not a national sample but an area-based sample; data were collected from 227 localities, and the unit weights were adjusted with an area factor so that the sample represented the nation. Also, six-digit SOC was the most detailed occupational level for which data were collected and published. Nevertheless, its similarities to the ORS sample make the NCS national wage publication a good source for gauging how much occupational data might be available from the ORS sample.
Wage estimates were published at various levels of detail, from the broad occupation (two-digit SOC) down to the detailed occupation (six-digit SOC). To be publishable, an estimate for an occupation had to be based on data from at least three unique responding companies (by Employer Identification Number, or EIN) and six unique occupational observations; its relative standard error had to be less than 50%; and it had to pass a P-percent dominance test. These criteria are less stringent than the ORS criteria are likely to be since we have less experience with ORS data and will be more cautious with the initial publication criteria until BLS has enough experience to determine if the less stringent NCS wage criteria are sufficient for protecting ORS respondent confidentiality. Even with these cautions, we believe the wage data are a good starting point for assessing how much data might be available.
We counted the number of occupation-specific estimates that were published in the All Civilian wage table (covering private industry and government combined) [8]. Table 4 shows that over two-thirds of all possible occupations were published, and over 90% of all possible occupations were published at the 2-digit and 5-digit SOC level. (Note that three-digit SOC was used only to group more detailed published occupations, so only a small number of the eligible occupations at that level were published even though many more could have been published.) At the most detailed level, six-digit SOC, almost three-quarters of possible occupations were published.
Occupational Level | Percent Published |
---|---|
2 Digit SOC | 95.65% |
3 Digit SOC | 11.25% |
4 Digit SOC | ... |
5 Digit SOC | 92.57% |
6 Digit SOC | 72.35% |
All Levels | 71.61% |
* Four-digit SOC is the same as three-digit SOC.
The NCS sample used in estimation in 2014 included about 10,000 establishments. It covered both private industry and state and local government units, in roughly the same proportions as are planned for the ORS. The NCS sample is being changed from an area-based sample, similar to the national wage sample in section 5.1, to a national sample akin to the ORS. The 2014 NCS estimation sample was a combination of several annual sample groups and included both area-based sample groups and national sample groups.
As discussed earlier, the current NCS sample is selected probability proportional to employment with certain target industry proportions, which results in a different distribution of establishments by industry from the ORS. (The NCS national wage sample discussed in section 5.1 did not have these target industry proportions.) Occupations were selected using the same probability selection method (PSO) that will be used for the ORS. Response rates by industry are known and, for the purpose of this study, assumed to be similar to those that will be experienced in the ORS, because collection procedures and staff will be largely the same.
We counted the number of unique occupations found in the NCS sample by various levels of SOC detail. Also, we counted the number of unique responding establishments by EIN within each SOC level. We applied an ownership-based factor to each count to adjust for the smaller size of NCS sample compared to the planned ORS sample size of 30,000. We determined the proportion of all possible occupations that met several EIN and occupation criteria levels.
Tables 5 and 6 show that at the most detailed SOC levels, the amount of available data quickly falls off as more stringent criteria are added. Requiring 3 establishments and 6 occupations yields about 80% of possible six-digit SOCs for both private industry and government, which is slightly higher than the results from the NCS wage publication. Requiring 15 establishments and 30 occupations drops the private industry yield to 53%. As might be expected from the smaller government sample, its yield at this level is much lower, at 37%; in fact, government yields for all but the most lenient criteria and broad occupations are quite a bit lower than the corresponding private industry results.
Publishability Criteria | Percent of Private Industry Occupations That Are Publishable (by SOC detail level) | |||
---|---|---|---|---|
6-digit | 5-digit | 3-digit | 2-digit | |
3 establishments, 6 occupations | 80% | 85% | 90% | 100% |
15 establishments, 30 occupations | 53% | 67% | 82% | 100% |
15 establishments, 60 occupations | 38% | 56% | 80% | 100% |
15 establishments, 90 occupations | 30% | 46% | 75% | 95% |
15 establishments, 120 occupations | 25% | 38% | 70% | 95% |
Publishability Criteria | Percent of State & Local Government Occupations That Are Publishable (by SOC detail level) | |||
---|---|---|---|---|
6-digit | 5-digit | 3-digit | 2-digit | |
3 establishments, 6 occupations | 81% | 83% | 92% | 100% |
15 establishments, 30 occupations | 37% | 47% | 74% | 95% |
15 establishments, 60 occupations | 23% | 31% | 66% | 95% |
15 establishments, 90 occupations | 16% | 24% | 60% | 95% |
15 establishments, 120 occupations | 11% | 18% | 49% | 95% |
The Occupational Employment Statistics (OES) program is a BLS survey program that produces wage estimates for over 800 occupations [10]. With OES cooperation, we were able to access a version of the 2011 private industry OES sample that the NCS program had used for previous research. The OES sample had three years of data covering over 1 million establishments and included data for all occupations in each of the establishments, including the 6-digit SOC code and number of employees for each occupation. The earlier NCS researchers had amplified the sample to produce a quasi-frame of establishments with occupation information.
Since we began with a full private industry (quasi) frame, we were able to apply the selected ORS sample design and select a private industry sample of 25,500 establishments (the private industry portion of the planned ORS sample size of 30,000). We stratified by 23 detailed industries, combining the aircraft manufacturing and all other manufacturing industries that had been separate in the studies described in section 3 because the first is small subset of the other and is not a particular focus for the ORS. The sample was allocated proportional to industry employment. Establishments were selected by systematic probability proportional to size selection with employment used as the measure of size. Occupation selection followed the planned ORS protocol as closely as possible. The number of quotes selected was based on establishment size, and occupations were selected using systematic probability proportional to size selection with the occupational employment used as the measure of size.
Each frame establishment was considered to be unique, though there were no EINs available to verify this assumption. The number of unique responding quotes was counted at several SOC code levels. The actual counts were multiplied by 0.75 to simulate non-response.
Table 7 shows ORS simulation results that are similar to the results from the private industry 2014 NCS estimation sample (similar establishment thresholds could not be used because of differences in how establishments were identified). The OES-based percentages are a bit lower for the most stringent criteria at the most detailed SOC levels; for example, requiring 120 occupations for a six-digit SOC yields 16% of possible occupations from the OES sample compared to 25% from the NCS sample. The general patterns hold, though, so the occupation projection studies provided a rough but consistent idea of the amount of data to expect from the ORS.
Publishability Criteria | Percent of Private Industry Occupations That Are Publishable | |||
---|---|---|---|---|
6-digit SOC | 5-digit SOC | 3-digit SOC | 2-digit SOC | |
6 occupations | 78% | 89% | 94% | 96% |
30 occupations | 40% | 60% | 89% | 96% |
60 occupations | 26% | 42% | 83% | 93% |
90 occupations | 20% | 34% | 77% | 91% |
120 occupations | 16% | 29% | 68% | 91% |
In addition to the ORS-only establishments selected in year one using the minimum overlap independent design described above, a supplemental set of establishments from the current NCS private sample will be included for collection of the ORS data elements. The modification is being made to increase the amount of data that can be published from the first production sample given the amount of trained and available field collection staff. The additional units are all active for the NCS but will be rotating out of the NCS sample after October of 2015. The NCS design at the time that these units were selected was a three-stage stratified design with probability proportionate to size selection used at each level. The first stage was the selection of 152 areas with the second stage being the selection of establishments within each selected area. The final stage was the selection of occupations within sampled establishments. For more details on this design see “Evaluating Sample Design Issues in the National Compensation Survey” by Ferguson et al. [11].
Since the two parts of the year one sample are selected independent of each other and each has weights that represent the entire frame, weight adjustment factors will be applied to all sample units in order to properly represent the frame. The data from this full sample will be used in all estimates generated from the first ORS production sample. Following the year one sample, all subsequent ORS samples will be selected independent of the NCS sample and will not be supplemented by the NCS.
The first production sample for the ORS has been selected according to the selected sample design described in sections 4 and 6. Data collection will begin in the fall of 2015. The following year’s sample will need to be selected before all data have been collected from the first sample, so we will assess the sample design at intervals throughout collection as well as at the end of collection.
We have assumed for our work so far that establishment (unit) response rates will be similar to those in the NCS, but the different subject matter could lead to different response rates. The first production sample will allow us to determine whether the sample is less efficient in certain industry strata or establishment size classes, which could lead to refining strata definitions and/or shifting sample. We will also monitor response rates of individual ORS elements (item response) to see if the data yield per establishment differs by industry or size class in a way that could be mitigated by the sample design. As part of our detailed response analysis, we will evaluate the potential for non-response bias in the ORS survey, focusing on the ORS data elements with the lowest item response rates. Similar response analysis will be evaluated for the 2015 pre-production test sample that was collected earlier this year. However, the pre-production test sample was selected using a different design in order to meet the test needs of that sample [6]. In addition, staff were just beginning to learn how to collect ORS data, so response rates from the test might not accurately reflect the response that will be obtained during production.
The number and distribution of responding units will contribute to the amount of data the ORS collects for detailed occupations. We will monitor the amount of usable data that we collect by occupational SOC code as well as the industries in which SOC codes are concentrated. Our studies so far have relied upon the six-digit SOC occupation as the most detailed occupation, but the ORS data will allow us to study occupational distribution at the more detailed eight-digit SOC level. We will observe how many estimates can be published at various levels of detail and also assess standard errors to determine whether more sample is needed from certain strata to ensure estimate reliability.
Researchers and members of the public have commented that it may be necessary to update ORS data for certain occupations more frequently than for others occupations. The sample designs studied so far do not allow for this flexibility, but we have identified and will evaluate additional sample designs that could allow such flexibility.
Minor changes to the design could be implemented quickly, perhaps as soon as the second production sample. Other observations might lead to more extensive research that could result in sample design changes further in the future. We will continually gather and assess data throughout the collection of all ORS samples.
[1] | Social Security Administration, Occupational Information System Project, www.ssa.gov/disabilityresearch/occupational_info_systems.html |
[2] | U.S. Department of Labor, Employment and Training Administration (1991), “Dictionary of Occupational Titles, Fourth Edition, Revised 1991” |
[3] | U.S. Department of Labor, O*Net Online, www.onetonline.org. |
[4] | U.S. Bureau of Labor Statistics (2008) BLS Handbook of Methods, Occupational Employment Statistics, Chapter 3. www.bls.gov/opub/hom/pdf/homch3.pdf. |
[5] | U.S. Bureau of Labor Statistics (2013) BLS Handbook of Methods, National Compensation Measures, Chapter 8. www.bls.gov/opub/hom/pdf/homch8.pdf. |
[6] | Ferguson, Gwyn R., Ponikowski, Chester, and Coleman, Joan. 2014. Occupational Requirements Survey Sample Design Evaluation. In JSM Proceedings, Government Statistics Section. Alexandria, VA: American Statistical Association.2924-2938. |
[7] | U.S. Bureau of Labor Statistics (2014), “Occupational Requirements Survey Consolidated Feasibility Tests Summary Report: Fiscal Year 2014.” www.bls.gov/ors/research/collection/pdf/fy14-feasibility-test.pdf. |
[8] | National Compensation Survey: Occupational Earnings in the United States, 2010, U.S. Department of Labor, U.S. Bureau of Labor Statistics, May 2011, Bulletin 2753, www.bls.gov/ncs/ncswage2010.htm. |
[9] | U.S. Department of Labor, Bureau of Labor Statistics, Standard Occupational Classification System, www.bls.gov/soc. |
[10] | U.S. Bureau of Labor Statistics (2013) BLS Handbook of Methods, Occupational Employment Statistics, Chapter 3. www.bls.gov/opub/hom/pdf/homch3.pdf. |
[11] | Ferguson, Gwyn R., Ponikowski, Chester, and Coleman, Joan. 2010. Evaluating Sample Design Issues in the National Compensation Survey. In JSM Proceedings, Survey Research Methods Section. Alexandria, VA: American Statistical Association.2838-2850. |
Last Modified Date: December 10, 2015