Revisiting the dilemma of review for modeled wage estimates by job characteristic

In August 2013, Michael Lettau and Dee Zamora reported on a new estimation model developed by the Bureau of Labor Statistics (BLS) that combines data from the National Compensation Survey (NCS) and the Occupational Employment Statistics (OES) survey to produce wage estimates by area, occupation, and job characteristic.¹ Drawing on the large sample size of the OES survey and on data on job characteristics from the NCS, the model provides more extensive information about the wage rates of workers than either survey could provide individually. With about 800 occupations at the two- and six-digit Standard Occupational Classification (SOC) levels; more than 400 areas for the nation, states, and metropolitan areas; and 38 characteristics by full-time and part-time status, union and nonunion bargaining status, time-paid and incentive-paid status, and full-time and part-time status by work levels,² the model has the potential to compute millions of wage estimates. To move the production of these modeled wage estimates toward regular publication, a process is needed to review and validate this massive volume of estimates in order to determine which ones are fit for publication. The material that follows describes the approach to reviewing and validating the estimates, goes on to discuss the criteria for their publication, and culminates in the presentation of a selected set of wage estimates by area, occupation, and job characteristic for 2013.

Three criteria have been developed for whether an NCS–OES wage estimate for an area, an occupation, and a characteristic is eligible for publication: (1) there must be sufficient OES data to support a published estimate for the occupation in the area, (2) there must be sufficient NCS data for the area, occupation, and characteristic contributing to the wage estimate, and (3) the NCS–OES wage estimate for the area, occupation, and characteristic must fit broadly within expectations based on knowledge of compensation data.

With regard to the first criterion, the OES survey does not collect information on job characteristics, but it does publish wage estimates by area and occupation. To be consistent with these wage estimates, an NCS–OES wage estimate for an area, occupation, and characteristic is eligible for publication only if the OES wage estimate for the area and occupation was published. For example, the wage estimate for part-time stock clerks in the Nassau–Suffolk, NY, Metropolitan Division is eligible for publication only if the OES wage estimate for stock clerks in the Nassau–Suffolk, NY, Metropolitan Division was published.

With respect to the second criterion, as described in the earlier article, the NCS–OES estimation method calculates characteristic proportions (e.g., the proportion of part-time workers) from the NCS data. These proportions are used to allocate OES employment by characteristic, and the allocated OES employment counts by characteristic are then used to compute a characteristic mean wage for the area and occupation. Thus, the amount of NCS data for the area and occupation by characteristic is an important determinant of the reliability of the NCS–OES wage estimate.

The modeled estimates are calculated for the same areas as are published for the OES survey, namely, the nation, states, and metropolitan areas, such as Metropolitan Statistical Areas (MSAs) and Metropolitan Divisions. The characteristic proportions that are used in the calculations are based on 24 areas that the NCS uses (15 large metropolitan areas and the 9 Census Divisions, excluding the 15 large metropolitan areas), and these proportions are applied to all OES employment counts by occupation for establishments within the NCS area. Therefore, with regard to the second criterion, the amount of available NCS data is based on the number of NCS observations for the occupation, characteristic, and NCS area for which the characteristic proportions are calculated, rather than the area to which the NCS–OES wage estimates refer. For example, whether the wage estimate for part-time stock clerks in the Nassau–Suffolk, NY, Metropolitan Division is eligible for publication will depend on the number of NCS observations available for part-time stock clerks in the New York–Newark–Bridgeport, NY–NJ–CT–PA, Combined Statistical Area. The New York–Newark–Bridgeport, NY–NJ–CT–PA, Combined Statistical Area, which includes the Nassau–Suffolk, NY, Metropolitan Division, is one of the 24 areas used for the NCS characteristic proportions. Specifically, it is the area used to allocate employment for stock clerks in OES establishments from the Nassau–Suffolk, NY, Metropolitan Division.

Under the publication criteria that were used for the (discontinued) NCS wage program, estimates that met the criterion for enough NCS data to support a publishable estimate nearly always also had a sufficiently low relative standard error. Thus, the requirement of a sufficient amount of NCS data supporting the NCS–OES wage estimate will serve to some degree as a proxy for a criterion based on the yet-to-be-developed standard errors.

The third criterion for eligibility for publication uses an expectation calculated from a regression model. Originally, the model was developed to facilitate the review and validation of the very large number of wage estimates that the NCS–OES procedure generates. However, it is also used to determine directly whether a wage estimate is eligible for publication. In the regression model, the NCS–OES wage estimates by area, occupation, and characteristic are regressed on variables for the six-digit SOC, the detailed geographic area, the interaction of the characteristic and detailed geographic area, and the interaction of characteristic and broader occupational groupings. The model is fitted separately for the various characteristics that are estimated: union and nonunion, time and incentive, full time and part time, and full time and part time by work level. The regression uses NCS–OES employment estimates by area, occupation, and characteristic as weights.

The coefficients that the regression model estimates are then used to calculate an expectation for the wage rate for an area and occupation by job characteristic. NCS–OES wage estimates that deviate substantially from their expectation are defined as anomalous and are not eligible for publication. Of the wage estimates for MSAs by occupation and characteristic, about 3.4 percent differ from their expectation by more than 20 percent, the threshold currently applied. The estimates making up that 3.4 percent are deemed ineligible for publication.

After the publication criteria have been applied, the set of estimates that are eligible for publication is further checked to see whether it exhibits qualities that are inconsistent or counterintuitive. One validation report lists the estimates that differ greatly from year to year. Only three sets of annual estimates have been produced thus far. The year-to-year review will improve when a longer history of estimates has been established. Another validation report lists the relationship of the wage estimates among the types of characteristics. Because the NCS data are used to allocate the OES employment counts to the characteristics, the relationship among the types of characteristics in the NCS data is expected to hold as well in the NCS–OES estimates. For example, if the union wage estimate in the NCS data is higher than the nonunion wage estimate for an area and occupation, then it is assumed that this relationship will be carried through to the NCS–OES estimates. When the relationship of a type of characteristic is the opposite from what is seen in the NCS data (e.g., the nonunion wage is higher than the union wage), the estimates are identified and reviewed to determine why that is so. In addition, a validation report is produced which lists the estimates that narrowly passed the residual threshold criterion, so the reviewer can assess whether, all other things being equal, the estimate is within expectations and suitable for publication.

The review of the most recent estimates revealed additional reports for consideration during the validation process, including one that flags when the publishable part-time estimate is greater than the publishable full-time estimate. The expectation is that the full-time estimate is greater than the part-time estimate, but there are known cases (e.g., registered nurses) where this expectation is not always met. Another expectation, which may be captured in a future report, is that wage estimates almost always rise from job level to job level (i.e., as the job level increases, the mean wage almost always does, too). In addition, there is a need to identify what is thought to be an atypical characteristic for an occupation. For example, incentive and union employees are expected in certain occupations, but they also are seen in other occupations once it is learned what the circumstances are for the occupation or establishment. These special cases might be identified in a frequency report of occupations for a given characteristic.

Although no schedule has been established for the modeled estimates to appear in a recurring BLS publication, efforts are being made to develop such a publication. Standard errors also are being developed that would be published with the estimates and would add another criterion to determine eligibility for publication and assess the reliability of the estimates.

Wage estimates for May 2013

The earlier article presented wage estimates for registered nurses and general office clerks in selected metropolitan areas for May 2011. In the current article, tables 1 through 4 show a select set of NCS–OES wage estimates by job characteristic for May 2013. The estimates in these tables pass the three criteria for publication. Table 1 shows the national wage estimates for the detailed occupations within office and administrative support occupations, a major occupation group, by union, nonunion, full-time, and part-time worker characteristics. These detailed occupations were selected on the basis of the preponderance of publishable estimates. Table 2 shows estimates for full-time and part-time workers by work level for the same occupations. These estimates are published for 15 levels based on a point system of four factors: knowledge, job controls and complexity, contacts, and physical environment. The work level is determined by the sum of the points for the four factors.

Tables 3 and 4 show wage estimates for stock clerks and order fillers (SOC 43-5081). In table 3, wage estimates for union, nonunion, full-time, and part-time workers are shown for selected metropolitan areas. In table 4, wage estimates are shown for full-time and part-time stock clerks and order fillers by work level for the same metropolitan areas. Tables 5 through 7 present wage estimates by bargaining status, time and incentive status, and full-time and part-time status for three MSAs with large employment: the New York–White Plains–Wayne, NY–NJ, Metropolitan Division; Los Angeles–Long Beach–Glendale, CA, Metropolitan Division; and Chicago–Joliet–Naperville, IL, Metropolitan Division, respectively. Finally, tables 8 through 10 provide information on full-time and part-time workers by work level for these same three MSAs, respectively. Data for which the work level could not be determined are indicated in the estimate “Not able to be leveled.” Together with estimates of identified work levels that also were published, estimates that were “not able to be leveled” may help form a more complete picture of the distribution of wage rates for the associated area and occupation.