Appendix B: Reliability of subnational estimates derived from the Current Population Survey
The estimates published in Geographic Profile of Employment and Unemployment (GP) are based upon a sample of the population rather than a complete count. Therefore, they may differ from the figures that would have been obtained if it had been possible to take a complete census using the same questionnaire and procedures that are used in the Current Population Survey (CPS). There are two types of errors in an estimate based on a sample survey: sampling error and nonsampling error. Tables B-2 through B-5 indicate the magnitude of the sampling error.
Sampling variability. The standard error is primarily a measure of sampling variability—that is, the variation that occurs by chance because a sample rather than the entire population is surveyed. The sample estimate and its standard error enable one to construct confidence intervals: ranges that would include the average result of all possible samples with a known probability. For example, if all possible samples were selected, each of these samples were surveyed under essentially the same conditions by use of the same sample design, and an estimate and its estimated standard error were calculated from each sample, then the following would occur:
- Approximately 68 percent of the intervals from 1 standard error below the estimate to 1 standard error above the estimate would include the average result of all possible samples.
- Approximately 90 percent of the intervals from 1.645 standard errors below the estimate to 1.645 standard errors above the estimate would include the average result of all possible samples.
- Approximately 95 percent of the intervals from 2 standard errors below the estimate to 2 standard errors above the estimate would include the average result of all possible samples.
The error of a sample estimate varies inversely with the size of the sample and directly with the size of the estimate. Hence, an estimate for a subgroup constituting a small proportion of a population will tend to have a larger error relative to its size than will an estimate for a larger subgroup.
Reliability standards
The CPS sample design takes into consideration both national and state reliability. For the state data, a minimum reliability standard is set: an expected maximum coefficient of variation (CV) on the level of total unemployment of 8 percent annually. This CV is calculated with the assumption of a 6-percent unemployment rate. Because each state's sample design must meet the reliability standard, the CPS sampling rate differs by state. (The sampling rate is the proportion of all households that are selected for the sample.) Generally, the smaller the state population, the higher is the sampling rate.
Publication standards for state CPS data
To achieve comparability of the data for regions, divisions, and states, for publication purposes, a unique requirement for minimum levels for the labor force and for employment and unemployment was developed for each area. This requirement is based on the known differences in sampling rates among these areas. Before estimates are published for a specific category (such as Hispanic unemployment in a particular state), a predetermined "critical cell" must meet a 50-percent CV requirement. As a result of this requirement, minimum bases for publication have been developed for each area. Table B-1 lists the minimum necessary base for publication of data in each of the regions, divisions, and states.
Estimates are not shown when they do not meet the minimum base for area listed in table B-1. In tables showing the labor force status of the population—that is, the number of employed and unemployed—publishability is determined by whether the labor force level exceeds the minimum base for unemployment in table B-1. If the labor force level is less than the unemployment minimum base, all data—labor force, employment, unemployment, and unemployment rate—are suppressed. In all other tables, the determining factor is whether the size of the base of the distribution exceeds the minimum base for employment or unemployment separately, depending on whether the table presents a distribution of employment or unemployment for the area or population subgroup. Data are not published for any cell with a level of less than 500 people or less than 0.05 percent of the total for a given characteristic.
Using the sampling error tables
Tables B-2 through B-5 provide sampling errors for use in constructing 90-percent confidence intervals (approximately 1.645 standard errors) for major labor force characteristics. The sampling errors provided are approximations and thus indicate the order of magnitude of the sampling error rather than the precise amount of the possible error in an estimate. Illustrations on the use of these tables are provided next. In all cases, the computations present the estimated levels in thousands of people.
Sampling error of an estimated number. Table B-5 shows that an estimate of 50,000 unemployed people in Massachusetts will have an absolute sampling error of 10,000, for a relative sampling error of 20 percent (10,000/50,000). In comparison, an estimate of 100,000 unemployed people in Massachusetts has an absolute sampling error of 14,000, yielding a relative sampling error of 14 percent (14,000/100,000). A statement that unemployment for a particular group is between 40,000 and 60,000 in the first instance, and between 86,000 and 114,000 in the second, can be made with approximately 90-percent confidence.
The latter statement can be interpreted as follows: if one were to draw all possible samples, make an estimate from each sample (using the same methods and techniques), and construct an interval around each estimate (with the sampling errors shown in the tables), then 90 percent of the intervals would contain the average value of all possible samples.
To convert a sampling error from 90-percent confidence, as displayed in the tables, to 68-percent confidence (1 standard error), multiply the sampling error shown in the tables by 0.63. To convert the sampling error from 90-percent to 95-percent confidence (approximately 2 standard errors), multiply the sampling error by 1.23. For the example given, the sampling error at 90-percent confidence is 10,000. At 68-percent confidence, the error would be about 6,300 (10,000 × 0.63). At 95-percent confidence, the error would be about 12,300 (10,000 × 1.23).
Sampling error of a difference. To compute the error of a difference from the tables, an additional step is required. If, for instance, one wishes to know whether a change in the unemployment rate from one year to the next in a particular area for a particular population group is statistically significant or whether the difference in the unemployment rate between two areas or population groups is statistically meaningful, the significance of the difference needs to be computed. (Differences between estimates for 2 consecutive years may be influenced to some extent by a redesign of the CPS concepts, questionnaire, and collection procedures, such as the one that occurred in 1994.)
As noted before, differences can take two general forms: (1) differences between population groups and/or geographic areas, and (2) differences for the same population group and geographic area over time. Either type of difference can be calculated with the following formula, noting the limiting covariance assumption discussed later:
SEd = [( SE12 + SE22 ) – 2C × ( SE1 × SE2 )]1/2.
In this equation:
- SEd = the sampling error of the difference,
- SE1 = the sampling error of one group or year,
- SE2 = the sampling error of another group or year, and
- C = the covariance (or relationship) term.
The SE1 and SE2 can be found in the appropriate table of Geographic Profile for each year if the comparison is between different years, because the size of the samples and, consequently, sampling errors may differ from year to year. Values for the covariance, or "C" term, for employment and unemployment for differences between consecutive years are as follows: for labor force or employment levels, C = 0.58; for unemployment levels or rates, C = 0.37. It is important to note that these C terms are usable only for calculating the sampling error of a difference for over-the-year change for the same geographic area and population group.
Covariance terms for the relationship between different population groups or geographic areas in this bulletin are not available. In calculating sampling errors for differences between two different population groups or geographic areas, a C term of zero must be assumed. The effect of this assumption is that (1) if the relationship between two groups, areas, or years (differences for nonconsecutive years) is small, then the C term can legitimately be ignored and the sampling errors will not be adversely affected, and (2) if there is a strong positive relationship between the two groups, areas, or years (differences for consecutive years), then the error computed without a C term will be overstated. An overstatement could lead one to state that a difference or change was not statistically significant when, in fact, it was. When there is a strong relationship over time for a characteristic such as employment (people tend to remain employed from one year to the next), the importance of using a C term to calculate the sampling error of a difference over time increases greatly.
The next example illustrates how to calculate the sampling error of a difference. Suppose one wished to know whether a hypothetical difference between an unemployment level of 250,000 for a particular population group in California and an unemployment level of 200,000 for the same group in New York was statistically significant at 90-percent confidence. Table B-5 gives the error for an unemployment level of 250,000 in California as approximately 27,000 and the error for an unemployment level of 200,000 in New York as 26,000. Using the formula described previously without the C term produces the following results (levels in thousands):
SE1 = 27; SE2 = 26;
SE12 + SE22 = 1,405;
SEd = ( SE12 + SE22 )1/2 = 37.
Because each state's sample is independent, there is no measurable correlation between the two estimates, and a C term of zero can be assumed. Thus, the error of the difference is approximately 37,000. Because the actual difference (50,000) is greater than the error of the difference, it can be stated with 90-percent confidence that the difference in the unemployment level is attributable to factors other than sampling variability alone.
Sampling errors for unemployment rates. Unemployment rates and error ranges for these rates are provided in tables 1 and 14. This information can be used to derive a sampling error for an unemployment rate if one is needed. The error range is a 90-percent confidence interval around the unemployment rate. By subtracting the estimated unemployment rate from the upper bound of the range (or subtracting the lower bound of the range from the estimated unemployment rate), the sampling error for the rate can be obtained. This sampling error can then be used in the formula given previously for computing the sampling error of a difference, or for any other purpose the user chooses.
Interpolation and extrapolation. Although sampling errors are listed for selected levels of employment and unemployment in tables B-2 through B-5, users may wish to know the sampling error for an estimate whose value is not listed. To derive such a sampling error, it is necessary to use interpolation or extrapolation.
For example, in order to derive the sampling error for the 2023 total unemployment level for women in Arizona, it is necessary to use interpolation because table B-5 contains no sampling error for an unemployment-level estimate of 57,000. The following formula and accompanying example show how to interpolate for this estimate:
SE = {[(A-G) / (F-G)] × (X-Y)} + Y.
In this equation,
- SE = the sampling error for the estimated value,
- A = the estimated value (57,000),
- F = the table value (100,000) immediately above the estimated value,
- G = the table value (50,000) immediately below the estimated value,
- X = the sampling error of F (17,000), and
- Y = the sampling error of G (12,000).
Thus (levels in thousands),
- SE = {[(57 – 50) / (100 - 50)] × (17 – 12)} + 12
- SE = ( 0.14 × 5 ) + 12
- SE = 0.70 + 12
- SE = 13
If the sample-based estimate lies outside the boundaries of the error tables, extrapolation can be used to approximate the sampling error. The formula for extrapolation is the same as that for interpolation; however, the F term becomes the highest value in the table and the G term becomes the next-highest value.
Derivation of sampling errors
The state and area sampling errors are developed with a generalized regression procedure and are not based on sample data for each individual area, population group, or labor force characteristic. As with all sampling error tables produced for CPS subnational data, a number of approximations are required in order to derive sampling errors that apply to a wide variety of items. As a result, these sampling errors indicate the order of magnitude of the error rather than a precise error for any specific item. The sampling error tables are derived from standard error equations and special parameters developed by the Bureau of Labor Statistics. These parameters are available upon request from the Division of Local Area Unemployment Statistics.
United States Department of Labor
