An official website of the United States government
Reliability of JOLTS EstimatesConfidence Intervals and Comparing JOLTS EstimatesAll sample estimates have inherent variability. The true population value estimated from a sample may be lower or higher than the sample point estimate. Comparing sample estimates to determine if they are different in terms of statistical significance requires the use of a statistical measure called the standard error of the estimate. Using standard errors allows the data user to probabilistically determine if the difference between sample estimates is within the bound of natural variability or if the difference exceeds the bound of natural variability. In other words, the data user can determine if the difference between sample estimates is considered statistically significant. Sampling and Nonsampling ErrorAll sample estimates, including the JOLTS estimates, are subject to both sampling and nonsampling error. When a sample rather than the entire population is surveyed, there is a chance that the sample estimates may differ from the "true" population values they represent. The exact difference, or sampling error, varies depending on the particular sample selected. Sampling error is a generic term for standard error. Sample estimates also are affected by nonsampling error. Nonsampling error can occur for many reasons, including the failure to include a segment of the population, the inability to obtain data from all units in the sample, the inability or unwillingness of respondents to provide data on a timely basis, mistakes made by respondents, errors made in the collection or processing of the data, and sampling and nonsampling error in the independent population controls of employment (also known as benchmark employment) data used in JOLTS estimation. This nonsampling error may be partially reflected in the standard error of an estimate. For more information on nonsampling error and benchmarking, see the JOLTS Handbook of Methods. Significance testing is a technique used to compare estimates using the estimates' standard errors. The BLS significance testing is generally conducted at the 90% level of confidence. That means that there is a 90% chance, or level of confidence, that an estimate based on the sample will differ by no more than 1.645 standard errors from the population value. The 1.645 is the t-distribution critical value used to construct a 90% confidence interval. For a 95% confidence interval, the value is 1.96. Confidence IntervalsA confidence interval is an interval around an estimate created by a lower bound and an upper bound. The lower bound is calculated by subtracting from the estimate the product of the estimate's standard error and the appropriate t-distribution critical value. As mentioned above, the t- distribution critical value is 1.645 for a 90% confidence interval. That same product is added to the estimate to calculate the upper bound of the confidence interval. Data users can compare two or more estimates by creating confidence intervals around each estimate and determining if the intervals overlap. If the intervals overlap, the two estimates are not different from each other in a statistically significant way; if the intervals do not overlap, the two estimates are different from each other in a statistically significant way. JOLTS publishes results of significance testing for monthly changes (over-the-month change) and for annual changes (over- the-year change) since these are the most common comparisons likely to be made by JOLTS data users. These results are accessible on the JOLTS Supplemental Table of Contents page. There are two ways to test for significance using JOLTS products:
Using the Significant Change TablesThe JOLTS program publishes significant change tables that evaluate over-the-month change and over-the-year change. The significant change tables provide the minimum change necessary in order for a change to be statistically significant. In the following example, the minimum significant change values from the significant change tables are used to determine whether the monthly (over-the-month) change is statistically significant. Note that the process described below is identical for rates and levels.
In the following example, the minimum significant change values from the significant change tables are used to determine whether the annual (over-the-year) change is statistically significant.
Using the Median Standard ErrorsThe significant change tables can be used to evaluate over-the-month and over-the-year changes. If the user wants to evaluate something other than over-the-month or over-the-year changes, then the user will need the median standard error of the estimate rather than the median standard error of the over-the-month change or the median standard error of the over-the-year change. The median standard error tables are available on the JOLTS median standard errors page. These tables contain standard errors for the level and rate of each data element (job openings, hires, etc.) for each published industry. To determine whether the change between estimates is statistically significant using median standard errors it is necessary to construct a confidence interval around the estimates being compared. To construct the confidence interval for a given estimate, follow these steps:
The process above must be repeated for each estimate that the data user is comparing. In the following example, confidence intervals are calculated for the seasonally adjusted total private job openings level for January 2013 and June 2013. From those two intervals, a determination of whether the change between the two time periods is statistically significant may be made.
From the bounds calculated above, we can say with 90% confidence that the population value for the January 2013 seasonally adjusted job openings level is between 3,038,388 and 3,349,612.
We can say with 90% confidence that the population value for the June 2013 seasonally adjusted job openings level is between 3,378,388 and 3,689,612. Since the 90% confidence interval for the January 2013 seasonally adjusted total private job openings (3,038,388–3,349,612) does not overlap the 90% confidence interval for the June 2013 seasonally adjusted total private job openings (3,378,388–3,689,612), it can be concluded that the change in seasonally adjusted job openings between January 2013 and June 2013 is statistically significant at the 90% confidence level.
Last Modified Date: January 10, 2020 |