What Is BLS Doing to Maintain Data Quality as Response Rates Decline?

Friday, March 24, 2023

Surveys are the backbone of the federal statistical community. BLS surveys are largely voluntary. A lot of time and effort goes into the delicate balance of gathering as much information as possible from survey respondents while minimizing the overall burden on respondents, so they (you!) are more likely to answer our questions.

Our statistics must meet rigorous standards before we can publish them. We are constantly assessing the quality of our data. We look at measures of variance and study potential bias to judge whether the results represent the populations they are designed to represent. We also ensure any statistics we publish don’t inadvertently identify our survey respondents, who have provided information under a legally binding confidentiality pledge. Another measure we examine closely when assessing the quality of our data is the survey response rate. The response rate is the number of units (businesses and other organizations, households, or people) that responded to a survey as a proportion of the number of units we asked to respond to the survey.

Historically, survey response rates have been used as a measure of how well a survey represents the sampled population. There are many reasons why response rates differ among surveys. These reasons include the amount of time available to collect data, constraints on who can respond, and the method of data collection (for example, in person versus over the phone).

Response rates to most federal surveys have been declining for many years. The decline accelerated during the COVID-19 pandemic as household and business patterns were greatly interrupted. For example, responses from our surveys of businesses declined as key individuals who previously responded were no longer onsite but working from home. BLS and other agencies had to pivot quickly to operate successfully in the new hybrid work environment.

BLS has tackled the problem of declining response rates from many angles. The method of collection has changed in some surveys. For example, instead of collecting data from businesses in person, BLS field economists are now using video to collect data in several BLS surveys, including the National Compensation Survey and our surveys of producer prices and import and export prices. In the Consumer Price Index (CPI), some prices previously collected in person are now collected over the phone, from respondents’ websites, or with web-scraping tools. BLS also extended the time respondents remain in the sample and in some cases increased the size of the sample. For example, the National Compensation Survey sample size increased from approximately 3,220 establishments before the COVID-19 pandemic to 4,700. Similarly, the Occupational Requirements Survey sample increased from 10,000 to 15,000 establishments. Larger samples enable us to obtain more data and reduce the impact of declining response rates.

In addition to shifting the data collection methods and increasing the sampling time and size, BLS accelerated its research into other data sources. For example, with the release of June 2021 indexes, the CPI program uses a secondary data source for gasoline prices, rather than traditional in-person or website collection. We don’t implement such changes without significant research. Much care is taken to ensure that BLS and other agencies can maintain the high quality of the data. As household and business patterns shift, especially because of the COVID-19 pandemic, BLS will continue to look for opportunities to blend directly collected data with other data sources.

Maintaining survey response is important, but it is just one way of ensuring the quality of BLS data. All data collection activities incorporate quality assurance assessments, including reinterviews with data providers, reviews by senior staff, computer-based data checks, and more. Information about the quality of published data is available to users through measures of variance, bias studies, and detailed descriptions of survey methods. Details on these and other steps BLS takes to measure and ensure data quality can be found throughout the BLS website, including in the Handbook of Methods.

We continue to monitor response rates, assess the quality of our data, and take steps to improve data collection. This is an exciting time to be in the world of data science because the blending of data sources may yield new types of data products that help us understand the economy. We will forge ahead in this frontier with our same statistical rigor to ensure we provide our nation with gold-standard data to help customers make better decisions.