United States Department of Labor
I recently spoke in Pittsburgh at the 2015 Policy Summit on Housing, Human Capital, and Inequality. The Federal Reserve Banks of Cleveland, Philadelphia, and Richmond sponsored this event. I spoke on a panel with Professor Charles Manski of Northwestern University and Jeffrey Kling of the Congressional Budget Office about measuring uncertainty in federal statistics. You can watch the full discussion below.
When I speak to groups around the country or write in the Commissioner’s Corner, I always discuss the importance of having good information to make good decisions. Federal, state, and local policymakers use information from BLS, and so do private businesses, nonprofit organizations, and households. But how do the users of our data and analyses know they can rely on BLS information? Our users shouldn’t simply have blind faith. After all, households, businesses, and governments make decisions based on our data, and those decisions can involve a lot of money. Users of statistics need to understand that all measures have limitations. Data are a tool. Just like screwdrivers or spatulas, data have specific uses and different levels of precision. Data users need to choose the right tools for their purpose and use them correctly. Our goal is to measure the true state of the economy, but data users must recognize that all measures of the truth come with some uncertainty.
So what are the sources of uncertainty in our measures? One source is what we call sampling error. Most statistics we publish at BLS come from sample surveys. Sampling error is the uncertainty that results by chance because we collect the information from a sample instead of the full population. Even though we select our samples carefully using scientific methods, the characteristics of a sample still may differ from those of the population. We rely on sample surveys because it is far too expensive to ask questions of all workers or all businesses every time we need new information about the labor market and economy. Fortunately, statisticians have developed tools to measure sampling error. We publish these measures on our website. For example, you can see whether the most recent monthly changes in our measures of the labor force, employment, and unemployment are statistically significant. If we want to reduce sampling error, we can increase the size of our samples. Larger samples cost more money, but our measures of sampling error can help us decide whether the benefit of reducing that source of uncertainty is worth the cost.
Other types of uncertainty are harder to measure. For example, some people and businesses choose not to respond to our surveys. If those who don’t respond have different characteristics from those who respond, it could bias our measures. Even when people and businesses agree to participate in a survey, they might not answer every question or their answers might not be accurate. It’s hard to measure the effects of these challenges in collecting information about the economy. We try to minimize the sources of uncertainty, however. For example, we try to design our surveys to make it easier for people and businesses to respond. We show people and businesses how they benefit from responding. We test our survey questionnaires carefully to make sure they are clear and easy to answer. We seek out other sources of information to supplement our surveys, using what many people call “big data.”
Most of all, we communicate with our data users about the strengths and limitations of our data and the methods we use to compile them. We’re always looking for better, clearer ways to explain our data, and I welcome you to share your ideas.