Opportunities and Challenges for using Big Administrative Data

David M. Talan

Abstract

The demand for information, insights and data is increasing in every segment of society. The ability to respond to ever increasing demand forces government statistical agencies to search for new opportunities and new methods. In these days of instantaneous demands, waiting a few years to build new data sources, investing valuable staff to the new development, then waiting for a sufficient time series to build for analysis, is usually an insufficient response to the original demand. If there is any other way to gain sufficient insight into a problem without waiting for several years to get a few answers to basic items, data users will gravitate to other, immediately available sources. The Bureau of Labor Statistics’ Quarterly Census of Employment and Wages program produces a dataset that some consider Big Data. There is no standard agreed upon definition of Big Data, but it is often defined as a data set with the following dimensions: volume, velocity, variety, and veracity. While the QCEW does not have the velocity of some other datasets, it certainly does have the volume, variety, and veracity that characterize Big Data. This paper describes the opportunities and challenges for using Big Data at BLS through its Business Register. We present some opportunities for developing new products such as recently released hurricane zone data, and employment and wage estimates for non-profits; and also prospects for employment estimates from foreign direct investment and other opportunities. We also present some challenges of using big data that include statistical, legal, and technical infrastructure issues. We also present the evolving nature of Big Data and describe how it may yield promising areas of future development.