Labor markets in 2040: big data could be a big deal for jobseekers

In 1990, the first successful connection between a server and a Web browser was established. Twenty-five years later, 27,000 gigabytes, roughly the content of 121 million books, of Web traffic is communicated every second. Practically all these transactions and exchanges of information are recorded—resulting in a massive amount of information on individuals’ online behavior.

Big data is already being used in healthcare delivery to help build a “learning” healthcare system where effective practices are identified from clinical data and then disseminated back to providers. It is being used to improve business sales and marketing, education delivery, national security, and law enforcement. It is even being used to improve traffic congestion, emergency response, and energy efficiency. These efforts merely scratch the surface of possibility. Over the next 25 years, we have the opportunity to transform big data into actionable labor market intelligence for business and jobseekers alike.

Big data is already affecting the labor exchange by matching qualified job candidates with employment opportunities and by aligning education and skill development with the needs of the economy. Education, work experience, and interests can all be connected to compile an accurate “qualifications profile,” which can then be aligned with the skills and experience employers need. As this capacity evolves, it will vastly improve the efficiency with which we match jobseekers to jobs.

We also have the opportunity to improve educational and career guidance by improving the alignment between an individual’s interests, talents, and personality and the educational requirements of any number of education and employment options. We can provide better information to colleges and universities, policymakers, and career counselors. Large sets of administrative records are now being linked across previously siloed governmental entities to provide measures of employment outcomes, employment projections, and job flows that were not available just a few years ago.

Just last year at the Minnesota Department of Employment and Economic Development, we unveiled our Graduate Outcomes data tool, for the first time linking postsecondary graduate outcomes to wage records in the state. This tool allows users to sort employment and wage outcomes by region, institution type, and major field of study. This tool—made possible by big data—is important for students, counselors, and education program planners.

Despite the many ways in which big data can inform and improve decision making, it’s important to recognize the risks. The use of government administrative records raises privacy, security, and confidentiality concerns. This has proven to be the greatest challenge to more rapid governmental progress with big data and merits careful consideration of future efforts. While these challenges can be overcome with new legislation, new technology, and continued vigilance, data security must remain an ongoing concern.

Private sector competition in data products and analysis will continue to put pressure on public sector progress and move data practices toward secure but open availability. The Billion Prices Project, the Google Unemployment Index, and ADP’s payroll employment estimates all compete directly with public sector data products. As this real-time private sector data continues to improve, it also means survey-based methods will become less valuable in the marketplace.

However, there are advantages to the use of administrative data for research, especially when linked to other datasets of interest. These data allow a level of detail and granularity not possible from much smaller representative sample surveys. Administrative data are often collected frequently, even in real time, and thus can provide more immediate measurement than surveys can. But using these data can be more time consuming and labor intensive; before drawing conclusions, researchers need to fully define and understand the population that the data represent and discover any existing holes in the data.

There is much more work to be done before the potential of big data is fully realized. Even beyond security concerns, we must develop techniques to properly apply these data to the questions at hand and even change the way we think about what questions to ask altogether.

But recent progress and the promise of future big-data solutions leave little doubt that 25 years from now we’ll have a hard time imagining life without big data, as we now have a hard time recalling life before the Internet.