Simon Sheather, head of the Department of Statistics at Texas A&M University in College Station, Texas, is looking through row after row of airfare data—nearly 8 million of them. But he isn't planning a vacation. He's using the huge dataset to create a model that predicts ticket prices to help customers save money, based on the route they fly.
The increased amount of data in the world has created many opportunities for the kind of analysis Sheather does. Recent advances in technology, such as e-commerce, smart phones, and social networking, are generating new types of data on a scale never seen before—a phenomenon known as "big data." According to some data experts, 90 percent of the data that exists in the world today was created in the last 2 years. And society increasingly relies on data to tell us things about the world.
This year, 2013, is The International Year of Statistics. It's a designation intended to highlight the role that data and statistical analysis have in society. To further that goal, this article describes work with big data. The first section outlines what big data is. The second section provides an overview of big data work. The third section explains some of the challenges that big data work entails. The fourth section describes how to prepare for this work. Sources of information are provided at the end.
Download the PDF