The first hundred years of the Consumer Price Index: a methodological and political history
From businesses to government agencies to senior citizens, groups with often competing aims and desires use the Consumer Price Index. In attempting to satisfy their disparate needs, the Bureau of Labor Statistics frequently is challenged to produce a statistic that is both timely and accurate. This technical and political history explains both how and why the Bureau has come to produce a family of Consumer Price Indexes to address the challenge.
Of all the economic statistics produced by the U.S. federal government, none has a direct impact on the lives of everyday Americans quite like the Consumer Price Index (CPI). Numerous government programs, such as Social Security benefits, are adjusted each year on the basis of changes to the CPI. Countless contracts—whether business agreements, government obligations, leases, or court orders—also utilize the CPI, to adjust the dollar amounts associated with these settlements. For some, the CPI seems to be a rather difficult and abstract thing to understand. Others view the index with suspicion, a statistic produced by the recondite, esoteric labors of government economists and statisticians.
The truth, however, is that the ideas, history, and workings of the CPI are neither too difficult nor too secretive for the layperson to comprehend. This article presents a history of the creation and evolution of the CPI: a history of both how the Bureau of Labor Statistics (BLS, the Bureau) has gone about measuring the change in the cost of purchasing some mix of consumer goods and services and how the CPI has been used over approximately the previous 100 years. The story related will demonstrate that the CPI has never existed in isolation of the day’s events. Rather, events of the time almost always demanded active engagement from the Bureau, and input from business groups, unions, and the public has influenced the evolution of the CPI in important ways. For those who are already knowledgeable about current CPI methods, the article recounts the backdrop to recent and past methodological decisions, providing the why along with the how pertaining to those decisions. Interweaving the technical history of the CPI with the context of the broader political economy elucidates a far more compelling account than does covering the methodological history in isolation.
The article tells the history of the CPI in seven short, self-contained minihistories. The story begins in the late 19th century, proceeds through World War I, the New Deal, World War II, the postwar era, and the 1960s and 1970s, and closes with events that took place from the 1980s through 2004. Throughout it all, the Bureau is seen to be a responsive, often proactive, sometimes passive agency that established and still holds forth the CPI as a vital, evolving statistic.
The ingredients for a “cost of living” index
The precursor to the modern CPI began with data published in 1919 for 32 major shipbuilding and industrial centers. The data were estimated to go back to 1913; an index for the United States was first published in 1921. The fact that the Bureau was able to estimate data back to 1913 suggests that the collection of data on retail prices and consumer expenditures began far earlier than the publication of information on shipbuilding and industrial centers in 1919. Indeed, the groundwork that laid the foundation for the modern CPI began nearly at the very beginning of the establishment, in 1884, of a federal bureau to “collect information upon the subject of labor, its relation to capital, the hours of labor, and the earnings of laboring men and women, and the means of promoting their material, social, intellectual, and moral prosperity.”1
The first major study conducted by the new Bureau of Labor (later to be renamed the Bureau of Labor Statistics) that influenced later work on a “cost of living” index was an examination of family expenditures and retail prices between 1888 and 1890. At the time, the federal government was accumulating large budget surpluses, and both Democrats and Republicans identified tariff policy as the preferred tool for reducing the surpluses. Tariffs had been set particularly high to pay off the large debt that accrued during the Civil War. Democrats proposed reducing tariff rates in order to reduce tariff revenue, whereas Republicans postulated that increasing tariff rates would restrict imports, thereby causing a reduction in tariff revenue. Congressional Republicans eventually won the debate and passed the Tariff Act of 1890, commonly called the McKinley Tariff, which increased the average tariff level from 38 percent to 49.5 percent. Concerned with the effect that this new tariff law would have on the cost of production in key industrial sectors, Congress requested the Bureau of Labor to conduct studies on wages, prices, and hours of work in the iron and steel, coal, textile, and glass industries.
As explained in Joseph P. Goldberg and William T. Moye’s The First Hundred Years of the Bureau of Labor Statistics, the “cost of living” at this time “referred to family expenditures, and thus the [cost-of-living] study sought to reflect the standard of living supported by the actual levels of family income.”2 From 1888 to 1890, expenditure data were collected from 8,544 families associated with the aforesaid industries. Retail prices were collected on “215 commodities, including 67 food items, in 70 localities,” from May 1889 to September 1891.3
These studies were significant because the data collected served as the source for what was likely the first application of indexing techniques to economic data collected by a U.S. federal statistical agency. Senator Nelson Aldrich, chairman of the Senate Finance Committee, hired Roland Falkner of the University of Pennsylvania to study the economic impacts of the new tariff policy as well. Falkner used the data collected by the Bureau of Labor to create weighted wholesale and retail price indexes for inclusion in the reports submitted to the Senate Finance Committee concerning the effects of the McKinley Tariff on the economy. It was remarked that the statistical methods and indexing techniques employed in the reports produced by the Bureau of Labor and Roland Falkner “constitute the most valuable contribution to the history of American economic conditions that has yet appeared.”4
About a decade after the Aldrich studies, the Bureau of Labor sought to create its own retail price index. Previous periods of economic misfortune and drastic changes in economic conditions, particularly a general increase in worldwide prices in 1896, initiated public debates on “living wages” and the “cost of living.”5 Bureau of Labor Commissioner Carroll Wright was repeatedly called upon by Presidents Grover Cleveland and Theodore Roosevelt not only to personally mediate labor disputes between industry and union leaders, but also to use the Bureau’s maturing statistical and survey expertise so that the agency could become the principal factfinder in investigations into strikes. One notable example, which provided the Bureau with further experience in collecting and indexing retail prices, was the use of the Anthracite Coal Strike Commission’s measure of the change in food prices in the anthracite coal region of Pennsylvania to award a wage increase to the striking United Mine Workers of America in eastern Pennsylvania in 1903.
After successfully fulfilling special requests for smaller, industry-specific statistical studies for the Congress and the President, the Bureau endeavored to conduct a comprehensive study of the condition of working families throughout the country. A survey of family expenditures from 1901 to 1903 was the first step in constructing a comprehensive index of retail prices. Bureau agents surveyed 25,440 families that were headed by a wage earner or salaried worker earning no more than $1,200 annually in major industrial centers in 33 states; with inclusivity in mind, the Bureau included African American and foreign-born families in its survey. Agents collected 1 year’s data on food, rent, insurance, taxes, books and newspapers, and other personal expenditures. Using data on the income and expenditures of 2,500 families, the Bureau derived expenditure weights, particularly for principal food items, from this study.6
The second step in the construction of the retail price index was the collection of retail prices on the goods reported in the expenditure survey. The Bureau collected these data from 800 retail merchants in localities which were representative of the data collected in that survey. The prices collected spanned the years from 1890 to 1903.7
The 3-year study culminated in the publication of a price index called the Relative Retail Price of Food, Weighted According to the Average Family Consumption, 1890 to 1902 (base of 1890–1899),8 the first weighted retail price index calculated and published by the Bureau of Labor. The index included monthly quotations for the average prices and relative prices (averages weighted by consumption) of 30 principal food items. The Bureau expanded this survey to include 1,000 retail outlets in 40 states, and the index ran through 1907 before ceasing publication.9
Criticism from labor organizations concerning conclusions reached about the purchasing power of wage earners’ wages in a July 1904 bulletin demonstrates that, early on, the Bureau of Labor committed itself to an unbiased assessment of the data it collected and analyzed. Comparing wage data from 1890 to 1904 with retail price data covering the same period, the Bureau concluded, “taking 1903, it is seen that hourly wages were 16.3 percent above the average of 1890 to 1899, while retail prices of food were 10.3 [percent]; making the increase in purchasing power of the hourly wage, 5.4 percent.”10 The Bureau was sharply criticized for its analysis because of the industrial and social unrest present at the time due to the banking panic and crisis of 1903, which led to layoffs, a decline in wages, and a reduction in purchasing power.11 Many of the same labor organizations that praised the Bureau for its work in providing relevant data during labor disputes—data which often led to favorable decisions for laborers, such as the decision in the Anthracite Coal Strike—now accused the Bureau of employing faulty methodology or giving way to political pressure (e.g., 1904 was an election year).
Although the public’s demand that the federal government address social and labor unrest caused by drastically higher prices in the late 19th century seemed to solidify the prominence of the Bureau’s nascent retail price index as a national economic statistic, countervailing economic conditions in the early 20th century, as well as political decisions by the Roosevelt administration, made the future of the Bureau’s pricing program anything but secure. With a shrinking budget leading President Roosevelt to demand that the Bureau shift focus to studying progressive reform topics, Commissioner Charles Neill decided that the retail pricing program could not continue within its extant scope while maintaining the degree of accuracy necessary for the data to be reliable. After curtailing the retail pricing program by having it collect and publish prices only in alternate, non-election years, Commissioner Neill temporarily ceased operation of the program in 1907.12 The program would resume in 1911, but only as a limited version of the earlier work completed.
The creation of a “cost of living” index
When Royal Meeker succeeded Charles Neill and became the third commissioner of the Bureau in 1913, the retail food price index’s scope was diminutive compared with its scope in earlier periods, because of the cost-cutting measures that had been implemented. The resumed index covered 39 cities in 32 states, in contrast to the 70 localities it had covered in 40 states at its peak. The number of food items priced had been reduced from 30 to 15. The most significant change in methodology was the collection of prices from retailers by mail-in questionnaires, rather than through onsite visits by BLS agents.
From the onset of his administration, Meeker was candid about the reservations he had about the quality of the retail price program. In 1915, he stated, to a group of economists and statisticians, “I had become very suspicious of the Bureau’s index numbers, especially its retail price index. Some people here present will no doubt recall that I was wont to have fun with the Bureau’s index numbers, I no longer have fun with them—they have fun with me.”13 Indeed, a revision of the retail price index was one of Meeker’s primary concerns, as he later recalled: “Before I had got settled in the saddle, I set about to revise and recalculate the index numbers published by the Bureau.”14
The index that resulted from Meeker’s reforms encompassed 46 cities and incorporated an expanded selection of food items; an increase of 13 categories, including 28 food items; the addition of eight new cloth and clothing items; and the creation of the new category of anthracite and bituminous coal and gas for domestic use. This expansion served both to improve the accuracy of the retail index and to represent the changes in workers’ living costs more completely. However, with mobilization for World War I escalating, industrial and labor management demands quickly led to the recognition that even the more expanded retail price index could not accurately represent the ever-increasing costs of goods and services in industrial centers throughout the United States.
The first wartime request for a study of the cost of living came in 1917 from the Shipbuilding Labor Adjustment Board, which was formed only shortly before the request, with the dual mission of informing itself of the cost of living in the many shipbuilding districts subject to its jurisdiction and tracking changes in the cost of living as time progressed. Commissioner Meeker worked quickly and earnestly to secure funding to complete the study and ultimately received $75,000 (approximately $1.35 million in 2012 dollars) from President Wilson to complete the surveys in 18 major shipbuilding centers.15 The board used the results of the study to set uniform national wage rates for various shipbuilding trades.
Because of the time constraints of the study—the surveys were conducted over the early months of 1918—national labor arbitrators and the Wilson administration acknowledged that the study was far too limited to serve as the key component in an indexing scheme to standardize and stabilize national wages. For example, the study still used the weights derived from the 1901 study of family expenditures. Furthermore, the Bureau did not have time to fully develop item specifications to ensure the future pricing of identical or closely comparable items, nor was there time to collect an adequate sample of prices for many of the items priced.
To fulfill a request of another labor arbitration board, the National War Labor Board (NWLB), the Bureau received $300,000 ($5.38 million in 2012 dollars) to complete a national cost-of-living survey; defense supplements between 1917 and 1919 brought the total to $650,000 (about $11.66 million in 2012 dollars).16 The Bureau used the funds to conduct a new household expenditure survey and a new retail price survey to develop a price index for the NWLB. This index represents the formal beginning of a regular, comprehensive national consumer price index program.
The first phase of the project entailed collecting expenditure data from 12,000 families in 92 cities in 42 states. As with the previous 1901–1903 expenditure study, the families studied were defined as those of moderate income, with annual incomes between $900 and $2,500. Unlike the previous survey, however, the new one excluded “non-English speaking families who have been less than five years in the United States.”17 Also, although the Bureau collected data from African American families, only data from families in which Caucasian, male wage earners and clerical workers contributed the majority to a family’s income were included in the official index. So, although the 1917–1919 study was more geographically comprehensive, it was much more restrictive demographically, compared with the 1901–1903 study. The data collected enabled the Bureau to publish average yearly family expenditures for food, clothing, rent, fuel and light, furniture, and miscellaneous goods, and to estimate average total yearly expenditures per family.
The collection of retail prices occurred in two stages. First, the Bureau retroactively used prices collected in retail establishments for December of each year from 1913 to 1917 in 19 industrial centers and in an additional 13 cities for December of 1917. Regular retail price collection began after 1917 in all 32 major industrial centers. Depending on the article priced, pricing occurred one to four times per year.
Using a weighting structure based on the 1917–1919 expenditure survey, the Bureau began semiannual publication of a retail price index in 1919. With the reference base period set to 1913 = 100, values of the index were estimated back to 1913 with the use of wholesale price movements. In 1921, Washington, DC, was added for price collection.
Again, the results of these studies were used to standardize and stabilize wage rates during U.S. involvement in World War I. A secondary use, due to wartime inflation, was to determine whether compensation maintained parity with price increases during the period. For example, the Congressional Joint Commission on Reclassification of Salaries used BLS data to conclude that federal employee compensation in the District of Columbia had not kept up with inflation. The Commission came to this conclusion by establishing a standard budget for a family of five. The budget represented “a sufficiency of food, respectable clothing, sanitary housing, and a minimum of the essential ‘sundries.’”18 The Commission noted that the budget did not include savings and insurance or recreational or educational items that ought to be included in a “proper American standard of living.”19 Using BLS data, the Commission estimated such a budget to be $2,288 for a family of five in the District of Columbia in 1919.20 Similar family budgets were constructed for families elsewhere in the country by federal agencies and advocacy groups, using data from the 1917–1919 expenditure study.
Immediately after World War I, the Bureau faced daunting challenges in maintaining the scope of the cost-of-living program. Congress voted to return federal agency budgets to prewar levels, and although the Bureau sought a “deficiency appropriation” of $475,000 in 1919 to continue cost-of-living and budget studies with the same frequency and scope as those carried out during the war, Congress approved just $12,000.21 Then, throughout the first half of the 1920s, BLS senior officials tried to persuade Congress of the desirability of increasing appropriations to a level sufficient to maintain the quarterly publication of a “cost of living” index, but Congress failed to fund the increase. In 1925, BLS Commissioner Ethelbert Stewart had no choice but to accept continued budgetary constraints, whereupon he reduced cost-of-living work to a semiannual schedule.
Judging from the retrenchment of cost-of-living work in the earlier half of the 1920s, it would have been fair to assume that, once again, the future of a consumer price index as a national economic statistic was uncertain. However, such an assumption would have been premature. Just as Congress was scaling back budgetary resources, a triad of businessmen, economists, and unionists was gradually acknowledging that there was a secondary, though no less important, role that a national consumer price index could play in an increasingly consumer-driven economy. Businessmen were interested in gaining insight into consumer behavior and in modeling consumer demand. Unionists were concerned with maintaining the purchasing power of wages relative to changes in productivity. Economists obviously desired to study both aspects of the nascent consumer economy. So, by the late 1920s, these same groups, along with Commissioner Stewart, acknowledged the need to begin work on updating the composition of the fixed basket of goods and services that was derived from the 1917–1919 consumer expenditure study and that the Bureau used in its “cost-of-living” index.
Although, by the close of the 1920s, the Bureau would not undertake a comprehensive study to update the basket, it did participate in two smaller surveys of income and expenditures. One survey was conducted on behalf of the Personnel Classification Board, with the purpose of aiding the Board in establishing a wage scale for government field service. The Bureau surveyed 506 families of federal employees from 1927 to 1928 in Baltimore, Boston, New York, Chicago, and New Orleans. The second survey was conducted on behalf of the Ford Motor Company and the International Labor Office in 1929 and was aimed at aiding Ford in setting wage rates for American employees in certain European cities. The rates were to be commensurate with the differences in the cost of living between those cities and Detroit. The Bureau surveyed 100 families in Detroit, and the information gained was then used by various European statistical agencies to establish the cost of living, relative to that of Detroit, of the American employees working for Ford and living in Europe.
The data collected from these smaller surveys were not sufficient to systematically revise the aging market basket from 1919. The Bureau did, however, garner enough information to indicate “the extent of the changes which were taking place in the type of goods purchased and the manner of living.”22 Technological innovation and mass production were changing the purchasing patterns of urbanizing Americans. These late 1920s surveys justified the need to undertake a comprehensive examination of consumer expenditures during the 1930s, an examination that would ultimately culminate in the completion of the first major revision of the CPI program in 1940.
Revitalizing the cost-of-living index in the New Deal era
By the close of the 1920s, the BLS cost-of-living program existed as a much smaller version of its World War I–era self and with its future uncertain. Wartime pressures to centrally manage economic activities eased, resulting in diminished congressional support for funding the program to the extent necessary to maintain an up-to-date national measure of consumer inflation. Also, BLS Commissioner Ethelbert Stewart had to contend with attempts by Commerce Secretary Herbert Hoover to assume responsibility over the publication of cost-of-living data. Although Commissioner Stewart fended off such attempts in the early 1920s, he could do no more than accept and adjust to the reality of dwindling resources and a shift away from reliance on federal economic statistics toward a more “associative state” under President Hoover in the early 1930s whereby “the federal government facilitated the dispersal of expert economic analysis (often created by independent organizations like the NBER23) to guide the voluntary collaboration of individual firms, local governments, and trade associations in efforts to mitigate swings in the business cycle.”24
The gradual neglect that took place during the 1920s and early 1930s became painfully obvious on March 20, 1933—just 16 days after President Franklin Roosevelt took office—when Congress passed the Economy Act of 1933, requiring a 15-percent reduction in federal salaries, on the basis of a more-than-20-percent decrease in the BLS cost-of-living index. Both unions representing federal employees and ex-Commissioner Stewart publicly voiced their dissatisfaction with the act, publicizing the fact that the market basket underlying the index was outdated. Prior to her appointment as Secretary of Labor, Frances Perkins was critical of the state of the Department of Labor, and of the Bureau of Labor Statistics in particular, stating that Commissioner Stewart had “kept an awful shop—oh a terrible shop!— padded with people who didn’t do anything and couldn’t do anything, and padded with untrained and uneducated people to whom he gave jobs.”25 After her appointment, she made known her reservations regarding using the outdated index to adjust federal salaries.26 The objections raised did not go unnoticed; quite the contrary, President Roosevelt was sympathetic to them and understood the importance of the Bureau’s role relative to the goals Secretary Perkins had for the Department of Labor at large. In her first year as Secretary, Perkins explained to the Advisory Committee to the Secretary of Labor (ACSL) that she intended “to make the Department a vigorous and energetic organization for the promotion of labor’s interests and that a revitalized BLS would provide the technical grounding and economic analysis for this role.”27 Thus, Secretary Perkins immediately set to work professionalizing the agency by recruiting technical experts who would oversee the first major modernization of cost-of-living methodology, with the ACSL serving as the main forum for proposals for reform.28
The 1935 ACSL-based revisions. The first step toward major reform that originated from the ACSL was a reweighting of the cost-of-living index in 1935 on the basis of an analysis by Margaret Hogg, a British academic assigned by the ACSL to help revise the index. Hogg’s analysis demonstrated that using expenditure data from 1917–1919 for a 1913-based index biased results, most notably by overstating the importance of the food category. The 1935 reweighting resulted in three changes in the calculation of the index. In the first change, which began in 1934, the cost-of-living program gradually increased the variety of food items priced, particularly fruits and vegetables, thereby allowing for the grouping of food prices by categories of foods thought to have similar price movements. Particular groupings were then weighted by the expenditure of the entire group, instead of one single item. For example, “cabbage prices [would] now [be] weighted, not only by the quantity of cabbage purchased, but also by an appropriated allowance for expenditures for sauerkraut, and carrot prices by expenditures for carrots, beets, and turnips.”29 The new construction of the food index caused an upward shift in the index from 1919 to 1935. Still, the general trend in price movements over the period paralleled that of the former food index.
The second change concerned the method in which the all-items category was aggregated. Prior to the revision, the all-items index was computed by weighting the percent change from the base period for each of the six major categories (food, clothing, rent, fuel and light, housefurnishing goods, and miscellaneous goods) and adding the weighted changes to compute the all-items change. After the revision, aggregation of the all-items index required calculating dollar aggregates for each category’s index, based on 1917–1919 expenditures, and adding together all the categories’ aggregates. The total represented the all-items category, and its index value was calculated relative to the 1913 average dollar aggregate. For example, the average all-items index value for 1917–1919, 152.6, was calculated by comparing the average all-items dollar aggregate for that period, namely, $1,436.18, with the 1913 average, $940.94. This change did not alter index values previously computed, but served to “simplify the calculation of the all items indexes for the United States, since some of the indexes for the 32 cities were based on costs in 1914, and some on costs in 1917.”30
The last change effected by the 1935 revision involved modifying the method by which the large metropolitan indexes were combined to create a “composite index of living costs for the larger cities of the United States.”31 At the time of the revision, food prices were collected in 51 cities and retail prices for all other goods were collected in 32 of the 51 cities. As stated in the Monthly Labor Review article explaining the revision, “it [had] been customary after weighting for each city the chain-store and independent-store prices, to average without further weighting all the quotations secured for each food priced and to multiply these average prices by weights representing average quantities purchased in the United States, [whereas] indexes for items other than food were computed for the United States by combining aggregates for [the] 32 cities and comparing the aggregates for one date with those for another.”32 To approximate more closely the inflation experience of large U.S. cities, the Bureau began weighting the aggregate costs for both food and items other than food by the combined population of both the metropolitan area in which prices were collected and nearby metropolitan areas in which prices were assumed to move in a similar manner. Because the number of cities in which prices were collected for food and for all other items differed, the weights assigned to the food index differed from those assigned to all other groupings of items.
Aside from changes pertaining to the calculation of indexes, the Bureau implemented changes related to sampling. The number of food items sampled doubled from 42 to 84. The rent sample also was revised, to better represent the population consisting of urban wage earners and lower salaried workers. To support these changes, the Bureau expanded its pool of onsite price collectors by training local residents in the urban areas where prices were collected and employed them on a contractual basis to collect some of the data. In addition, the Bureau improved the content and use of written specifications of items for pricing, thereby ensuring greater comparability among geographic areas (for a given period) and among periods (for a given geographic area).33 Besides their importance in creating a more accurate measure of inflation, these changes also were key in that they were the first signs of a less constrained budget and greater operational capacity for the Bureau under Secretary Perkins’ leadership during the New Deal era. Indeed, between 1934 and 1939, the Bureau’s operational budget increased by $400,000, nearly doubling, and its total budget (which included appropriations for special projects) more than quadrupled, increasing by approximately $1.56 million.34
The 1935 revisions were an acknowledgment that the continued use of the 1917–1919 expenditure data was causing bias in the estimation of consumer inflation. Still, the revisions were to serve mainly as a temporary mathematical fix until a new consumer expenditure survey could be completed. So, even while the Bureau was working with the ACSL to address the bias, Commissioner Isador Lubin, Secretary Perkins’ personal selection for the post, immediately and deftly set to work securing funds for the new survey. He procured the funding by presenting the survey as a work-relief program,35 thereby circumventing Congress and receiving funding from the Works Progress Administration.36
Limitations of the new survey. The expenditure survey covered the years 1934 through 1936; spanned 42 cities, each with a population of at least 50,000, throughout the United States; and included nearly 14,500 families. The racial composition of the families surveyed was 12,903 families of Caucasian descent and 1,566 African American families. The average annual salary was $1,524 for Caucasian families and $1,008 for African American families. To be eligible for selection, a family had to have an annual income of at least $500, with at least one earner employed for 36 weeks and earning at least $300 per year. If employed as a clerical worker, the primary earner could make no more than $200 per month or $2,000 per year. Furthermore, the survey excluded families that received financial assistance from either private or government relief programs during the time covered by the survey, families with boarders or numerous37 lodgers, and single individuals. Lastly, the survey excluded the self-employed, those who worked on commission (e.g., street vendors selling either goods or services, such as shoe shiners), and workers who might receive considerable38 nonfinancial reimbursement as part of their compensation (e.g., domestic workers and caretakers).39
The restrictions on the survey reflected primarily the established beliefs about what constituted a working-class family at the time, but also served to simplify data collection and analysis. As Stapleford explains, however, “these restrictions had gender and racial consequences that undercut other efforts by the New Deal staff to make the survey and the cost-of-living index more inclusive.”40 Nonetheless, there were two inclusive policies stemming from the new survey. The first was the abandonment of the 1917–1919 survey’s focus on households whose primary earner was male. The second was the rejection of the earlier survey’s policy of excluding data on African American families from being collected and playing a role in constructing the cost-of-living index. Viewed in juxtaposition, the restrictive sampling parameters and the new inclusive policies clearly clashed, the one limiting the eligibility of female-headed households and the other allowing data on African American households to be collected. Still, the downside was considerable: Stapleford continues, “The elimination of domestic servants and single individuals nevertheless cut out a large swath of the female workforce. Similarly, the occupational restrictions and the no-relief policy [i.e., the exclusion of those receiving financial assistance] likewise greatly reduced the size and influence of the sample of black families.”41
It is important to note that, unlike the situation with the older index, the practical exclusionary effects that may have resulted from the new index based on the 1934–1936 survey were not overtly intended, particularly concerning minority families. In a 1941 bulletin, the Bureau explicitly explains the ways in which the data collected on African American and Hispanic families were incorporated into the construction of the new index:42
The weights in the original index were derived from the expenditures of white families only. In the new index, in each city in which the Negro population is of importance among employed wage earners and clerical workers, expenditure data were summarized for the white and Negro groups separately and combined for the purposes of weighting the index by means of weights representing the relative importance of the white and Negro groups in these cities, as shown by the United States census of 1930. Moreover, for all those commodities and services generally purchased in different outlets by the two groups, e.g., haircuts, the Bureau is now securing prices in the different outlets patronized by the two groups. In Houston and Los Angeles, expenditures for Mexican workers’ families were averaged with those of other white workers’ families; in Houston in the ratio of 1 to 19 and in Los Angeles in the ratio of 1 to 13.
There is nothing in the bulletin in which the preceding text appears which suggests that there was any bias due to underrepresentation of African American families in the underlying sample, although Stapleford notes that “the national averages that included black families showed only a ‘negligible difference’ from those for whites alone (despite racially distinct purchasing habits).”43 It is hard to determine conclusively whether the sampling criteria did indeed somewhat undercut the Bureau’s effort to create a more encompassing measure of the effect of inflation on moderate-income families. What can be said is that the Bureau did take the necessary steps in creating an inflation measure that was more representative of all moderate-income, urban families, but that it still had a ways to go in creating an all-encompassing measure of consumer inflation for the urban population in general.
Introduction of the new cost-of-living index. In early 1940, the BLS Cost of Living Division introduced the new index, titled the Cost of Living of Wage Earners and Lower-Salaried Workers in Large Cities. The index was based on the 1934–1936 expenditure study. The revisions incorporated into the new index included both methodological changes and changes in the structure and composition of the index. As was suspected, the older item structure failed to reflect contemporaneous consumption habits, and the new index introduced items that had become common consumer goods between 1919 and 1934. The new index reflected the two realities that goods normally associated with upper-income families were becoming less expensive and were now within reach of moderate-income families and that the electrification of daily life was quickly becoming the norm for urban wage-earning families as well as the more wealthy.44
The new index encompassed 198 goods and services. Generally, the weighted importance of the food and clothing categories decreased whereas the categories for (house)furnishings and miscellaneous items increased. The Cost of Living Division introduced a new aggregate index for household fuel, electricity, and ice. The weights for each individual item in the group index reflected a marked shift away from coal and gas for heating, cooking, lighting, and power, substituting electricity for these purposes at home. After increasing the number of items priced in the food index to 84 in the 1935 revision, the Division found that the price movements of some items “could be predicted from those of others.”45 As a result of this observation, the Division reduced the number of priced food items from 84 to 54, although some items (mainly within the fresh fruits and vegetables category) increased in number and others (fresh and frozen fish) were newly incorporated into the index. The reduction in the clothing category reflected the fact that new, less costly alternatives (e.g., the introduction of synthetic fabrics such as rayon) allowed moderate-income families to devote less of their budget to fulfill their clothing needs. With the electrification of the household came the prevalence of the electric appliance, resulting in a higher weight for the housefurnishings index. The transportation component, which consisted of only streetcar fare in the older index, now included several means of transportation, with the majority of the weight of the component assigned to automobiles and automobile-related goods and services (fuel, taxes, insurance, etc.).
Equally important were the methodological revisions of the new index. In fact, many aspects of the changes in methodology that resulted are still valid in current CPI methodology. With expanded resources, the Cost of Living Division reinstated onsite price collection by trained BLS staff. The Division also standardized and expanded the use of detailed pricing specifications to ensure that items selected when a price collector returned to an outlet in a subsequent pricing period were comparable to those selected earlier at the outlet by the same price collector. Standardization also enabled price collectors to minimize the variation between like items when substitution became necessary because certain items were out of stock or shoppers changed their consumption habits. When substitution was necessary, the Division adopted the linking method, whereby, when “another article is substituted, of approximately the same grade and serving the same purpose, [and if] there is a price differential between the two articles…this differential is not reflected in the index [for that pricing period]. The new article is introduced by a linking method.”46 Weights for a group of like items (e.g., meats) were “imputed to a small group of selected items for which prices are actually obtained each quarter or each month.”47 In the particular example of the meats group, 13 specific cuts of beef represented the purchase of all meats by urban wage-earning and clerical families. Lastly, and further demonstrating the influence that academia and technical experts had in reforming the index, the BLS established official policy on how taxes were treated as part of the price collected for an item. Sales taxes and consumption taxes (such as automobile taxes) were included in the measurement of an item’s price, but only after prices had been collected and only during index calculation. Property taxes were assumed to be implicitly reflected in rental prices, and income taxes and Social Security taxes were excluded from the index (the latter being treated as savings in the conceptual framework at the time of the reform).
When Frances Perkins became Secretary of Labor, the BLS Cost of Living Division was lacking the budgetary and, as Secretary Perkins argued, personnel resources to maintain a technically robust national consumer price index that reflected current consumer preferences in a dynamic modern economy. The data underlying the index that was used to adjust federal salaries in the early 1930s were painfully outdated; they reflected a time before household electrification, for example. Secretary Perkins’ leadership oversaw a complete revitalization of the Bureau and the cost-of-living program. The modernization program implemented during the New Deal era was timely, for, once again, federal economic planners would rely on the CPI as a primary federal economic indicator as the U.S. economy mobilized for World War II. Apart from the influence the index had on wartime planning, however, a reading of the most current CPI methods manual reveals that the legacy of the modernization of the cost-of-living index during the New Deal era is still present today in some aspects of current CPI methodology.48
From technical to political: managing challenges to the cost-of-living program during World War II
Before World War II, the role of a government statistic estimating the change in the cost of living was often limited to serving as an aid in federal economic planning and policymaking. If Congress or the President’s administration saw an important need for the CPI, it would direct the Bureau to undertake the task of creating or updating that index. Often, when the need for the index diminished, the budget for the index would subsequently diminish as well. As evidenced in the history of the CPI through the 1930s, this approach made regular, periodic updates in methods and data difficult to maintain. However, the approach would change beginning with World War II. In addition to serving the same role in national economic policymaking as it did during World War I, the CPI would begin to play a prominent role in industrial relations when an increasing number of corporations and national unions used the index for wage and contract negotiations. The adaptation of the index to industrial relations purposes would bring about greater public scrutiny, particularly from national union leaders, concerning CPI methodology. The heightened scrutiny during World War II and the immediate postwar period resulted in three external reviews of the CPI program. Prior to 1943, there had been only one major external review of CPI methodology: the ASCL conducted such a review in the early 1930s.49 The debates and reviews initiated by the dual, and often competing, uses of the CPI during World War II and the immediate postwar era as a monitor of inflation and as a mechanism for adjusting wages would propel the CPI program to gradually abandon the constant-goods framework in constructing a price index, to be replaced over time with the constant-utility conception of price index construction.
In previous decades, there was usually a long lull in revision activity for CPI staff after the introduction of a revamped cost-of-living index. Following the unveiling in early 1940 of the Cost of Living of Wage Earners and Lower-Salaried Workers in Large Cities index,50 however, there would be no respite for the CPI program. Wartime mobilization of the U.S. economy through 1941 quickly strained the ability of the new index to reflect changes in the prices paid for commonly purchased consumer goods and services. Throughout 1942, the Bureau’s Cost of Living Division made incremental changes to the index to cope with disappearing goods, government rationing orders, and changing population patterns in an attempt to make “its cost-of-living index represent each month changes in the costs of the goods and services which wage earners and clerical workers [could] actually buy in the war years.”51 Many of the changes made in the early months of 1942 were in response to government rationing orders that affected the availability of various transportation-related goods and the resulting modifications of purchasing patterns. For example, because the federal government issued rationing orders on new automobiles and new tires and tubes, the Cost of Living Division dropped these two items entirely from the index on January 15, 1942. They were replaced with used automobiles and used, recapped, and retreaded tires. Also, the relative weights of automobile repairs and public transportation costs were increased to reflect changing consumer behavior. These types of modifications were characteristic of a wide range of changes that were made in the clothing, housefurnishings, and “miscellaneous” categories throughout 1942. Table 1, taken from an article in the July 1943 issue of the Monthly Labor Review, summarizes these early wartime adjustments.
Changes to the food index were less drastic than those made to the indexes just discussed. The BLS pricing program sought to adjust the weights of the food index, which were based on expenditure data from 1935 to 1939, to “conform as nearly as possible to current buying.”52 Broadly, the reweighting was achieved by applying, to each component, a factor that represented “the change between average per capita consumption in the country as a whole in the years 1935 through 1939 and expected consumption in 1943, as estimated by the Department of Agriculture.”53 Staff made adjustments to other indexes as well when they expected the consumption of those goods to be affected particularly by the number of ration points allotted to the number of persons in the average family headed by an urban wage earner or clerical worker in each of the 56 cities covered by the Bureau’s food index.
The Cost of Living Division adjusted the population cost weights of each sampled city in a similar manner; the weights were then used to aggregate prices in order to calculate the combined indexes of the various large cities. Although the Sixteenth Decennial Census of Population and Housing, conducted in 1940, was completed by the time of the wartime revision of the CPI, an analysis of sugar ration data conducted by the then Bureau of the Census demonstrated that the country had already experienced significant shifts in population from April 1940 to May 1942. Therefore, before applying the 1940 census data, BLS used the figures from the sugar ration study to adjust the data. The population weights derived from the adjusted 1940 census data were meant to reflect the wartime migration of workers throughout the country.
Shortages and disappearing goods also presented a challenge in the field when field agents visited retail outlets to price goods and services. Only as recently as 1935 had the Bureau begun to use detailed pricing specifications in a manner consistent enough to ensure that items of like quality were priced in the same way from one period to the next. The use of specifications would prove invaluable, for not only did the Bureau have to decide how to handle discontinued or disappearing goods, but also field agents had to be vigilant in perceiving quality changes in items that remained available on store shelves. The measurement of price change for a particular item was contingent on the information provided by field agents who recorded a price in the current pricing period, t. When agents obtained a price that was within specification for a given product, but noted some type of deterioration in the quality of the product, the pricing program treated the situation as a price increase from period t – 1 to period t. For example, an agent “noted that a box of corn flakes, priced the same as last year, is now marked as containing 6 ounces, where it formerly held 8 ounces. This change [would appear] in the index of food costs as an increase in the price of a half pound of corn flakes.”54 Division staff developed two ways to reflect a price increase in the case when a lower price line item was discontinued and the only option was to purchase a higher price line item of similar specification. If the higher and lower price line items were simultaneously available in periods t – 1 and t, then the price increase was calculated as the change between the price of the higher line item in t and the average price of both items in t – 1. If the higher price line item was not available until after the discontinuation of the lower price line item, then the index reflected the full price increase between the higher priced item in t and the lower priced item in t – 1.
Wartime pressures on the CPI. Despite the Bureau’s attempts to maintain a representative index that accurately estimated the change in the cost of living for the average wage earner during wartime, political developments outside the control of the Bureau brought public scrutiny upon CPI methods and embroiled top BLS staff in the ensuing public debates on wartime economic policy. Wartime mobilization caused substantial inflationary pressure on both prices and wages; the administration of President Franklin D. Roosevelt called inflation the domestic “Enemy.”55 To monitor and control inflation, Roosevelt reestablished the tripartite NWLB and created the Office of Price Administration (OPA). “On the prices side, the index became the main measure of the OPA’s success in controlling prices; simultaneously the tripartite [NWLB] used the index to establish a ceiling for wage increases.”56
It is important to contrast the role the CPI played in each of the world wars. During World War I, the NWLB used the CPI to stabilize wage rates through standardization, particularly to maintain parity with increases in the general price level. During World War II, however, the NWLB used the CPI to cap wage increases as a means of controlling inflation. The change in emphasis was a source of criticism directed at the CPI. While the NWLB and the OPA saw the CPI as one tool in the administration’s toolkit for holding down inflation, national union leaders continued to view the CPI as the main, if not only, tool to use in arguing for increasing wages to maintain purchasing power with wartime inflation. The tension between the dual use of the CPI came to a head when the NWLB used the CPI in a wage decision to settle a dispute between smaller steel companies and their employees.
Rendered in July 1942, the NLWB decision granted a 15-percent wage increase to the steelworkers. The decision was based on the change in the cost-of-living index since January 1941. When President Roosevelt made the NWLB the arbiter of all wage disputes, the decision from the “Little Steel” case became universal wage policy and the formula used would come to be known as the Little Steel wage stabilization formula. The 15-percent cap on wage increases and the change in the cost-of-living index quickly diverged after the initial Little Steel decision, and although the index had increased by 23 percent by January 1944, the Little Steel cap remained unaltered. The NWLB’s steadfast adherence to the Little Steel formula caused national labor unions to identify the Bureau’s cost-of-living index as the vulnerable element of the federal government’s administration of price and wage controls. With newly created research divisions, national unions aggressively challenged not only the Bureau’s index methods, but even whether the cost-of-living index would adequately reflect the many economic shifts that occurred during the war and affected the average wage earner’s or clerical worker’s costs to live, work, and play.57 The Bureau’s response to those who criticized the cost-of-living index was that “changes in the plane of living are quite different from changes in the cost of living, and it is the cost of living which the Bureau’s index measures.”58
The then Acting BLS Commissioner A. Ford Hinrichs desired to quell the mounting criticism by requesting Secretary Frances Perkins to invite the American Statistical Association (ASA) to create a neutral panel of experts “to review and appraise the cost of living index with reference both to its construction and its uses.”59 The ASA appointed Frederick C. Mills, a Columbia University economist and research fellow in price statistics for the National Bureau of Economic Research (NBER), to chair the review committee. The Mills Committee issued its report in the fall of 1943, and its conclusions generally coincided with the BLS position. In discussing what the BLS called “changes in the plane of living,” the Mills Committee acknowledged that “alterations in the manner of living and in income have been particularly important during the war,” but that “many individual and family incomes have advanced more rapidly than living costs.”60 The report noted a Department of Commerce study which illustrated that average compensation in private nonagricultural industries increased 37 percent from 1939 to 1942 whereas the Bureau’s cost-of-living index increased 17 percent in the same period.
The point was to demonstrate that the cost-of-living concept included many measures of well-being. Although officially titled the Cost-of-Living Index, the BLS index measured only one aspect of the actual cost of living: the change in average retail prices paid by urban wage earners and clerical workers. The Mills Committee concluded that the index “provide[d] a trustworthy measure of changes in the prices paid by consumers… [and] that many of the difficulties and doubts which ha[d] arisen concerning the index ha[d] their origins in attempts to use it uncritically for purposes to which it is not adapted.”61
Critics of the index remained undeterred or unconvinced by the ASA’s investigation of the controversy enveloping the cost-of-living program and continued to lobby the Roosevelt administration while the Mills Committee conducted its appraisal. The lobbying came to a head in November 1943, just as the committee concluded its review. At that point, the Roosevelt administration directed the NWLB to form a tripartite review committee, to include labor and industry representatives, that would examine and resolve the “controversy and dispute as to what the cost of living is.”62 The President’s Committee on the Cost of Living met throughout the next year, producing multiple reports representing the various groups’ views.
Organized labor and the CPI. Shortly after the formation of the President’s Committee, the labor representatives on it submitted a report asserting that the true cost of living had increased 43.5 percent from January 1941 to December 1943 whereas the BLS cost-of-living index had increased by only 23.5 percent. The report emphasized that the BLS index failed to accurately and fully reflect the many changes the wartime economy had inflicted on the cost of living for urban wage earning and clerical families.63 The Bureau riposted that evidence affirmed otherwise; after all, the Mills Committee had confirmed that the Bureau had, as best as possible given the scale of challenges, adequately adjusted its index methods to account for conditions (such as disappearing items) that affected retail prices. In effect, the committee had affirmed the index’s accuracy as a measure of the change in retail prices. The discrepancy between labor’s numbers and the BLS index was not a matter of different methods used to measure the same phenomenon, namely, changes in prices; rather, national unions argued that the cost-of-living index published by the Bureau was too narrow in scope. Speaking to Commissioner Hinrichs at a committee meeting, American Federation of Labor (AFL) Representative George Meany stated, “If the index is a retail price index and not actually a cost-of-living index, we have no particular interest in what the index has done in the past….My attitude is…let’s go into partnership and see what we can do together—this committee and your department—to make the index do the thing that it has not done.”64 Commissioner Hinrichs insisted that, by definition, cost-of-living indexes were constant-good, retail price indexes, and as the ASA report explained, this type of index was limited in its ability to fully capture changes in all factors that affect an individual’s well-being. The index’s limitations were especially exacerbated when wartime conditions caused the “plane of living” for many families to be in a state of flux.
The impasse between the national unions and the Bureau caused NLWB chair William Davis to form an external panel of technical advisors in 1944 to assess the BLS index in relation to the controversies at hand. Known as the Mitchell Committee, the panel was chaired by NBER economist Wesley Mitchell and included Simon Kuznets, a fellow NBER economist who served as an associate director in the War Production Board, and Margaret Reid, of the Budget Bureau’s Office of Statistical Standards and former member of the ASA committee. The Mitchell Committee issued its final report in June 1944 against a backdrop of strained relations between national labor officials and BLS staff.65 The Mitchell Committee generally concurred with the findings of the Mills Committee, asserting that, given the challenging economic conditions, the Bureau had “done a competent job…in providing a measure of price changes for goods customarily purchased by families of wage earners and lower-salaried workers living in large cities.”66 The Mitchell Committee also agreed with the Bureau’s position that many were confusing the additional expense of attaining a higher standard of living for an increase in the cost of a fixed standard of living. Without offering any methodological changes, the report proposed that the BLS cost-of-living index underestimated inflation by at most 3 percent or 4 percent from January 1941 to December 1943 by not fully capturing the effects of deteriorations in quality. The only notable suggestion the Mitchell Committee had for the Bureau was that it consider a change in the name of the index to clarify any ambiguity or misunderstanding about what it actually measured.67
In November 1944, William H. Davis, chair of the President’s Committee, submitted his report to President Roosevelt. Drawing largely from the Mitchell Committee report, Davis concluded that the cost-of-living index accurately measured what it was intended to measure. He also concluded that, in failing to take into account hidden increases in prices (e.g., an effective increase in the price of a good whose quality has been lowered but whose price remains the same) and in excluding small cities from the geographic sample, the index was likely biased downward by between 3.5 percent and 4.5 percent, but not more than that. In February 1945, Fred M. Vinson, the director of the Office of Economic Stabilization, used the findings of the President’s Committee report to maintain the Little Steel formula for federal wage stabilization policy.68
Although neither the Mitchell Committee nor the President’s Committee suggested methodological changes to the Cost-of-Living Index, BLS responded to the recommendations of the reports in two ways. First, a month after the Mitchell Committee submitted its report to the President’s Committee, BLS began to include a short disclaimer with each regular monthly release of the index. The disclaimer stated,69
The Bureau of Labor Statistics index indicates average changes in retail prices of selected goods, rents, and services bought by families of wage earners and lower-salaried workers in large cities. The items covered represented 70 percent of the expenditures of families who had incomes ranging from $1,250 to $2,000 in 1934–36.
The index does not show the full wartime effect on the cost of living of such factors as…changes in housing and eating away from home.
It does not measure changes in total ‘living costs’—that is, in the total amount families spend for living. Income taxes and bond subscriptions are not included.
Then, in July 1945, the Bureau changed the name of the retail price index. The index would no longer contain the phrase “cost of living.” Instead, Commissioner Hinrichs proposed the title “Consumer’s Price Index for Moderate Income Families in Large Cities.” Secretary Perkins adamantly refused to accept Hinrichs’ proposal, telling him in a note that “The people who object to this index don’t object to its name, you know. They object to its shape and size, and it won’t be any more comfortable or acceptable under another name.”70 Frances Perkins’ tenure as Secretary of Labor ended June 30, 1945, however, and her successor, Lewis B. Schwellenbach, quickly agreed to Hinrichs’ title upon assuming leadership over the Department of Labor.
A permanent role for the CPI in postwar industrial relations
In the years immediately after World War II, it appeared that the Bureau and its CPI program were set to suffer the same fate of substantial cutbacks as had occurred after World War I. Indeed, the Bureau experienced two successive rounds of a 40-percent reduction in its budget in 1947 and 1948, a total reduction of 64 percent over the 2-year span. The cutbacks left new Commissioner Ewan Clague with a dramatically reduced workforce: approximately 700 of 1,700 BLS employees were dismissed as a result of the budgetary contractions.71 The CPI program responded by curtailing the scope of the items and areas used in the survey sample and by pricing items less frequently. An unexpected ally of the program appeared immediately after the 1948 cuts, in what Commissioner Clague deemed “an industrial relations ‘miracle’”: the executive staff of the General Motors Corporation lobbied Congress not only to reverse the cuts to the program’s budget, but also to provide additional funds for a revision of the index.
As the federal government relaxed its wage stabilization policy, consumer prices began to rise rapidly in the immediate postwar years. Industrial strife and disruption was particularly acute in 1946.72 As a strategy to prevent union strikes, in 1948 General Motors negotiated a 2-year labor agreement with the United Auto Workers (UAW) that included “two adaptive wage clauses,” one of which required the use of the CPI.73 The CPI component of the agreement first increased wages by 8 cents per hour, a number based on the change in the index from its average in 1940 to April 1948. Thereafter, the contract stipulated that, for every 1.14-point change (upward or downward) in the index, wages would be adjusted accordingly by 1 cent, except that downward adjustments were bounded if the index decreased to 164.7 from the April 1948 level of 169.3. As a result, General Motors could decrease wages by 4 cents an hour at most. Reviews would occur quarterly. Beyond maintaining a constant purchasing power, the parties agreed to increase wages by an annual improvement factor of 3 cents an hour, based on long-term trends in national productivity; the 3-cent-an-hour increase was derived from the mutually agreed-upon supposition that national productivity improved by 2 percent a year.74
General Motors and the UAW agreed to another contract in 1950, with terms almost identical to those of the 1948 agreement. The 1950 contract would extend for a “historic” 5 years, however. At the time of the first agreement, nearly all major corporations and unions were reluctant to adopt the General Motors–UAW agreement as a model for labor contracts. But with General Motors having the assurance of no wage strikes for 5 years, and with the UAW securing a guaranteed 20-cent-per-hour increase over the 5 years by negotiating the annual improvement factor upward, to 4 cents an hour, the new General Motors–UAW agreement became the standard upon which many companies and unions based their escalator clauses. By the end of 1950, some 2 million workers were covered by escalator clauses. The number increased to about 3½ million workers by the end of 1951. The majority of the clauses were explicitly tied to the CPI.75 By 1952, it was estimated that “every index point (0.5 percent change) shifted roughly $70 million annually through COLAs [cost-of-living adjustments].”76
General Motors’ labor strategy now depended upon a well-funded, comprehensive CPI, and company executives prodded Congress to rescind the 1948 cuts to the Bureau’s budget. Recognizing the index’s new stabilizing role in industrial relations, Congress went further than General Motors’ request and approved additional funding in 1949 for a major revision of the index. Hostilities on the Korean peninsula would delay the completion of the revision because, in an attempt to prevent a repetition of the controversy experienced during World War II, the Bureau instituted a temporary revision in the index at the start of the Korean War. The Bureau relied on recently conducted housing and expenditure surveys to complete the revision, finally introducing the revised index in February 1951. Linked to the existing series as of January 1950, the index reflected a “revision of city population weights, [a] correction of the rent index [meaning the inclusion of newer rental units], [the] addition of new items, and [a] revision of market basket weights.”77 The Bureau published the interim index and the old series concurrently until 6 months after the introduction of the revised CPI in February 1953; concurrent publication would give parties to escalation agreements time to adopt the revised index.
The revised index was based primarily on data from the 1950 Survey of Consumer Expenditures in 91 cities, but also incorporated data from “surveys made for other cities in earlier postwar years.”78 The expenditure survey included 8,000 urban families headed by a wage earner or a salaried clerical worker and with an average size of 3.3 persons and an average 1952 after-tax income of $4,160. The survey continued to exclude domestic workers and single individuals living alone. Families whose 1950 total after-tax income exceeded $10,000 also were excluded. For the first time, however, the Bureau sampled families living in small urban areas, which it defined as areas with a population of at least 2,500; previously, weights were based on families living only in large cities with a population of at least 50,000.
The new index priced approximately 300 items. The measurement of housing costs differed substantially from how the Bureau previously had calculated costs. Previously, the Bureau had used rents as the sole proxy for housing costs. In the 1953 revision, however, the Bureau changed “from a rent-based method to an ‘asset’ formulation based on five specific costs associated with homeownership: House prices, mortgage interest, property taxes, insurance, and maintenance and repair costs.”79
The geographic sample included 46 cities, selected on the basis of a stratification of all urban areas by size, income, and climate. While food, fuel, and rent were priced monthly everywhere, the frequency of pricing other items varied by city size. Field agents collected prices monthly in the 5 largest cities, on a rotating cycle quarterly in the next 25 large and “medium-sized” cities, and every 4 months in the smallest 16 cities of the geographic sample. Within a city, the Bureau used probability sampling to select outlets for the pricing of food and rents. For all other items, the Bureau selected “judgment samples…based on size, type of operation, quality of commodities sold or service rendered, location, and clientele.”80
In addition to publishing the U.S. index, the Bureau calculated and published indexes for the 20 largest cities of its geographic sample. Indexes were published for the all-items category, 9 major categories (food, housing, apparel, transportation, medical care, personal care, reading, recreation, and other goods and services), and 15 subgroups.
Renewed public scrutiny of the CPI’s methods grew throughout the 1950s owing to a number of socioeconomic factors or events, such as changing demographics and buying patterns, labor disputes, and the 1957–1958 recession. Whereas academic economists had always been particularly active in critiquing the Bureau’s methods for constructing the CPI, extensive adoption of the index as a wage escalator in labor contracts between businesses and unions caused business economists to begin taking a critical look at those methods as well. This community “complained that BLS included too many luxury items” and “ignored the bargain-hunting and substitution habits of American consumers. They pointed to specific problem areas, such as [the Bureau’s] treatment of quality change and introduction of new products.”81
With scrutiny and criticism mounting by the close of the 1950s, the Bureau resolved to begin another major revision of the CPI. However, just as the revision got under way, the Bureau of the Budget independently commissioned the NBER to undertake a comprehensive review of all government price statistics. Throughout World War II and the postwar era, union leaders had lobbied tirelessly to convince the government apparatus charged with federal economic and statistical policy that BLS underestimated changes in the true cost of living. The NBER review, headed by University of Chicago economist George Stigler, would bring about a paradigmatic shift in the theoretical framework within which BLS would make methodological choices. In response to the review’s findings, BLS would abandon the constant-goods conception of a price index and adopt the constant-utility framework as the guiding theoretical perspective in revising future indexes. Implicit in this switch was the acknowledgment that, contrary to union arguments, the CPI overstated, or at least was an upper bound of, changes in the true cost of living.
Wages to welfare: conceptual changes in the CPI
The continuing theme that runs through this article is the notion that challenges presented by specific historical events often compel changes in the methodology that underlies the CPI. In other words, the answers to questions concerning why the Bureau uses a particular index formula or defines particular concepts as it does are rarely simply a matter of technical precision. Rather, they are frequently a matter of identifying some event that creates a need for a specific kind of price index and analyzing the (often competing) demands that different users place on that index. Without understanding the push and pull among the various users of the index, one cannot fully appreciate the features of the CPI at any particular point in time. Up until the late 1950s, decisions concerning the construction of the CPI were made with a firm adherence to what price index theorists and practitioners call the constant-goods framework of price index construction. That is, design, implementation, and calculation decisions were made with the idea that the CPI estimated the change in the cost of purchasing the same market basket of goods and services over a specified period. Furthermore, this market basket of goods and services was meant to represent only a specifically defined subset of the U.S. population, which, for much of the early history of the index, was those urban households headed by a married Caucasian male wage earner or clerical worker working full time and making an annual income falling within a certain range.
Without any context, these decisions appear to be ill conceived if the goal was to measure general consumer inflation continuously. As previous sections explain, however, generally that was not the goal of the CPI from its conception through at least the early 1950s. Rather, the CPI was primarily a statistic used to guide arbitration in industrial relations between unions and the major industrial sectors of a wartime economy. Given the context of measuring inflation over a relatively short time (during a war), for a defined purpose (arbitration between unions and vital industrial sectors), and under rather challenging conditions (a tumultuous wartime economy), the constant-goods framework for a relatively narrowly defined population is not as ill conceived as might be first thought.
The competing uses for the CPI as a tool in industrial relations arbitration and as an indicator of the effectiveness of government price controls during World War II exposed the many weaknesses of this rather limited conception of what the CPI estimated. Scrutiny over CPI methods expanded beyond academia as major corporations followed General Motors Corporation in adopting the index for escalation purposes in multiyear collective bargaining agreements throughout the 1950s. The expanding role of the CPI furthered the assertion that the limited constant-goods framework upon which the index was built was inadequate to fulfill the need for a national statistic that estimated the change in the cost of living over time for the urban consumer.
The conceptual foundation for the CPI is no longer the constant-goods framework of the past. The remaining history of the index, from the 1960s to the present, is really one narrative about how the Bureau incrementally discarded that framework for the alternative constant-utility conception of price index theory and about the changes that would result from such a switch. A defining moment of this narrative is the commissioning in 1959 of the Price Statistics Review Committee, commonly known as the Stigler Committee, by the Office of Statistical Standards of the Bureau of the Budget and the recommendations that followed in the published report in 1961.
The Stigler Committee and a constant-utility conception of the CPI. As Thomas Stapleford explains, the new tools and Keynesian ideas that grew out of the World War II experience exposed policymakers to the potential for national economic planning and the search for steady economic growth.82 Planners in the early 1950s saw growth as a means to achieve many ends, most crucially as a way to overcome the threat posed by the Sino–Soviet alliance of that decade. Amid a slowdown in aggregate growth and an accelerating rate of increase in the CPI in the late 1950s, the congressional Joint Economic Committee initiated hearings on growth and inflation. As union and corporate representatives exchanged words over why growth had slowed and inflation increased, a curious line of argument was presented by two economists: Richard Ruggles of Yale University and Albert Rees of the University of Chicago. In separate testimonies, Ruggles and Rees argued that the disparity between slowing aggregate growth and rising inflation was illusionary because the CPI was overstating true inflation. They explained that the CPI inadequately reflected innovation that was occurring throughout the economy: the index was failing to account for quality improvements and the introduction of new goods. Given the testimony of these two renowned economists, the Joint Economic Committee in December 1959 “recommended that the government ‘improve the design of [the CPI and the Wholesale Price Index] so that they would more accurately reflect quality and productivity changes and the introduction of new products.’”83
The Bureau had spent much of the 1940s and 1950s addressing claims that the CPI underestimated the true cost of living, but with Ruggles and Rees highlighting methodological deficiencies that likely served to overestimate inflation, and with the detrimental effect that this bias could have on other economic statistics, policymakers’ focus quickly shifted in the direction of addressing the potential overestimation measurement problem. It is amid this concern that the Office of Statistical Standards asked the NBER to form the Stigler Committee to review the CPI program. Among other selections, George Stigler’s choice to include Ruggles and Rees on the Committee is indicative of the conclusions it reached.
The Committee presented its report in 1961. The authors of the report wasted no time in addressing the weakness of the guiding conceptual framework of the then-existing CPI program, opening the report with the following statement:84
It is often stated that the Consumer Price Index measures the price changes of a fixed standard of living based on a fixed market basket of goods and services. In a society where there are no new products, no changes in the quality of existing products, no changes in consumer tastes, and no changes in relative prices of goods and services, it is indeed true that the price of a fixed market basket of goods and services will reflect the cost of maintaining (for an individual household or an average family) a constant level of utility. But in the presence of the introduction of new products, and changes in product quality, consumer tastes, and relative prices, it is no longer true that the rigidly fixed market basket approach yields a realistic measure of how consumers are affected by prices. If consumers rearrange their budgets to avoid the purchase of those products whose prices have risen and simultaneously obtain access to equally desirable new, low-priced products, it is quite possible that the cost of maintaining a fixed standard of living has fallen despite the fact that the price of a fixed market basket has risen. [Emphasis added]
In other words, the Stigler Committee argued that, in estimating the change in the cost of living, CPI methodology held consumer preferences and behavior constant through time and that this approach did not reflect reality very well. Rather, consumer tastes and behavior responded dynamically to changes in relative prices and to the introduction of new goods and services, all in an attempt to satisfy consumers’ needs and desires (utility) at the lowest possible expenditure. Therefore, the report asserted, the CPI needed to adopt new methods and definitions that better modeled the actual dynamics of consumer behavior. This approach meant moving away from the idea that the CPI measured the change in the cost of consuming a fixed set of goods and services over time (usually between 10 and 15 years) and adopting the idea that the CPI should measure the change in the cost of maintaining a constant utility (by adjusting consumers’ preferences and spending behavior) through time as relative prices changed.
“To modify the CPI in the direction of a welfare index,” the Committee offered recommendations concerning both the scope of the CPI and addressing ways in which the CPI differed from a “true constant-utility index.”85 Pertaining to the scope of the index, the report urged the Bureau to move in the direction of “a more unified and comprehensive consumer price index program.”86 Such a change meant, first, including single-person households in the current index and, second, expanding the index to cover the entire population, including the “rural nonfarm and farm areas.”87 These recommendations implied preparing a sampling frame that showed “the distribution of consumer expenditures for particular goods and services by market area and type of retail establishment, as the means for determining the location of the current data within the universe and for determining the feasibility and cost of the collection of price statistics representative of the entire Nation.”88
To better align the CPI with a constant-utility index, the report offered recommendations in five areas in which the Committee believed that the CPI deviated from the Committee’s conception of an ideal welfare index and could implement changes with reasonable effort. The authors suggested that the CPI revise expenditure weights more frequently; introduce new products as early as possible and with small weights, and increase weights as products become more widely available and affordable; research and adopt scientific methods (e.g., hedonic regression) to account for quality change; measure the flow of services provided by durable goods, rather than the purchase price of those goods, in expenditure surveys; and review methods of weighting expenditures covered by various insurance products so that the weights would reflect only net insurance expenditures of the population (premiums minus payouts for claims).
To guide BLS staff in moving the index in the direction of a constant-utility index, the Stigler Committee suggested the creation of a permanent internal research division to study “prices, price indexes, and the measurement of welfare changes.”89 Lastly, to determine the extent of the upward bias in the constant-goods CPI, the Committee recommended that the Bureau calculate a fixed-weight index with current expenditure weights applied to previous periods—in other words, a Paasche index, for the 1952–1963 period. This index would underestimate a true constant-utility index, but the average of the then-existing (Laspeyres) CPI and a Paasche CPI would be a rough estimate of the welfare CPI the Stigler Committee envisioned.90
The Bureau did not respond enthusiastically to the Stigler Committee’s conclusion that it should change the conceptual framework of the CPI. Remarking defiantly in a 1964 Monthly Labor Review article introducing the revised CPI, the Bureau stated, “The basic index concepts of the CPI, now in its 51st year, have not changed. The national index still measures average changes over time in prices of goods and services bought by urban wage earners and clerical workers.”91 In a publication published later in 1964, the Bureau restated its position:92
Because of the intensive review of concepts of the index in the 1953 revision program, the Bureau did not anticipate a need for any major definitional changes in basic concepts or coverage….the index would continue to be a measure of price change for a “constant market basket” of purchases made by urban wage earner[s] and clerical workers. This decision prevailed even though the Price Statistics Review Committee had expressed a preference for an index of “constant satisfaction,” in which comparisons would be made between different market baskets judged to provide equivalent satisfaction, by some means yet to be determined.
That the Bureau initially rebuffed the Stigler Committee is not surprising. The Stigler report renewed a debate that the Bureau thought it had settled in the early 1920s. During that time, Irving Fisher, the leading price index theorist of his day, lobbied the Bureau to adopt his “ideal” index formula in calculating the CPI. Fisher’s formula was the very same one the Stigler committee would later suggest using to estimate the bias of the CPI in relation to a true utility index. Wesley Mitchell, another well-known early-20th-century economist, argued against the use of the Fisher Ideal index, asserting that the CPI should be a measure of price change only and that, by including both changing prices and changing quantities, the Fisher Ideal index muddled any interpretation of changes in the index. In 1921, BLS Commissioner Meeker sided with Mitchell, writing about the Fisher index,93
We have here an inextricable mixture of changes in prices and changes in quantities of goods. What does this “ideal index” which has been achieved by so much mathematical labor, represent? Is it an index of price changes? Or is it an index of quantity changes? The answer seems clear that it is both and neither….The all important fact is that this index does not mean anything that can be clearly grasped.
The Bureau echoed Meeker’s sentiment 43 years later in one of its 1964 publications explaining the revised CPI. “the revised CPI,” the Bureau asserted, “continues to be what it has always been—a measure of price change, and of price change only, in items purchased by urban wage and clerical workers for their own consumption.”94
One other area of contention was the treatment of durable goods, particularly home purchases. As mentioned earlier, the Stigler Committee recommended estimating the cost of the flow of services provided by durables. The Bureau wrote in response, “Since an expenditure is considered consummated when the obligation is incurred, rather than when the payment is made, the total purchase price is included even when houses and durable goods have been bought on mortgage or installment credit.”95 The treatment of home purchases would prove to be a constant source of criticism until the CPI began measuring rental-equivalent prices for owner-occupied housing in the mid-1980s.
Although the CPI continued to be based on a constant-goods framework, the revised 1964 index did include changes that accorded with Stigler recommendations pertaining to its scope. First, the index now included single workers living alone. Second, the Bureau eliminated income restrictions: all households were now in scope if 50 percent or more of their total income during the survey year was earned by a wage earner or clerical worker who worked at least 37 weeks of the year, regardless of the total income earned. Third, the geographic sample was expanded to include smaller urban areas, and field agents began to price in the suburbs surrounding major urban areas.
There were also two notable statistical enhancements associated with the 1964 revision. With guidance from members of the Stigler Committee, the Bureau established a system of replicated samples, enabling CPI statisticians to measure sampling error for the first time. Furthermore, the CPI made greater use of probability sampling techniques in selecting the new samples of areas, retail outlets, and items. These changes demonstrate that the Bureau was willing to collaborate with the Stigler Committee to improve the CPI—so long as the improvements occurred within the realm of a constant-goods index.
A welfare index for government indexation. In 1972, work began on developing concepts relating to “a long-sought but elusive cost-of-living oriented index,”96 concepts that would come to underlie the Chained Consumer Price Index, which essentially embodies the Stigler Committee’s call for a “true constant-utility index.” This work represented a relatively quick reversal from the stance the Bureau had presented only 8 years earlier with the introduction of the revised CPI in 1964.97 Then, in 1974, a leading economist of the time said, in testimony before the U.S. Senate,98
Indexation is now spreading rapidly on a strictly voluntary basis. The urgent need is for the U.S. government to legislate indexation for its own activities, and to remove obstacles to private indexation….These changes should be enacted not only to facilitate ending inflation but also to promote fairness and justice in the relations between government and citizens.
Perhaps in response to this call to action, BLS Commissioner Julius Shiskin announced that the Bureau would begin publishing two CPIs starting in April 1977: “an improved index for urban wage earners and clerical workers to meet the requirements of collective bargaining, and an index for all urban households, which will provide a new comprehensive measure of price change for the economy.”99
How did these developments come about? Part of the answer lies in attrition: as Stapleford explains, the older “New Deal” senior staff and economists who led BLS and the CPI program through the mid-1960s retired or otherwise left the Bureau and were replaced with younger economists who took a keen interest in creating a CPI that was as accurate as practical limitations would allow.100 One of the Stigler Committee’s recommendations that the Bureau implemented and that proved to be particularly influential in the conceptual shift from the constant-goods approach to the constant-utility framework was the establishment of the BLS Division of Price and Index Number Research, which led the Bureau through much of the transition toward a constant-utility framework.101
Although these internal developments were influential (especially in the amount of research conducted on concerns addressed in the Stigler report), as the earlier quote from Milton Friedman suggests, it was the use of the CPI for government indexation that created a tangible need for a more representative and accurate index. A targeted Consumer Price Index for Urban Wage Earners and Clerical Workers (CPI-W) was suitable for the 5.1 million workers affected by collective bargaining agreements between major corporations and unions, but inadequate for the “44 million persons [who found] their incomes affected by the index, largely as a result of statutory action.”102 Indeed, the Congressional Budget Office (CBO) estimated that changes in the CPI affected $177.522 billion in federal outlays in 1981, representing approximately 26 percent of outlays for that year.103
Before assuming his BLS post, Commissioner Shiskin oversaw efforts to coordinate the country’s national statistics as chief economist in the White House Office of Management and Budget. With this experience under his belt, he recognized the prominence the CPI could have as a broader indicator of national inflation. He supported efforts that were underway to make the CPI more broadly representative and applicable, and made conceptual decisions in support of those efforts very early on in his tenure as BLS commissioner. For example, when a staff memo presented the problem of weighting expenditures democratically or plutocratically,104 Commissioner Shiskin decided to go with the plutocratic approach, which “would be best for ‘deflating GNP and component income distribution’ for the national accounts.”105
In embracing the ever-expanding role of the CPI, the Bureau initially intended to replace the CPI-W with a CPI that represented all urban households. In April 1974, the Bureau announced that this change would occur beginning April 1977. The new, broader index would increase population coverage from under 45 percent to approximately 80 percent of the total noninstitutional U.S. population. Immediately upon learning of the Bureau’s plans, national union leaders lobbied Congress against a discontinuation of the CPI-W. As a compromise, in May 1974 the Bureau announced that it would publish an updated CPI-W concurrently with the new Consumer Price Index for All Urban Consumers (CPI-U) for at least 3 years, during which “the comparative movements of these indexes and their components will be studied.”106 At the end of the 3 years, the BLS would reexamine the question of continuing to publish the CPI-W.
Introduction of the 1978 new, improved CPI. The Bureau published the revised CPI-W and the new CPI-U in January 1978. Although an official constant-utility CPI still eluded the agency, the 1978 revision represented a complete revamping of the CPI that affected every input used to compute average changes in the general price level of consumer goods and services. For the most part, the revision reduced the lag between a reference month and the CPI, and made more varied and detailed data accessible to the public.
The revised 1978 CPI utilized three completely redesigned samples to create a new market basket, weight the items therein, and price them in outlets throughout the United States. Expenditure data were based on the 1972–1974 Consumer Expenditure Survey. For the first time, BLS partnered with the U.S. Census Bureau to administer the survey. In a partnership that endures today, BLS developed the questionnaire and data specifications, and relied on the Census Bureau’s expertise to select the household sample and conduct interviews. The Census Bureau was able to reduce the number of areas from which households would be sampled from 366 to 216, representing both rural and urban households. Approximately 40,000 families were interviewed, with 20,000 families participating in quarterly interviews during 1972–1973 and 20,000 families maintaining 2-week diaries of their frequent purchases. The diary survey covered the period from July 1972 to June 1974. Expanding the population coverage of the survey to include 80 percent of the total noninstitutional population decreased the average annual income of the index population from $12,200 to $11,700. The decrease was due to the incomes of the unemployed and those not in the labor force more than offsetting the incomes of salaried and self-employed workers.
For the previous index, areas and types of outlets were chosen by probability sampling, but actual outlets were not selected probabilistically. For the revised 1978 index, the Bureau designed a Point-of-Purchase Survey to obtain the data needed to scientifically select the exact outlets at which items were to be priced. This survey, conducted in 1974, asked approximately 23,000 families where they purchased various goods and services. As with the Consumer Expenditure Survey, BLS teamed with the Census Bureau to administer the new survey of retail outlets.
BLS also redesigned the rent survey in its entirety. Previously, CPI field staff collected rent data semiannually from two (for most cities) or three (for the five largest U.S. cities) subsamples of 500 units in each sample city. Beginning in 1974, BLS incorporated smaller rent samples within each city, but divided each city into six subsamples for semiannual pricing. This change permitted pricing every month, rather than every 2 or 3 months as in the previous rent survey. BLS found that smaller, more frequent sampling “improved the timeliness as well as the accuracy of the rent index.”107
A simplified item structure replaced the fixed market basket of 400 specific items on which the previous CPI was based. The Bureau developed 250 new, general item categories to represent the new market basket. For example, “one category which was formerly ‘piano lessons, organs,’ [would] now [read] ‘fees for lessons—golf, swimming, tennis, piano, etc.’ Also, what used to be ‘movies admissions’ [was] now ‘admissions to movies, theaters, concerts, etc.’”108 The expenditure share of each item within each general category determined that item’s representation within the total market basket. In addition, the Bureau began using a new item selection process within outlets. Called “disaggregation,” the process permitted field staff to use probability sampling techniques to select an exact item for pricing on the basis of general descriptions of a representative item provided by national office staff. The Bureau offered the following example to demonstrate the new disaggregation process:109
[T]he market basket item which was previously “Vitamin D, Grade A Homogenized milk in half-gallon containers” is now “Whole fresh milk.” Through the disaggregation process, the pricing agent selects the specific kind of fresh whole milk that will be priced continuously in each outlet. By this process, each kind of whole milk is assigned a probability, or weight, based on the quantity of it the store sells. If Vitamin D, Homogenized milk in half-gallon containers makes up 70 percent of the sales of fresh whole milk, and the same milk in quart containers accounts for 10 percent of all whole milk sales, then the half-gallon container will have a 7 times greater chance of being chosen than the quart container. After probabilities are assigned, one kind of milk is chosen by an objective selection process based on the theory of random sampling. The particular kind of milk that is selected by disaggregation will continue to be priced each month in that outlet.
Although the number of priced item categories was reduced, using the disaggregation procedure to select specific items for pricing within more broadly defined categories resulted in a more representative mix of goods than could be achieved under previous pricing selection processes. Since the 1978 revision, there have been further modifications to reflect changing consumption patterns, but the general organization of the item structure continues today, so that an individual who used CPI data tables immediately after the 1978 revision could use current CPI data tables with relative ease and familiarity. Also, CPI field economists continue to employ disaggregation when initiating the selection of an item to be priced at a particular outlet for a given period.
Yet another lasting revision of the CPI was the redesign of the area sample, whereby coverage expanded from 56 Standard Metropolitan Statistical Areas (SMSAs) to 85. The expanded sample permitted CPI staff to reduce the number of price quotations substantially, because it represented the urban population at large more accurately. The smaller number of price quotations enabled the staff to price nearly every item more frequently. The percentage of items priced monthly increased to 53 percent of the items in the CPI market basket (from 48 percent), and 41 percent of the items that were previously priced quarterly would now be priced every other month. The more frequent pricing halved the number of days by which the CPI lagged the reference month: the lag decreased from 22 days to 11 days.
For the first time, BLS began publishing, in addition to the 28 already separately published area indexes, population-size class indexes for the Northeast, North Central, South, and West Census Bureau geographical regions. These regional indexes would permit users of the CPI to select, on the basis of geographic location and population characteristics, an index that approximated the user’s inflation experience. Thus, the new area sample not only improved the reliability of the national CPI, but also greatly enhanced CPI data users’ ability to customize data requests.
Limitations of the 1978 CPI. In introducing the revised 1978 CPI, the Bureau identified three conceptual problems that remained unresolved and that would prove to generate much interest in the index in the future. The most obvious unresolved issue was that of population coverage. Prior to the 1978 revision, the CPI represented the experience of an average consumer from just one population group: urban wage earners and clerical workers. Under the 1978 revision, Commissioner Shiskin intended to maintain only one population group, but that group would be all urban consumers. However, negative reaction to the planned discontinuation of the CPI-W persuaded the commissioner to temporarily publish both the CPI-W and CPI-U (with increased funding approved by Congress in the 1976 budget). Declaring that he would revisit the issue no sooner than 1981, Commissioner Shiskin offered the idea of a family of consumer price indexes, “roughly analogous to the array of unemployment data published by the Bureau of Labor Statistics,”110 but he also explained the necessary expansion in inputs (financial and statistical) such an approach would require and conditioned it with the following caveat:
Yet, no single segment of the population is a completely homogenous group; each is made up of many individuals who each have their individual way of life. In any group, a price index would rise more rapidly for some than for others. Therefore, even under the current wage-earner and clerical-worker index, some individuals will gain and some will lose when wages are escalated by the CPI.111
The 1978 revision of the CPI also left the Bureau’s treatment of quality change unresolved. The Stigler Committee had urged the Bureau to adopt methods that were more scientific in order to account for quality change. In particular, the Committee wanted the CPI to make greater use of hedonic regression and demonstrated the use of the method in a staff paper that was included with the main report.112 But the Bureau did not incorporate hedonic regression into the 1978 revision: as stated in a 1977 BLS report, “The BLS is continually researching better methods to measure quality change, but has not as yet developed a viable alternative to the present methods,” which involved using “an indirect method to measure the quality change by evaluating the additional cost associated with producing the change in quality.”113
Lastly, the Bureau left unresolved the issue of measuring changes in the cost of home ownership. Historically, the Bureau viewed the treatment of owner-occupied housing as “a two-tiered problem: (1) The concept under which housing is to be priced, [and] (2) the most accurate and efficient way to measure prices and price change under that concept.”114 Under Commissioner Ewan Clague’s leadership, in 1964 the Bureau rejected the Stigler Committee’s recommendation of adopting a “flow of services” approach in pricing the cost of the services “consumed” by the owner of a house. Later, Commissioner Shiskin attempted to be more accommodating by proposing an alternative to both the contemporaneous method and the Stigler recommendation: “Under [the] proposed alternative, home purchase [would] no longer [carry] an explicit and separate weight in the CPI. The new approach [would] also [reflect] equity costs and appreciation by adjusting house prices by these two elements in the base year and by allowing for future variations in them.”115 However, because of the dearth of empirical support for the proposed alternative approach and the diversity of views concerning the topic, in April 1977 Commissioner Shiskin directed CPI staff to continue to maintain the existing BLS method of pricing home ownership.
The publication of the Price Statistics Review Committee report in 1961, along with that of the 1978 revision, should rightly be viewed as the defining moment that separates two eras of the CPI. Before these events, the index was firmly rooted in the constant-goods conception of a price index. After them, without any fanfare, in adopting the changes in CPI methodology for the 1978 revision, the Bureau demonstrated a willingness to develop a price index that more closely resembled a constant-utility index. To be sure, questions and concerns remained after the revision: the Bureau specifically acknowledged population coverage, quality change, and home ownership as three areas requiring future attention. In addressing these and other, unforeseen concerns in future revisions, the Bureau would come to explicitly adopt the “true constant-utility index” as the conceptual measurement goal in the agency’s future methodological decisions. The remaining history of the CPI is an extension of the debate and changes initiated by the Price Statistics Review Committee.
Toward an ideal cost-of-living index
By 1981, “almost a third of federal expenditure [was] directly linked to the CPI or related price measures, and over half of the federal budget [was] affected if indirectly indexed expenditures [were] added.”116 Small movements in the CPI could dramatically affect government outlays; for example, a 1-percent increase would “automatically trigger nearly $2 billion of additional federal expenditures, at 1981 program levels [approximately $5 billion in 2012 dollars].”117 In addition, Congress passed the Economic Recovery Tax Act of 1981 in August of that year. The act indexed income tax brackets and the dollar amount of the personal exemption to the CPI. With the index affecting both more than half of federal government expenditures and the single largest source of government revenue, researchers and economic policymakers were particularly concerned about unresolved or undiscovered biases in the CPI as a reliable measure of the change in the cost of living in the United States. Indeed, the period since the 1978 revision of the index has been, arguably, the most prolific period of research in consumer price index theory and methods, with BLS researchers often leading the way.
The salience of the CPI and the interest it generated culminated in the formation, in 1995, of the Advisory Commission to Study the Consumer Price Index (hereafter, informally, the Boskin Commission, so named for Stanford University’s Hoover Institution’s Michael Boskin, who chaired the Commission). Congress tasked the Commission with developing a set of recommendations regarding how the Bureau could improve the CPI. Although the Boskin Commission generated a lot of attention directed at the CPI program, many of the findings of the Commission were already known to the Bureau, particularly because of the research conducted by its own economists. Still, the Bureau benefited as a result of the Commission’s study, which compelled Congress to augment BLS resources to the extent necessary to continue the progress toward a more accurate approximation of a cost-of-living index. The ultimate result would be the official publication in 2002 of that “long-sought but elusive cost-of-living index”:118 the Chained Consumer Price Index for All Urban Consumers.
Changes in the treatment of homeownership costs.On July 1, 1972, President Richard Nixon signed into law P.L. 92-336, which established the automatic, yearly adjustment of Old Age, Survivor, and Disability Insurance (i.e., Social Security) benefits based on the CPI, to begin in 1975. That year, the agency’s board of trustees reported that funding for the program could be exhausted as early as 1979. As Joseph P. Goldberg and William T. Moye explain,119
The resurgence of inflation in 1978—about the time [Janet] Norwood became Acting Commissioner—and its acceleration in the following 2 years intensified concern about the rising costs of the indexation process. Attention centered on the CPI, and specifically on its homeownership components. In 1979 and 1980, the Council of Economic Advisers pointed out that shelter costs, which had a substantial weight in the index, were rising faster than most other components of the CPI and questioned the Bureau’s treatment of the purchase of homes and the associated costs of home financing. The Council described the Bureau’s exploration of alternative treatments of this component and the failure of any of these to satisfy major users of the CPI. It suggested that using a rent index to represent the costs of using the services of a house might provide a better measure of changes in the cost of living to the average consumer, particularly in periods of sharp changes in costs of homes and home financing.
By the early 1980s, numerous government agencies had examined and critiqued CPI methods. Among those agencies were the CBO, the General Accounting Office,120 the Senate’s Committee on Governmental Affairs, and two task forces from the House Budget Committee.121 Much of the attention was directed at the Bureau’s treatment of homeownership costs; the CBO critique is representative:122
Because the CPI does not distinguish between a home as an investment and as a source of shelter services, the rapid rise in the asset value of homeownership has been treated as an increase in the cost of living rather than as the increase in wealth that it has been for anyone already owning a home. In addition, homeownership is given an unrealistically large weight in the index because mortgage costs are counted along with the full purchase price. This makes the CPI very sensitive to swings in the mortgage rate of interest.
Well aware of the problems associated with its treatment of homeownership costs in the CPI, the Bureau tried to implement an alternative with the 1978 revision, but could not get its various advisory groups to reach a consensus on which approach to pursue. Once again, the Bureau found itself defending its methods amid controversy that really necessitated a political solution. Testifying before Congress, BLS Commissioner Janet Norwood stated, “It is for the Congress—Congress and the administration—to determine what the purpose of the indexation should be,” and noted that escalation “sometimes produces results that were not anticipated.” She concluded, “Some people would like an index that doesn’t go up so much, and other people would like an index that goes up more. And when they don’t have that which they want, they feel there must be something wrong with the indicator itself.”123
With the various government reports coalescing around adopting a flow-of-services approach in measuring homeownership costs, Commissioner Norwood announced that the Bureau would replace its existing treatment of homeownership costs in the CPI with a rental equivalence alternative. In a detailed report, the Bureau explained that the “rental equivalence approach as incorporated into the CPI attempts to answer the following question: How much rental income do the owners of housing units forego when they choose to occupy the units themselves instead of renting them out? In what follows, this is called the ‘implicit rent.’ It is useful to stress that, although homeowners’ rent is implicit, it is not hypothetical. It is a cost all homeowners actually incur.”124
The change first occurred in the CPI-U in 1983. In passing the Economic Recovery Tax Act of 1981, Congress indexed federal income tax brackets and the dollar amount of the personal exemption to that measure. Indexation would begin in 1985, and in order to have sufficient data to use the new rental equivalence approach, it was necessary to implement the change in the CPI-U no later than 1983. The CPI-W was another story. Because many bargaining agreements were tied to that index, the Bureau delayed the implementation of rental equivalence for the CPI-W until 1985, thereby allowing sufficient time for contract negotiators to assess the impact of the change on the index and alter bargaining agreements accordingly.
In fiscal year 1984, the Bureau received funding for its fifth comprehensive revision program for the CPI. The revised CPI was introduced with the release of January 1987 data. Unlike previous major revisions, the 1987 revision did not include any major methodological changes. The Bureau used data from the 1982–1984 Consumer Expenditure Survey to update the content and weights of the market basket of consumer goods and services. Changes in the area sample were based on data from the 1980 Census of Population and incorporated the new Consolidated Metropolitan Statistical Area definitions. The area sample increased by 6 areas, from 85 to 91, and included 39 new areas. The revision also included an adjustment in how the Owners’ Equivalent Rent index was estimated: “Before 1987, the [O]wners’ [E]quivalent [R]ent index was obtained by simply reweighting rented units in the rent sample to represent all the owners from the same set of blocks. Beginning in 1987, the rate of implicit rent charged for each owner unit in a sample of homeowners is estimated by using a set of rents for housing from the same geographic area and with similar characteristics.”125 The revision also “improved sampling, data collection, data processing, and statistical estimation methods,” and “introduced techniques to make CPI production and calculation more efficient.”126
CPI revisions generated by the Boskin Commission. Close congressional scrutiny of the CPI subsided in the early 1980s after the Bureau implemented the rental equivalence approach and elevated rates of inflation retreated. In addition to completing the 1987 revision, the Bureau continued to address various challenges in aligning CPI methods as closely as possible to what would be dictated by a cost-of-living framework for a consumer price index and in identifying systematic biases in the CPI.
During the early 1990s, BLS research economist Marshall Reinsdorf uncovered a systematic upward bias in the CPI by comparing the rate at which average prices for various food categories were increasing with corresponding CPI food indexes.127 Reinsdorf’s research demonstrated that, for similar food categories, the food indexes tended to increase more rapidly than average prices that the Bureau also published for similar categories. He identified the source of the bias as the way in which the Bureau introduced (or linked) new items into the item sample when updating the outlet sample.128
Fixing this “functional form bias” involved eliminating the correlation between the weight of a newly sampled item and its initial (or link month) price. The Bureau accomplished that task by collecting what are known as overlap samples for a minimum of 3 months before the introduction of new item samples.129 This “seasoning” of the new samples guaranteed that the first price collected was not correlated with the weight or importance the item was given in the sample.
The Bureau first implemented this fix in January 1995 for food items; it extended the use of the procedure to all item categories in June 1996. Initial estimates indicated that the changes for food items would reduce the rate of growth of the overall CPI by 0.11 percent and the extension to all item categories would result in a further reduction in the rate of growth of the overall CPI by another 0.10 percent.130
In December 1993, almost a year after the publication of Reinsdorf’s initial research, the Bureau published a series of articles in the Monthly Labor Review by a number of BLS researchers who were exploring other sources of bias in the CPI relative to an ideal cost-of-living index. Three of the five articles published discussed in detail measurement problems that would become of particular interest a few years later with the formation of the Boskin Commission.
In the first of the three articles, Brent Moulton discussed how Reinsdorf’s identification of “possible problems with the CPI’s Laspeyres-type formula that may cause it to overstate inflation…is yet another manifestation of the effects of substitution by consumers as relative prices change.”131 Moulton concluded that “the geometric mean index represents an alternative formula for estimating basic index components. This formula has many beneficial features. The assumption underlying it is that expenditure shares are constant, so that consumer substitution reduces quantities purchased proportionately with any increase in price.”132
In the second article, Ana Aizcorbe and Patrick Jackman sought to demonstrate that an upward bias was being introduced into the CPI “by keeping quantities fixed and not allowing substitution among goods in response to relative price change.”133 Whereas Moulton looked at consumer substitution within item categories (e.g., golden delicious apples and Mcintosh apples), Aizcorbe and Jackman examined the behavior between item categories (e.g., pork roast and beef roast). By using alternative, superlative index formulas (those which do not hold quantities fixed, thereby measuring consumer substitution behavior) to aggregate past CPI data, Aizcorbe and Jackman could determine the extent of the divergence between the traditional CPI and a superlative index, which can be viewed as closer to a true cost-of-living index. Using the Fisher Ideal and Törnqvist formulas for their study, they found that “the calculated aggregate price indexes were not particularly sensitive to the choice of formulas over the 1982–91 period. A comparison of Laspeyres measures with the calculated superlative indexes provided measures of the substitution effect only slightly higher than those found in earlier studies [in which the effect ranged from 0.1 percentage point to 0.5 percentage point for the years studied].”134
The last of the three articles, by Mary Kokoski, discussed the various quality adjustment approaches that CPI analysts employed and how hedonic regression could be a superior tool in handling quality adjustment in many, but not all, cases. Kokoski states, “It [hedonic regression] thus is almost certainly an improvement over procedures that, at worst, implicitly attribute all of an observed price difference to either pure price change or real quality change or, at best, make imputations based on subjective judgment or averages of price movements for other goods or services.”135 Rather than addressing any specific biases caused by those quality adjustment techniques employed, Kokoski sought to motivate further research in the application of hedonic regression by developing “a set of hedonic regressions for selected categories of food-at-home commodities” to demonstrate “the effects on price, or ‘implicit price,’ of item characteristics, including the types of outlets where items are purchased.”136
In 1994, Mark Wynne and Fiona Sigalla, two economists from the Federal Reserve Bank of Dallas, published an article surveying the current state of research concerning biases in the Consumer Price Index. The article discussed the following kinds of bias: substitution bias, outlet substitution bias, quality bias, new-goods bias, and bias in the treatment of durable goods. Wynne and Sigalla concluded their review by stating, “In view of the paucity of evidence on the various potential biases in the CPI, we are inclined to think that it is better to err on the side of conservatism in guesstimating the size of the overall bias. A figure of less than 1 percent thus strikes us as a plausible estimate of the overall bias.”137 Although the conclusion Wynne and Sigalla reached from their review is important in itself, an examination of the sources listed in the article demonstrates that BLS researchers were often at the forefront in investigating biases in the CPI and in suggesting solutions for eliminating the biases they found (hence the earlier discussion of the Reinsdorf and December 1993 MLR articles).138 Unfortunately, this fact was sometimes lost in the controversy that soon followed.
Days after the 104th Congress commenced its business in 1995, Federal Reserve Chairman Alan Greenspan testified before the House and Senate Budget Committees to discuss issues surrounding deficit reduction. Chairman Greenspan testified that the CPI exaggerated annual inflation by 0.5 percentage point to 1.5 percentage points and that, by changing the way the CPI was calculated, Congress could save the federal government $150 billion over 5 years. A quick fix until the CPI could be changed to eliminate the overestimation of inflation, he offered, was for “Congress to pass a law providing that current laws that adjust benefits based on the Consumer Price Index be amended so that the adjustment would be lower by a percentage point, or half a percentage point.”139 Although BLS researchers were well aware of the upward bias of the CPI relative to a true cost-of-living index and were actively engaged in addressing measurement problems, some in Congress mistook Chairman Greenspan’s comments to mean that the Bureau was ignoring the various causes of that bias and that perhaps the CPI program ought to be transferred to a different federal agency.140 However, instead of removing the program from the Bureau, Congress appointed five economists to form the earlier mentioned Boskin Commission.
The Boskin Commission issued its final report, titled Toward a More Accurate Measure of the Cost of Living, to the Senate Finance Committee on December 4, 1996. The first several chapters provided an overview of how the CPI is constructed, how the federal budget is indexed, and how the CPI “overindexed” (through its upward bias) many federal programs. Following this overview, the Commission addressed what it saw to be the major sources of bias in the CPI and assigned a magnitude to each. The following tabulation summarizes the Commission’s evaluation of biases in the CPI, in percentage points per year:141
Source of bias Estimate Upper-level substitution 0.15 Lower-level substitution .25 New products or quality change .60 New outlets .10 Total bias 1.10 Plausible range .80–1.60
The total bias of 1.10 percentage points was within the range provided to Congress by Chairman Greenspan, and the lower limit of the plausible range of the total bias in the Boskin Commission report coincided with Wynne and Sigalla’s plausible estimate of CPI bias. The Commission’s estimate for bias due to new products or quality change became a topic of contention not only for its magnitude, but also because it was based on little supporting evidence. In response to a letter from Jim Saxton, chairman of the Joint Economic Committee, the Bureau published a piece that contained the following text:142
In putting forward these estimates, the commission does not cite any published or unpublished studies, and indeed they comment on the absence of such evidence. Moreover, the commission does not specify how their estimates were developed in the absence of evidence. In several places the report characterizes the commission's specific estimates of bias as “conservative,” but it generally is not clear why this is believed to be so. The commission’s standard, the cost-of living index, is defined as a function of consumer preferences, so reasonable questions to ask are, “Whose preferences are being described?” and “How were they assessed?” Although economists have methods for drawing inferences about preferences from market data on observed consumer choices, the report does not indicate that the commission used such methods in these cases. Appendix B presents an analysis of two categories, fresh fruits and vegetables and motor fuel, which attempts to quantify the missed consumer benefit or “surplus” that was described by the commission. In both cases this analysis concludes that the commission's estimates overstate the bias.
A family of indexes: addressing the challenge of producing a timely, accurate constant-utility CPI. Following its exposition of the sources of bias in the CPI, the Boskin Commission report offered 13 recommendations for the Bureau to pursue in order to improve the CPI as a measure of consumer inflation, as well as 4 recommendations to the President and Congress to aid the Bureau in pursuing the 13 recommendations. Among the Commission’s recommendations to the President and Congress was one to provide the Bureau with additional resources to expand the Consumer Expenditure Survey and to increase the frequency of the Telephone Point-of-Purchase Survey.143 Of the 13 recommendations directed at the Bureau, the first 4 would be the most influential in providing the CPI program with direction in the years following the report. The first recommendation reiterated the conclusion reached by the Stigler Commission: “The BLS should establish a cost of living index (COLI) as its objective in measuring consumer prices.”144 Combining the next 3 recommendations, the Boskin Commission urged the Bureau to begin to publish only two indexes: a monthly CPI that adopted a superlative index formula to aggregate higher level indexes and an annual cost-of-living CPI that “would use a compatible ‘superlative-index’ formula and reflect subsequent data, updated weights, and the introduction of new goods (with their history extended backward).”145
This last combined recommendation created a challenging question with which BLS leadership had to grapple: How does one balance the call for a more accurate measure of changes in the cost of living with the economic need for a timely measure of inflation that is final when published? As Commissioner Katherine Abraham and her colleagues explained,
To take a recent example, the fact that the CPI is not revised retroactively was one of the principal reasons for its selection by the Department of the Treasury as the index for its inflation-protected bonds, first issued in January 1997. The comments received from individuals, academicians, investment management firms, dealers and institutional investors in response to Treasury’s 1996 Advance Notice of Proposed Rulemaking “indicated a clear consensus that the selected index should be: recognized widely, published frequently, accurate, easily obtainable, easily understood, and not revised retroactively” (Federal Register, September 27, 1996, italics added).146
Also, the Bureau already had been calculating experimental indexes that used a superlative formula at the higher level of aggregation and had found that, although “the timeliness of the CPI might be maintained by using some form of an approximation to a superlative index,” the “commission’s proposed ‘trailing Tornquist’ formula…has been shown to produce price changes that systematically understate the increases in the cost of living, as measured by the superlative formulas.”147 In the end, the Bureau would resolve the competing demands of the CPI program in 2002 with the publication of the Chained Consumer Price Index, but in the meantime the program was nearing the finalization of its sixth major revision, which was implemented for the January 1998 index.148
Most basically, revising the CPI once more involved updating geographic, outlet, item, and housing samples on the basis of the 1990 Census of Population and updating expenditure weights. The updated expenditure weights were based on consumer expenditure data for the years 1993 to 1995 and were introduced in February 1998 for the January 1998 index. In addition, the Bureau redesigned the item structure of the CPI in its entirety, including a revised hospital services item structure and sample that were introduced in February 1997 for the January 1997 index. Then, in June 1998, BLS field agents began pricing the housing sample, using a new computer-assisted data collection system. Lastly, in 1999, addressing another recommendation of the Boskin Commission report, the Bureau began shifting CPI methodology from area sample rotation to item category rotation, which would “facilitate accelerated sample rotation in product areas where markets are most dynamic.”149
Research aimed at improving the CPI in the direction of a closer approximation to a cost-of-living index continued in parallel with the 1998 revision. Indeed, after years of research (recall Moulton’s article in the December 1993 issue of the Monthly Labor Review), in October 1998 the Bureau announced that it would incorporate a geometric mean formula for calculating most basic components of the CPI beginning in January 1999. Initially proposed in 1993 by Moulton as a fix to the functional form bias present in the CPI at the time, a geometric mean formula subsequently was included in the Boskin Commission’s list of recommendations, but instead as way to address substitution bias at the lower level of aggregation. In explaining the phenomenon of lower level substitution to the public, the Bureau posited a scenario in which a consumer was offered five options in response to an increase in price of a particular brand of ice cream at a particular store. The consumer could substitute (1) among brands of products, (2) among sizes of products, (3) the same item among different outlets, (4) across time (purchasing the same item during a different time of the month), or (5) among types of items within the same CPI category (e.g., from ice cream to frozen yogurt).150 Not all item categories that compose the CPI can be expected to exhibit much, if any, consumer substitution behavior within the category, so the switch to the geometric mean formula did not pertain to all basic index categories. When implemented, the formula affected categories representing “approximately 61 percent of total consumer spending” in the CPI and was expected to “reduce the annual rate of increase in the CPI by approximately 0.2 percentage point per year.”151
In 2002, the Bureau implemented a number of improvements in the CPI to further the goal of estimating the change in the cost of living more accurately. There were three related improvements: extending the use of the computer-assisted data collection system in pricing all commodity and service categories, increasing the sample size of the Consumer Expenditure Survey, and beginning biennial weight updates. Together, these improvements would help ensure that the CPI reflected consumers’ current allocation of spending, more accurately capturing the inflation actually experienced by urban consumers.152 As alluded to earlier, the Bureau also began publishing the Chained Consumer Price Index in August 2002, with data beginning in January 2000. (Expansion of the Consumer Expenditure Survey sample also served to accommodate the needs of producing the Chained Consumer Price Index with monthly expenditure estimates.)153
The Bureau opted to keep publishing the modified Laspeyres CPI instead of replacing it with the Boskin Commission’s suggestion of a monthly index calculated with the use of a superlative index formula. Adopting a more inclusive strategy, the Bureau added another index to the “family” of CPIs (at the time consisting of the CPI-W and the CPI-U) and let users decide which index would best suit their needs. Thus, rather than publishing the Chained Consumer Price Index as an annual cost-of-living index, a Boskin Commission recommendation, the Bureau would publish a monthly Chained Consumer Price Index in three versions: an initial version contemporaneously with the current-year CPI, a revised interim version the following February, and a final version the second February following publication of the initial index. For example, in 2003 all monthly Chained Consumer Price Indexes for that year were published as initial index numbers, in February 2004 all monthly initial 2003 Chained Consumer Price Indexes were then published as revised interim index numbers, and in February 2005 the final index numbers were published for all 2003 monthly Chained Consumer Price Indexes.
In final form, the Chained Consumer Price Index aggregates 8,018 elementary indexes (211 elementary item categories × 38 areas), using the Törnqvist superlative index formula and monthly weights from the current month and the previous month.154 The benefit of the Törnqvist formula is that it captures actual observed consumer substitution behavior by using current- and previous-month expenditures to weight the elementary indexes when they are aggregated.155
Because current expenditure data required to aggregate basic indexes with the use of the Törnqvist formula are unavailable when the initial and interim Chained Consumer Price Indexes are calculated, the Bureau “opted to select an aggregation methodology for the [i]nitial and [i]nterim versions of the [Chained Consumer Price Index] that would best predict the [f]inal [Chained Consumer Price Index] Tornqvist version—constrained by the use of the most contemporaneous expenditure data available at the time of index publication….An adjusted Geometric Mean Index formula was ultimately adopted.”156 In selecting a geometric mean formula to aggregate basic indexes, the Bureau assumed that consumers would hold constant, at least over short periods, the share of expenditures devoted to purchasing a given type of item. In other words, if a consumer devoted a certain percentage of his or her income to purchasing apples, then, if apple prices increased, the consumer would maintain that percentage by purchasing fewer apples at the new, higher price. If apple prices decreased, the expenditure share devoted to apples would remain the same if the consumer now purchased more apples at the lower price. Empirical studies demonstrated that this aggregation approach biased the initial and interim indexes so that they were slightly below the final Chained Consumer Price Index; therefore, the Bureau decided to adjust price changes calculated with the geometric mean formula by an adjustment factor that “represents the historically observed difference between the Tornqvist and Geometric Mean upper-level aggregation of CPI elementary indexes.”157 On the basis of multiple simulations and actual observed divergences, the Bureau stated that “it is reasonable to expect that during the next few years CPI-U inflation estimates will exceed those of the [Chained Consumer Price Index] by more than 0.2 percentage point”;158 as things turned out, the difference was approximately 0.25 percentage point from December 2001 to December 2012. Table 2 contrasts the Chained Consumer Price Index with the CPI-U.
|Category||CPI-U||Chained Consumer Price Index for All Urban Consumers|
Lower level index formula
Upper level index formula
|Laspeyres||Adjusted geometric mean||Adjusted geometric mean||Törnqvist|
|Biennial, lagged 2–3 years||Biennial, lagged 2–3 years||Biennial, lagged 1–2 years||Previous month|
|Biennial, lagged 3–4 years||Biennial, lagged 3–4 years||Biennial, lagged 2–3 years||Previous month|
Frequency of weight update
|1-month lag||1-month lag||2- to 13-month lag||14- to 25-month lag|
Source: “An Introductory Look at the Chained Consumer Price Index” (Bureau of Labor Statistics, March 14, 2002), https://www.bls.gov/cpi/additional-resources/chained-cpi-methodology.pdf.
In 2004, a final, major improvement in the estimation of the CPI involved expanding the collection of price data to all business days of the month. Previously, price data were collected during the first 18 business days of the month from January to October and during the first 15 business days in November and December. Expanding data collection to all business days would decrease any bias that resulted by not collecting data when prices could change at the beginning or ending of a sale held outside of the first 15 or 18 business days of the month.
Summary: 100 years of the CPI
In covering the evolution of the CPI since its inception as a World War I wartime statistic, this article has sought to explain the major technical changes that gradually modified the index from being a statistic that measured consumer inflation for a specific population (urban wage earners in major shipbuilding centers) and a specific purpose (wage-setting policy during World War I) to a generalized index that serves many roles for a broad array of users. Not every change in the CPI is covered here; that is evidenced by the fact that the history presented ends in 2004. Still, the article covers the details necessary for understanding how the CPI-W, CPI-U, and Chained Consumer Price Index are currently estimated and why particular methods are employed in calculating the estimates. A second motivation behind presenting this history has been to demonstrate that the decisions made are not always solely technical ones. The Bureau operates within the parameters set by Congress, meaning that Congress determines for whom the CPI should estimate consumer inflation and for which uses the CPI will be employed at the federal level of government. In addition, the CPI has numerous uses that are outside the purview of the federal government.
In response to an ever-expanding number of users, the Bureau has developed a family of CPIs, including a number of experimental measures. Even this approach, however, will not totally resolve the sometimes conflicting objectives of the various users of CPI data. For, in reality, each consumer unit (be it a household or an individual) purchases a unique mix of goods and services, and uniquely responds to changes in the relative prices of the goods and services in this basket. Therefore, each consumer unit would technically need a unique CPI to capture its inflation experience over time. It follows that any CPI which approximates the average change in prices for an average consumer of a particular population (e.g., elderly consumers) will have inherent limitations.
The Bureau also must weigh the sometimes conflicting goals of timeliness, relevance, and precision. What good would a CPI serve if, to adopt particular theoretically preferred methods, the index can be published only 2 years after the fact and is difficult for all but academic economists and statisticians to understand? When contemplating a change to the CPI, the Bureau has placed slightly greater emphasis on timeliness and relevance in order to have a CPI that is as broadly understood and accepted as possible. Commissioner Abraham and colleagues may have summed it up best:
The BLS is intensely aware of the sensitive nature of the data it produces, and of the critical need for these data to be as accurate as possible. Although we believe that we can make important improvements in the CPI, we do not believe it to be possible to produce a perfect cost-of-living measure. It is, in fact, commonplace to observe that there is no single best measure of inflation. It is evident that the expanding number of users of the CPI have objectives and priorities that sometimes can come into conflict. The BLS response to this situation has been to develop a “family of indexes” approach, including experimental measures designed to provide information that furthers assessment of CPI measurement problems, or to focus on certain population subgroups, or to answer different questions from those answered by the CPI. All of these measures are carefully developed but have their own limitations. Those who use the data we produce should recognize these limitations and exercise judgment accordingly concerning whether and how the data ought to be used.159
Darren Rippy, "The first hundred years of the Consumer Price Index: a methodological and political history," Monthly Labor Review, U.S. Bureau of Labor Statistics, April 2014, https://doi.org/10.21916/mlr.2014.13.
10 “Wages and cost of living,” Bulletin of the Bureau of Labor, Bulletin No. 53 (U.S. Bureau of Labor, 1904).
11 Goldberg and Moye, The first hundred years, p. 37.
12 Thomas A. Stapleford, The cost of living in America: a political history of economic statistics, 1880–2000 (New York: Cambridge University Press, 2009), pp. 53–55.
13 Ibid., p. 87.
14 Goldberg and Moye, The first hundred years, p. 91.
15 Ibid., p. 103.
16 Stapleford, The cost of living in America, p. 85.
17 Ibid .
18 “Tentative quantity–cost budget necessary to maintain family of five in Washington, D.C.,” Monthly Labor Review, December 1919, pp. 22–29, quote from p. 25.
20 Goldberg and Moye, The first hundred years, p. 106.
21 Ibid., p. 107.
22 The Consumer Price Index, pp. 2–3.
23 National Bureau of Economic Research.
24 Stapleford, The cost of living in America, p. 169.
25 Ibid,, p. 148.
27 Ibid., p. 171.
28 See Faith M. Williams, Margaret H. Hogg, and Ewan Clague, “Revision of index of cost of goods purchased by wage earners and lower-salaried workers,” Monthly Labor Review, September 1935, , p. 819, note 2, for a description of some of the immediate steps taken in beginning the modernization of the cost-of-living program under Secretary Perkins.
29 Ibid., p. 822.
30 Ibid., p. 826.
31 Ibid., p. 820.
32 Ibid., p. 828.
33 Goldberg and Moye, The first hundred years, pp. 149–150.
34 Ibid., p. 171; see also Stapleford, The cost of living in America, pp. 173–174.
35 Apparently, presenting new sought-after programs in this manner was common practice by federal statistical administrators during the New Deal. (See Stapleford, The cost of living in America, p. 165.)
36 Ibid., p. 165.
37 Ibid., pp. 177–178; no specific threshold number was given.
38 Ibid.; no specific dollar amount was given.
40 Ibid., p. 178.
42 Faith M. Williams and Stella Stewart, Changes in cost of living in large cities in the United States, 1913–41, Bulletin 699 (U.S. Bureau of Labor Statistics, 1941), p. 34. It could not be determined from this bulletin or from Faith M. Williams, Money disbursements of wage earners and clerical workers, 1934–36, summary volume, Bulletin 638 (U.S. Bureau of Labor Statistics, 1941), whether “Mexican” referred only to families of Mexican heritage or whether the term was applied to all families of Hispanic heritage.
43 Stapleford, The cost of living in America, p. 178.
44 Williams and Stewart, Changes in cost of living in large cities, p. 2.
45 Ibid., p. 16.
46 Ibid., p. 7.
47 Ibid., p. 6.
49 Stapleford, The cost of living in America, p. 311.
50 Williams and Stewart, Changes in cost of living in large cities.
51 Faith M. Williams, “Bureau of Labor Statistics Cost-of-Living Index in wartime,” Monthly Labor Review, July 1943, pp.82–95, quote from p. 82.
52 Ibid., p. 85.
54 Ibid., p. 89.
55 Battle stations for all: the story of the fight to control living costs (U.S. Office of War Information, 1943).
56 Stapleford, The cost of living in America, p. 187.
57 For a thorough discussion of labor’s opposition to the Bureau’s cost-of-living program, see Stapleford, The cost of living in America, chapters 5 and 6.
58 Williams, Bureau of Labor Statistics Cost-of-Living Index, p. 82.
59 Goldberg and Moye, The first hundred years, p. 154.
60 Frederick C. Mills, E. White Bakke, Reavis Cox, Margaret G. Reid, Theodore W. Schultz, Samuel Stratton, Dorothy S. Brady, and Solomon Fabricant, “An appraisal of the U.S. Bureau of Labor Statistics cost of living index,” Journal of the American Statistical Association, Volume 38, no. 224, 1943, pp. 387–405, quote from p. 389.
61 Ibid., p. 388.
62 Goldberg and Moye, The first hundred years, p. 155.
64 Stapleford, The cost of living in America, p. 200.
65 As recounted in Stapleford’s The Cost of Living in America, just 2 weeks prior to the release of the report AFL Representative Meany appeared unannounced at the fifth annual conference of union researchers (which he was not) and BLS staff and offered the following comments: concerning Secretary Perkins, “The Secretary of Labor does not represent the views of labor in the Cabinet”; on Commissioner Hinrichs, “a bureaucratic monkey on a stick who moves up and down in conformity with the dictates of administrative wage policy”; on the Bureau in general, “prostituted its research functions,” “obsequiously acquiesced” to administrative orders to use the index to keep down wages, and was “no longer a free agency of statistical research.” (See page 243 for a more thorough account of the events at the meeting.)
66 Goldberg and Moye, The first hundred years, p. 157.
67 Stapleford, The cost of living in America, p. 244.
68 Goldberg and Moye, The first hundred years, p. 157.
69 Emil D. Schell and Laura Mae Webb, “Cost of living and retail prices: cost of living in large cities, May 1944,” Monthly Labor Review, July 1944, pp. 179–182, quote from p. 180.
70 Stapleford, The cost of living in America, p. 250.
71 Goldberg and Moye, The first hundred years, p. 178.
72 Stapleford, The cost of living in America, states that 1946 produced the greatest wave of strikes in American history. (See p. 253.)
73 Ibid., p. 253.
74 Henry Lowenstern, “Adjusting wages to living costs: a historical note,” Monthly Labor Review, July 1974, pp. 21–23; see especially p. 23.
75 Ibid., p. 25.
76 Stapleford, The cost of living in America, p. 260.
77 Goldberg and Moye, The first hundred years, pp. 193–194.
78 “The main features of the revised Consumer Price Index,” Monthly Labor Review, February 1953, pp. 162–165, quote from p. 162.
79 Goldberg and Moye, The first hundred years, p. 229.
80 “The main features,” pp. 164–165.
81 Goldberg and Moye, The first hundred years, p. 196.
82 Stapleford, The cost of living in America.
83 Staff report on employment, growth, and price levels, 86th Cong., 1st sess. (U.S. Congress, Joint Economic Committee, December 24, 1959), p. 109, quoted in Stapleford, The Cost of Living in America, pp. 308–310.
84 Price Statistics Review Committee, “The Consumer Price Index,” in The price statistics of the federal government (New York: National Bureau of Economic Research, 1961), pp. 51–60, quote from p. 51.
85 Ibid., pp. 52, 55.
86 Ibid., p. 56.
87 Ibid., p. 57.
88 Ibid., p. 58.
89 Ibid., p. 55.
90 The Laspeyres type of CPI applies expenditure weights from a previous period to current-period prices. The index equation is , where represents the Laspeyres ratio (or index), pt is the current price of an item, p0 is the price for the same item during an initial sampling period, and q0 is the quantity of the item purchased during the initial period. Notice that the Laspeyres formula holds the quantity fixed over future time. The formula answers the question “How much more (or less) would it cost to purchase the same initial basket of goods at today’s prices relative to prices in the initial sampling period?” By contrast, the Paasche type of CPI would apply current-period expenditure weights to previous-period prices. The equation is , where represents the Paasche ratio (or index), pt and p0 are as before, and qt is the quantity of the item purchased in the current period. Notice that the Paasche formula also holds the quantity fixed but over past time. The formula answers the question “How much more (or less) does it cost to purchase the same current basket of goods at today’s prices if one could have purchased that basket of goods at prices which existed at some time in the past? This index can be calculated only ex post facto, because of the lag in time needed to collect and analyze expenditure data. The geometric mean of the Laspeyres and Paasche indexes is known as the Fisher Ideal index and is given by . The opposing inherent biases of both indexes, upward for the Laspeyres and downward for the Paasche, result in an estimate that is not systematically biased in either direction from the true change in the cost of living.
91 Phyllis Groom, “New features of the revised CPI,” Monthly Labor Review, April 1964, pp. 385–390, quote from p. 385.
92 “Major features of the revision program, 1959–1964,” The Consumer Price Index: History and Techniques, Bulletin 1517 (U.S. Bureau of Labor Statistics, 1964), pp. 13–18, quote from p. 15..
93 Royal Meeker, “On the best form of index number,” Quarterly Proceedings of the American Statistical Association 17, no. 135 (1921), p. 912, quoted in Stapleford, The cost of living in America, p. 113.
94“Statistical and conceptual structure of the revised Consumer Price Index,” The Consumer Price Index: History and Techniques, pp. 19–24, quote from p. 19.
95 Ibid., p. 21.
96 Geoffrey H. Moore and Maxine Stewart, “New developments in labor statistics,” Monthly Labor Review, March 1972, pp. 3–13, quote from p. 5.
98 Milton Friedman, “Statement on indexing as a tool for economic stabilization,” Subcommittee on Production and Stabilization of the Committee on Banking, Housing, and Urban Affairs, U.S. Senate, 93rd Congress, June 10, 1974, https://miltonfriedman.hoover.org/friedman_images/Collections/2016c21/Gov_06_10_1974.pdf.
99 Julius Shiskin, “Updating the Consumer Price Index—an overview,” Monthly Labor Review, July 1974, pp. 3–20, quote from p. 3.
100 Stapleford, The cost of living in America, pp. 347–348.
102 Shiskin, “Updating the Consumer Price Index,” p. 3.
103 Indexing with the Consumer Price Index: problems and alternatives (Congressional Budget Office, June 1981), see especially pp. 23–24, 26–30.
104 A democratic index weights each household’s expenditures equally, whereas a plutocratic index weights each household’s expenditures proportionally. Thus, in a plutocratic index, the inflation experience of those households which spend more relative to others has more influence on the overall index.
105 Stapleford, The cost of living in America, p. 347.
106 Shiskin, “Updating the Consumer Price Index,” p. 14.
107 The Consumer Price Index: concepts and content over the years, Report 517 (U.S. Bureau of Labor Statistics, 1977), see especially pp. 8–9.
108 Ibid., p. 7.
110 Ibid., p. 11.
111 Ibid., p. 12; see also Shiskin, “Updating the Consumer Price Index,” p. 14.
112 See Zvi Griliches, “Hedonic price indexes for automobiles: an econometric analysis of quality change,” in The price statistics of the federal government (New York: National Bureau of Economic Research, 1961), pp. 173–196.
113 The Consumer Price Index, p. 12.
114 Ibid., p. 13.
115 Ibid., p. 14.
116 Indexing with the Consumer Price Index: problems and alternatives (Congressional Budget Office, June 1981), p. xiii.
118 Geoffrey H. Moore and Maxine Stewart, “New developments in labor statistics,” Monthly Labor Review, March 1972, pp. 3–13, quote from p. 5.
119 Goldberg and Moye, The first hundred years, pp. 230–231.
120 See Measurement of homeownership costs in the Consumer Price Index should be changed (U.S. Government Accountability Office, April 1981).
121 Stapleford, The cost of living in America, p. 375.
122 Indexing with the Consumer Price Index, p. xiii.
123 Goldberg and Moye, The first hundred years, p. 232.
124 “Changing the homeownership component of the Consumer Price Index to rental equivalence,” CPI detailed report (U.S. Bureau of Labor Statistics, January 1983), pp. 7–11, quote from p. 8.
125 The Consumer Price Index: 1987 revision, Report 736 (U.S. Bureau of Labor Statistics, January 1987), p. 7.
127 Marshall Reinsdorf, “The effect of outlet price differentials on the U.S. Consumer Price Index,” in Murray F. Foss, Marilyn E. Manser, and Allan H. Young, Price measurements and their uses, (Chicago: University of Chicago Press, 1993).
128 In 1981, the Bureau began rotating one-fifth of the outlet and item samples on a yearly basis, so that an entirely new sample would have been introduced every 5 years.
129 See Paul A. Armknecht, Brent R. Moulton, and Kenneth J. Stewart, “Improvements to the Food at Home, Shelter, and Prescription Drug Indexes in the U.S. Consumer Price Index,” BLS Working Paper 263 (U.S. Bureau of Labor Statistics, February 1995).
130 Ken Stewart, “Extending the improvements in CPI sample rotation procedures and improving the procedures for substitute items,” CPI detailed report (U.S. Bureau of Labor Statistics, March 1996), pp. 1–99, quote from p. 5.
132 Ibid., p. 22.
133 Ana M. Aizcorbe and Patrick C. Jackman, “The commodity substitution effect in CPI data, 1982–91,” Monthly Labor Review, December 1993, pp. 25–33, https://www.bls.gov/opub/mlr/1993/12/art3full.pdf.
134 Ibid., p. 30.
136 Ibid., p. 45.
137 Mark A. Wynne and Fiona D. Sigalla, “The Consumer Price Index,” Economic Review (second quarter, 1994), pp. 1–22, quote from pp. 17–18.
138 Arguably, the creation of the BLS Division of Price and Index Number Research is the most consequential outcome of the Stigler Commission’s report.
139 Adam Clymer, “The 104th Congress: the overview; as parties skirmish over budget, Greenspan offers a painless cure,” The New York Times, January 11, 1995, p. A1, http://search.proquest.com/docview/430011416?accountid=46300.
140 For example, on January 14, 1995, House Speaker Newt Gingrich told attendants at a town meeting in Kennesaw, Georgia, “We have a handful of bureaucrats who, all professional economists agree, have an error in their calculations. If they can’t get it right in the next 30 days or so, we zero them out, we transfer the responsibility to either the Federal Reserve or the Treasury and tell them to get it right.” (See John M. Berry, “GOP leaders join challenge to price index; refiguring inflation could reduce deficit,” The Washington Post, January 16, 1995, http://search.proquest.com/docview/307849400?accountid=46300.) During a presentation to the American Statistical Association on August 6, 1996, in Chicago, then Commissioner Katharine Abraham responded to Speaker Gingrich’s comments: “If there were problems with the CPI that Bureau of Labor Statistics (BLS) staff knew about and knew how to fix but were just refusing to address, I’d agree with the Speaker: he should zero us out. That is not, however, an accurate characterization of the BLS performance. Indeed, as other speakers have indicated in their comments—and I would like to express my appreciation for their kind words—the staff of the BLS have been at the forefront of trying to identify problems with the way that the CPI is put together, figuring out how to fix those problems, and making improvements in the index.” (See Katharine G. Abraham, “Statistics in the spotlight: improving the Consumer Price Index: statement,” paper presented at a meeting of the American Statistical Association, Chicago, IL, August 6, 1996, https://www.bls.gov/news.release/cpi.br12396.a09.htm.)
141 Final Report of the Advisory Commission to Study the Consumer Price Index(Advisory Commission to Study the Consumer Price Index, December 1996), p. 44.
142 Measurement issues in the Consumer Price Index (U.S. Bureau of Labor Statistics, June 1997), https://data.bls.gov/cgi-bin/print.pl/cpi/cpigm697.htm.
143 The Telephone Point-of-Purchase Survey (TPOPS) began in 1998, replacing the Point-of-Purchase Survey (POPS). As with the POPS, the Census Bureau administers the TPOPS. The agency employs a random-digit dialing technology to select households for participation. The data collected from the TPOPS is used to scientifically select the outlets at which economic assistants collect prices.
144 Final report of the Advisory Commission, p. 2.
146 Katharine G. Abraham, John S. Greenlees, and Brent R. Moulton, “Working to improve the Consumer Price Index,” Journal of Economic Perspectives, Winter 1998, p. 30, note 5.
147 Measurement issues, p. 9.
148 See John S. Greenlees and Charles C. Mason, “Overview of the 1998 revision of the Consumer Price Index,” Monthly Labor Review, December 1996, pp. 3–9, https://www.bls.gov/opub/mlr/1996/12/art1full.pdf.
149 Ibid., p. 8.
150 Kenneth V. Dalton, John S. Greenlees, and Kenneth J. Stewart, “Incorporating a geometric mean formula into the CPI,” Monthly Labor Review, October 1998, pp. 3–7; see especially p. 5.
151 Ibid., p. 3.
152 “Future schedule for expenditure weight updates in the Consumer Price Index” (U.S. Bureau of Labor Statistics, December 18, 1998), https://data.bls.gov/cgi-bin/print.pl/cpi/cpiupdt.htm.
153 Robert Cage, John Greenlees, and Patrick Jackman, “Introducing the Chained Consumer Price Index,” paper presented at the Seventh Meeting of the International Working Groups on Price Indices, Paris, France, May 2003; see especially p. 10, https://www.bls.gov/cpi/additional-resources/chained-cpi-introduction.pdf.
154 For the formulas used to calculate the different versions of the Chained Consumer Price Index, see “An introductory look at the Chained Consumer Price Index” (U.S. Bureau of Labor Statistics), https://www.bls.gov/cpi/additional-resources/chained-cpi-introduction.pdf.
155 This fact is worth noting because critics of the Chained Consumer Price Index often believe that the Bureau manipulates the data it collects to artificially force that index to display “assumed” substitution behavior. Thus, argue the critics, the Bureau artificially creates the divergence between the Chained Consumer Price Index and the CPI-U.
156 Cage, Greenlees, and Jackman, “Introducing the Chained Consumer Price Index,” p. 18.
157 Ibid., p. 27.
158 Ibid., p. 37. From 2000 to 2011, the mean difference was 0.28 percentage point. Excluding 2000, when the difference was 0.8, the mean difference was 0.23 percentage point.
159 Abraham, Greenlees, and Moulton, “Working to improve the Consumer Price Index,” p. 35.