Impacts of COVID-19 on collection and missing data in the CPI

The COVID-19 pandemic presented unique challenges to the U.S. Bureau of Labor Statistics (BLS) in compiling the Consumer Price Index (CPI).¹ In March 2020, in-person data collection ceased almost overnight at brick-and-mortar stores and switched to online collection, increasing the rate of survey nonresponse and missing data in the CPI surveys. To monitor the impacts of these issues on the CPI, BLS developed and began releasing new metrics each month.² This article takes a closer look at some additional measures of the impact of the COVID-19 pandemic on the CPI’s Commodity and Services (C&S) survey.³ First, I examine price changes associated with changing the mode of collection from in-person to online. Next, I evaluate an alternative imputation method for when no sample is present in an elementary item-area. Then, I show a measure of data completeness as an extension to the existing measures that are used to determine index eligibility for publication. Finally, I offer the rate at which products were missing from store shelves and websites, an indication of pandemic-related shortages.

The effects of collection modes on prices and data quality

When BLS selects the set of outlets (stores) from which it will track prices, BLS assigns each outlet with either a specific physical location or internet address.⁴ Each product or service that is observed and priced over time for use in the CPI is called a “quote.” Prior to the pandemic, BLS data collectors gathered most prices by visiting a physical store location. In March and April 2020, most data collection abruptly moved to websites (as shown in chart 1) and only in the second half of 2022 began to return to physical locations in earnest. Chart 2 shows how many quotes switched collection mode in each month.

During March and April 2020, price changes in outlets in which the mode of collection changed from personal visits to online collection were reflected in the CPI.⁵ The price change resulting from switching to online collection varied by item category and outlet and cannot be separated from the price change that would have been observed had data collection continued in the physical locations. Price changes caused by the change in collection mode could result from two cases: the same exact product being sold at a different price online than in physical stores; or if the exact product was unavailable and a replacement product was selected and the price change (possibly adjusted for quality differences) between the two was shown in the price index.

To estimate the impact of the switch to online collection, I estimate research indexes to compare the price change for quotes in which the mode of collection changed with those that did not change.⁶ To do so, I classify each observation used in the calculation of the CPI by its collection mode during the current month and the previous month. For example, a price could have been collected by personal visit during February 2020 and then switched over to collection via the store’s website in March 2020. The two collection patterns of interest are prices that were collected the same way in both periods and prices that were collected by personal visit in the earlier period and from a website during the later period.⁷ I exclude quotes that were off cycle, previously deleted, or carried forward.⁸ Within each combination of item-area and collection pattern, I compute an unweighted geometric average. To aggregate across areas, I take an unweighted arithmetic average over all the item-areas for each pattern. I perform no imputation; if an outlet did not change status in the manner described (collected by personal visit in both periods or collected by personal visit in the earlier period and from a website during the later period), I remove it from the calculations. Chart 3 presents the research price index for quotes that switch from in-person to online collection versus those that were collected online in both periods.

Generally, steady or declining prices are observed with the change in collection method from in-person to online collection. However, pricing food online rather than in person was associated with higher prices. The results are presented in table 1.

As personal-visit collection resumed, BLS used the difference between the prepandemic price and the price after in-person visits resumed to calculate the price change over that period.⁹ In the preceding analysis, I do not attempt to compute the price change associated with the return to in-person data collection for several reasons. First, the shift from in-person to online data collection occurred over 2 months, whereas the return to data collection in stores is occurring over many months. Therefore, in any given month, relatively few quotes are changing collection modes, and this transition is still ongoing, as shown in chart 1. Second, it would also be difficult to separate the effects of the inflationary trend in late 2021 and 2022 from any price change associated with changing the data collection from online to in-person. Third, routine sample rotation and product replacements would cause a relatively small number of cases in which the same unique product at the same physical location of the same store could be observed in early 2020 and again in late 2022.

Missing data

The CPI C&S survey has imputation procedures to estimate price change when a price is unavailable. Although the CPI surveys experienced a reduction in the number of collected prices during the COVID-19 pandemic, BLS did not change the CPI imputation procedures.¹⁰

CPI background and traditional missing-data process

The CPI is structured into a set of basic (or elementary) item categories, like bread, and basic (or elementary) geographic areas, for example Philadelphia. There are 243 item categories and 32 areas yielding 7,776 basic item-area cells, e.g., bread in Philadelphia. When no prices are collected for an entire basic cell, the price change for the cell needs to be imputed. Although the need for this imputation is relatively rare, it does occur in item categories in which gaining cooperation from the survey respondents is difficult or unusually burdensome. In recent years, the frequency of missing cells increased (as shown in table 2). The practice BLS uses in the CPI is to impute the price change for the missing basic item-area cell from the price change of the same basic item in a nearby basic area. For example, the basic cell for bread in Philadelphia would be imputed from the basic cell for bread for smaller cities in the mid-Atlantic. An alternative approach would be to impute the price change by first computing the item category price change at the national level excluding any missing areas. Then one could impute the missing cell with the value of the category at the national level and recompute the national estimate based on all the areas.¹¹ A potential downside is that this would dampen variance because missing price changes would be replaced by the average price change. Although the impact of nationally imputed values is generally small, nationally imputing prices can have a larger impact in item categories with small samples or respondent burden issues. For example, the impact on the all-items U.S. city-average level is negligible, but domestic services has a small sample, so the impact of the alternate imputation approach is more noticeable, as shown in charts 4 and 5.

The biggest difference when one imputes the missing cells with national data is in food at employee sites and schools. However, during the COVID-19 pandemic many school lunches were free, which introduced unusually large price changes to this item category. Even with these extreme price changes in food at employee sites and schools, the difference between the official index and the nationally imputed index is around 1.4 percentage points per year (see chart 6).

Differences in index values are one way to look at alternative imputation methods. Another possibility is to count how many cells are imputed, as shown in table 2. In the months that had the most imputations, the number of item-areas without sample increased from 2020 through mid-2023. The pandemic forced some delays to the sample rotation process in which BLS selects a new set of stores from which to obtain prices. Generally, approximately one-eighth of the sample rotates every 6 months. Table 2 shows the research index 1-month percent change minus the official index 1-month percent change for each basic cell.¹² I then take the median difference over the basic item-area cells with missing values in each month. The differences tend to be small (see table 3). The increase in missing item-areas from 2020 through 2023 can be at least partially explained by operational obstacles. Gaining respondent cooperation and initiating products is more challenging without a personal visit and can often cause delays.

Density, adequacy, and the impact on publication

Whether BLS publishes an index depends on the “adequacy” of the index. A cell is adequate if it has at least one collected price and inadequate if it has no collected prices. Adequacy for an aggregate index is a weighted arithmetic average of its constituent basic cells’ adequacies, where the weights are the expenditure shares of each basic item-area cell to the aggregate item-area. All adequacy values are between 0.0 and 1.0, and for an aggregate index to be published, its weighted adequacy must exceed 0.5.¹³

Even with the difficulty in collecting data during the pandemic, BLS has maintained publication at the U.S. city-average level. From January 2020 through July 2023, there was a small drop in the number of published not seasonally adjusted indexes at the U.S. city-average level (see chart 7).

A density approach to publication criteria

There are several approaches for handling heavily imputed indexes.¹⁴ The approach used by Eurostat and a similar one used by the United Kingdom’s Office of National Statistics (ONS) depends on the share of data successfully collected. The ONS imputes an index if more than 20 percent of the data are missing, and both agencies flag indexes in which more than 50 percent of the weight of the index is imputed. In both cases, the indexes are published, unlike in the United States, where indexes that fail to reach an adequacy of 0.5 are withheld from publication. The dramatic effect of COVID-19 on BLS ability to collect data presents an opportunity to see what a share-of-data approach to publication might have looked like in the CPI. For this analysis, I will call this approach the “density” approach to publication. BLS has no current plans to change to using a density measure as a publication criterion and, outside of the pandemic period, these two approaches yield similar publication results.

Index density is a weighted percentage of quotes that were successfully collected. This measure has a stronger connection to the amount of data collected than a traditional adequacy measure. For example, suppose there are 100 quotes in a basic item category-area. The basic cell adequacy is the same whether 1 or 100 quotes are collected, but the density measure will increase as the sample moves from 1 quote to 100 quotes. The basic-cell densities can then be aggregated in the same manner as that for adequacy.

Results for density

As shown in chart 8, under both criteria (density and adequacy) the all-items index would meet publication criteria.¹⁵ The new information here is that the density graph shows more clearly the reduction in collected prices at the beginning of the pandemic, followed by a partial recovery.

One can also look at density for each major group. As chart 9 shows, density drops in the food and beverages, other goods and services, and recreation major groups, but the education and communication major group was steady over the period studied.

Next, I compare density and adequacy for some specific categories that experienced notable decreases in adequacy during the pandemic. One of the more high-profile examples is leased vehicles (see chart 10). In leased vehicles, both the adequacy and density drop. In this case, the density measure is not providing much information about data quality that was not already present in the traditional adequacy statistic. Generally, there is a strong correlation between the existing adequacy measure and this new density measure. At the U.S. city-average level for each basic item category from January 2020 to July 2023, the correlation coefficient is 0.87.

Even with a high correlation between density and adequacy, there are a handful of food series in which there was little deterioration in adequacy but density dropped noticeably. Chart 11 shows the indexes for one such series, ham.

Some series became unpublishable on a recurring basis after the start of the pandemic. For these series, the value of density relative to adequacy is unclear. There are three basic CPI item categories that were inadequate at the national level for more than 3 months from March 2020 to July 2023: gardening and lawncare (for 25 months), repair of household items (for 23 months), and leased vehicles (for 25 months). The first two categories were inadequate for 2 months in the 12 months prior to March 2020, while prior to March 2020, leased vehicles was always adequate. For these basic item categories, the adequacy and density are similar, and again it does not seem that density is providing much new information (see chart 12).

Possible impact of density on publication

Density and adequacy often seem to track closely. One could, however, look at how publication rates would change if there were a rule that the density level must be at least 50 percent for publication at the national level.¹⁶ If one were to use a 50-percent threshold for density, how would that compare with the current system? Looking at the published indexes at the U.S. city-average level from January 2020 to July 2023, one finds 948 item-months at the national level that were published but would not have been published at a 50-percent threshold for density. With 6,586 item-months at the national level in the dataset, this would reduce the amount of published series by 14 percent.

Temporary stock-outs

Past research using BLS data from the Commodities and Services (C&S) survey has looked at rates of missing prices to estimate how often stores are out of particular products.¹⁷ Taking a cue from this work, I attempt to see how often a quote that was collected as “temporarily unavailable” later becomes available again. Temporary stock-out in the C&S data would be an indication of how often stores are missing products that they typically have for sale.¹⁸ This missingness could be caused by supply chain, transportation, and labor issues during the pandemic.¹⁹

For each month from January 2020 to July 2023, I identify quotes that were unavailable. This includes quotes in outlets in which the entire outlet was unavailable (for example, the outlet was temporarily closed or had no online presence). Then, for each unavailable quote, I examine the data for the next 6 months to see what eventually happened to the quote.²⁰ Chart 13 shows rates of temporarily unavailable quotes returning, quotes being replaced, and those disappearing for some other reason, for all quotes.

Charts 14 and 15 show the rates of temporarily unavailable quotes (and their eventual outcomes) for goods and services, respectively. The impacts of the COVID-19 pandemic and the switch to online collection on goods availability was generally greater than that on services. 

Some of these categories, like groceries and paper products, were known to have supply chain issues at the start of the pandemic.²¹ The rate at which quotes were unavailable has recovered fully, but only relatively recently, as shown by the dotted lines in charts 16 and 17.

Conclusion

This article explores four topics about the impact of COVID-19 on the CPI operations. First, I construct a price index that examines the impact of changing to web collection from the traditional brick-and-mortar collection. In certain categories, there is a price difference observed associated with the change from in-person to online price collection. Next, I carry out some alternative imputation of missing cells. There is an impact from this change in some categories with particularly acute nonresponse issues. Next, I compute a measure of data completeness that more fully considers what is occurring in terms of collected prices at the lowest level. Here, for certain categories, the new statistics show some aspects of data deterioration that were not immediately clear with existing metrics. Lastly, I measure the rate at which products are missing from store shelves and later returned. These new metrics show that the data collection disruptions caused by the COVID-19 pandemic were substantial, but in many cases the new metrics have returned to their prepandemic levels.