• Overview
• Concepts
• Data sources
  • Survey data
  • Initiation of price survey
  • Monthly price collection
  • Alternative data
  • Administrative trade records
  • Other alternative data sources
  • Validation checks
  • Data confidentiality and integrity
• Design
• Calculation
• Presentation
• History
• More Info

• Methods Home
• About
• Programs
• Glossary
search

Handbook of Methods International Price Program Data Sources

International Price Program: Data Sources

There are two ways that data sources are used in calculating the U.S. Import and Export Price Indexes. The first data source type is for the sample frame. Frame data is the cornerstone of the sample selection process, representing the target population of the survey. For an explanation of sample frame data sources and how they are used, see the design section. The second data source type is the price data used for monthly index calculation. Price data are collected through the U.S. Import and Export Price Indexes survey and through alternative data sources. Alternative data sources may be public or private survey or nonsurvey data that provide price and item information in accordance with the Bureau of Labor Statistics (BLS) concepts and methods, for one or more product categories.

Approximately 60 percent of the value of goods trade depends on survey data as a source for prices. Directly collected prices are required for heterogeneous product categories, such as high-tech products, electronics, and capital goods and other specialized product areas from representatives of participating companies. Approximately 40 percent of the value of goods trade depends on administrative trade records as an alternative data source. Trading companies provide these import and export transaction records to the U.S. Customs and Border Protection Agency (for imports) and the U.S. Commerce Department (for exports) for the government to determine admissibility. With strict security protocols in place, BLS is permitted to use the administrative trade data for statistical purposes only. The administrative trade records are an alternative data source, and the millions of timely trade transactions make them a primary source of prices for product categories for which a unit price is acceptable. These prices are acceptable for homogeneous product categories and consist of metals, grains, and other generic-type products. The administrative trade records are collected and processed to calculate high-quality unit prices. For services trade, survey respondents are the source of air freight prices, and a third-party data provider is the source of air passenger fare prices. In some instances, for product categories not using the data sources described above, price data are collected through other government or third-party data sources.

Survey data

As described in the design section, administrative trade data from other government agencies form the sample frame to select the U.S. Import and Export Price Indexes survey. The trade records are reported by importers and exporters for regulatory reasons, and BLS uses the information for statistical purposes only. Administrative trade records from other government agencies are securely shared for statistical purposes only to reduce burden on respondents and increase operating efficiency, following federal regulations. The trade records used for the sample frame represent the most recent complete calendar year. That is, the sample is based on trade transactions that occurred in the past. The first two stages of sample selection result in a list of establishments and the number and type of items. For more information on these two stages, see the design section. The third and final stage of the sampling process requires updated item selection by a BLS field economist with the cooperation of the company representative at each sampled establishment. A successful interview with a sampled establishment results in a more detailed list of current items that are actually and currently traded and establishes the items that best represent the mix of goods for the sampled establishment identified during the item sampling stage. The subsequent collection of the monthly price for each item, if traded, forms the basis of price changes that are aggregated into the final price indexes.

Initiation of price survey

For the final stage of the sampling process, a BLS field economist meets with a company representative of a sampled establishment and conducts the initial price survey. The selection of the actual items that constitute the market basket occurs at this stage. During the personal interview, the field economist asks a series of questions to identify the actual products presently being traded and confirms that the establishment is actually still trading in the sampled-product categories. The field economist describes the types of items required for the market basket using detailed product category or classification group descriptions, and the respondent provides examples of commonly traded items with the respective prices and share of total company trade. Items that are traded at least once a year are chosen for the categories required to fill out the survey. The choices depend on the information available starting with actual trade dollar value, then estimated trade dollar value, and finally an estimate on trade dollar value ranking among potential choices. If no information is known about the dollar value, then items are chosen randomly. The field economist records item and price information and price-determining characteristics which include size, composition, country of origin or destination, range of capabilities, and transaction terms; and instructs the survey respondent on the monthly data collection process. Once items are selected, those items are then initiated into the market basket that is then repriced on a monthly basis.

Monthly price collection

Upon the initiation of an item into the market basket, the BLS national office staff coordinates continued monthly price collection with the company respondent. (For more information on the types of goods and services that are covered by the U.S. Import and Export Price Indexes survey, see the concepts section.) Survey respondents may transmit the monthly transaction price of the items selected for the survey by a secure internet site or by telephone if more convenient for the respondent. BLS staff may contact the survey respondent to assure price information is submitted by the deadline or to request clarification and validate changes to item characteristics or prices. All reporting is voluntary and confidential, in accordance with the BLS report to the Office of Management and Budget’s confidential information protection and statistical efficiency act (CIPSEA).

Respondents provide prices for items based on actual transactions that occur as close as possible to the first day of the month for which the data are to be published. The recorded price is the actual transaction price for which an item is purchased or sold. The transaction price ideally accounts for discounts and surcharges paid or charged; duties on imports are not counted as part of the price. For more information on what is included in the transaction price, see the concepts section.

Several prices collected come from intercompany trade, production sharing, or cases for which a company can only provide a unit price. For each of those cases, the price still represents the transaction price at the point an item crosses the border. Intercompany trade occurs when the price of items that are purchased or sold are determined through trade within a company or between affiliates of related companies. The price reported may be determined by market-related factors, but that is not always the case. Production sharing is product manufacture shared between two (or more) independent companies or various affiliated units of the same company that are located in different countries. In addition, average prices provided by respondents are accepted.

Price information may be collected on a less-than monthly schedule if the item is not traded monthly, if the price is set by contract and does not change in intervening months, or to alleviate burden on the respondent. If an item is not traded in a given month, a respondent may provide an estimated price based on what the item price would have been if the item were traded. When the item price is set by contract, the transaction price is carried forward during the duration of the contract until a new price is set. For other methods of handling missing prices when a respondent cannot provide an estimate, see the calculation section.

Alternative data

Many commodity categories are not collected directly through the U.S. Import and Export Price Indexes survey. For most homogeneous, generic-like goods, administrative trade records are the source of price data. Administrative trade records from other government agencies are securely shared, following federal regulations, for statistical purposes only to reduce burden on respondents and increase operating efficiency. The trade records used for prices represent the most recent month(s) for which data are available. For a few product areas, specifically diamonds and air passenger fares, prices are sourced from third-party data providers.

All alternative data sources provide detailed price quotes. The level of detail and type of data may require price quotes to be preprocessed to calculate a more aggregate average price, unit price, or unit-value index that corresponds with an item in a product classification group. The items in a product classification group are the point at which the calculation of published price indexes begins. The same index calculation methods are applied to both directly collected data and alternative data.

The alternative data source is selected to represent an item or group of items that are representative of the market basket. As with survey data, the preferred price is the actual transaction price for which an item or shipment is purchased or sold. The transaction price may be one price or an average of more than one transaction. Spot prices or list prices may also be accepted. Some alternative data sources are protected from disclosure in accordance with federal regulation. Data are securely transmitted on a monthly schedule for processing.

Administrative trade records

As mentioned previously, approximately two-fifths of the value of goods trade depends on the alternative data source of administrative trade records that companies report to other government agencies and that BLS is permitted to use for statistical purposes only. The administrative trade data consists of trade shipment records that are submitted by traders to the U.S. Customs Automated Commercial System (ACE), which is an alternative data source and is a primary source for prices for those import and export price indexes for which calculation using a unit price is acceptable.

Prices for homogeneous product categories previously were collected with a survey. Technical advances permitted alternative data to replace directly collected prices, while also increasing the number of prices collected and the number of price indexes published.

The administrative trade records are individual shipping transactions, and significant preprocessing converts these shipping transactions into unique price quotes for an item in a product classification group.

Millions of trade records are reported monthly, reflecting actual trade across U.S. borders. These trade records are grouped by similar characteristics to create unique matched-model product varieties that are then able to be consistently priced over time. The unit price of each trade transaction, or record, is the trade dollar value divided by the quantity. The characteristics used for grouping include but are not limited to 10-digit Harmonized System (HS) product classification, employer identification number (EIN), and measurement basis. The result of these groupings are distinct product varieties. Each product variety corresponds to a 10-digit HS product classification, and anywhere from one to thousands of product varieties compose a 10-digit HS product classification. The grouping of characteristics is maintained indefinitely, so each product variety can be priced consistently over time. Each month’s trade records will either fit into an existing product variety or, if characteristics are different, result in a new product variety. The quantity-weighted average unit price is the weighted arithmetic mean of each product variety. The average unit price accounts for all transactions of a product variety throughout the reference month. The selection of the characteristics to group records into distinct product varieties is based on analysis of the match-adjusted R-squared statistic.

The average unit price of each unique product variety is aggregated into a single point estimate of a unit-value index for each 10-digit HS product classification. Because current period quantity and price information are both available for each product variety, a superlative index formula is applied. The Tornqvist superlative index formula first calculates a geometric average of the price relatives of the current to base period prices of the average unit price for each product variety. Current period prices are calculated for each of the 4 months of the revision period. Base period prices are the arithmetic average of all prices of the previous year. The ratio of current-period price to previous-year price, also called a mid-term relative (MTR), is calculated for each month. The Tornqvist calculation then weights the MTR price relatives of the product varieties by the arithmetic average of the value shares for the two periods to calculate the unit-value index for each 10-digit HS product classification group. The index levels in each month are then linked to calculate month-to-month price changes for each classification group. Estimating a price by imputation or linear interpolation is a normal procedure when there are missing price data. For administrative trade data, the process for estimating a missing price depends on the level of calculation. At the level of transactions and product varieties, there is no price when there is no trade; thus, the price is not missing. However, to start pricing a new product variety, an initial price for the previous month is imputed to start the series. Accounting for an entire year of trade for the base period results in many product varieties contributing to the index, even though not all product varieties are traded each month. This year-long base period approach greatly increases the consistency and number of product variety prices that contribute to the unit-value index estimation.

With the administrative trade records, each unit-value index is equivalent to an item price in the calculation of import and export price indexes. Thus, the item prices that are aggregated to the published indexes are composed of directly collected prices and unit-value indexes. Together they form two nonoverlapping subsets of item prices that cover the target population of merchandise goods trade, except for a few product areas that use third-party alternative data.

For more information on the calculation of the unit-value indexes, see Enhancing Import and Export Price Indexes: A New Methodology Using Administrative Trade Data.

Other alternative data sources

Monthly pricing data for diamonds are obtained both directly from survey respondents and from an independent diamond industry resource. The diamond pricing data obtained from the independent diamond industry resource are for round, brilliant-cut diamonds. Some of those diamond prices are used in both the import and export price indexes because of their global commodity nature, while other prices are used exclusively in the import or export indexes. For more information see the fact sheet for diamonds.

Air passenger fares for all U.S. and foreign carriers and country market import and export data are collected from a third-party data provider rather than directly sampling to select flights for pricing. The data are used for monthly pricing and include tickets sold by travel agencies, travel websites, and the airlines for scheduled departure dates during the reference month. For more information see the fact sheet for air passenger fares.

Validation checks

Once collected, monthly prices undergo review and verification to validate the quality of responses for surveys and pre-processed calculations for alternative data sources. When information that is collected is incomplete or shows changes to important data elements, such as the item description, price-determining characteristics, or survey respondent information, additional validation is needed. For survey data, industry analysts complete this additional validation by contacting the company respondent to ensure that the item priced remains consistent from month to month. For information on how substitutions, replacements, and quality adjustments are handled, see the calculation section. Industry analysts might also contact respondents in the case of a nonresponse or if the price change leads to an index percent change surpassing a statistically determined value. For alternative data, industry analysts complete this additional validation by statistically evaluating price and price change observations for outliers and determining the required action to keep, suppress, or revise the price quote.

Data confidentiality and integrity

BLS is strongly committed to protecting the confidentiality of information provided by respondents and by other government agencies ensuring that the information is used only for statistical purposes. BLS staff are statutorily required to protect confidential information from disclosure. The pledge of confidentiality means that only BLS employees and agents, such as contract employees working on statistical programs under BLS agreements, will ever have access to respondent information. The confidentiality pledge means that BLS published measures will combine respondent information to produce descriptions, estimates, and analyses regarding the state of the economy without identifying or risking exposure of respondent identifiable individual information.