Department of Labor Logo United States Department of Labor
Dot gov

The .gov means it's official.
Federal government websites often end in .gov or .mil. Before sharing sensitive information, make sure you're on a federal government site.


The site is secure.
The https:// ensures that you are connecting to the official website and that any information you provide is encrypted and transmitted securely.

Consumer Expenditure Surveys

2021 CE Online Diary Paradata Analysis

Authors: Graham Jones and Parvati Krishnamurty

This paper was published as part of the Consumer Expenditure Surveys Program Report Series.

1. Introduction

Paradata are a byproduct of the survey fielding process and offer information on how survey data are collected. Analysis of paradata can provide insights for optimizing survey fielding procedures and data quality, especially for web surveys, where large amounts of paradata can be captured. Recently, the Bureau of Labor Statistics (BLS) Consumer Expenditure Surveys (CE) program developed an online version of the CE Diary Survey and made a concerted effort during development to ensure the collection of detailed paradata. In June 2020, the BLS authorized the use of the online expenditure diary in response to the temporary halt on personal visits during the COVID-19 pandemic.[1] The online diary was officially implemented in July 2022, but analysis of the paradata from this emergency use period has provided insights on respondent behavior during diary keeping.

This report will provide background on the implementation of the CE online diaries, along with detailed information on the structure of the CE online diary paradata, and its potential uses for CE research. The analysis presented in subsequent sections of this report focuses on online diary paradata from the 2021 CE Production year and illustrates the abundance of information on respondent activity and engagement that can be gleaned from its analysis.

2. Overview

2.1 Background

The CE Diary Survey collects respondent expenditure data from sampled households by placing them with two one-week expenditure diaries, which until recently were only offered in paper form. This standard for data collection changed in June of 2020 with the introduction of a supplemental online diary mode as a contingency measure due to the COVID-19 pandemic. The online diary was designed as part of the effort to redesign the CE Surveys, and was field tested in the Large-Scale Feasibility (LSF) Test of the Online Diaries in 2019-2020.[2] The LSF test, as the name suggests, was a field test of a protocol that incorporated an online diary compared to the existing production protocol that only included a paper diary.[3] This LSF test analysis ultimately resulted in a detailed report with recommendations for the online diaries to be officially implemented into CE Diary Survey Production, which took place in July 2022.[4] The early implementation of the online diary due to the pandemic provided CE with a full year of additional online diary data for 2021. As previously mentioned, this 2021 data not only gave CE researchers the opportunity to examine expenditure reporting and household characteristics between the paper and online diary modes earlier than expected, but it also provided researchers with a wealth of additional paradata.

2.2 Paradata Structure

In the case of the CE online diary, paradata contain information about respondent activity within the online expenditure diary itself. The structure of the CE online diary paradata is such that the data, in table form, exist at the user-event level. This means that each action (or event) made by the respondent is recorded in a row containing information on the user-event time, event type (e.g., login, data entry, hyperlink click, exit, etc.), language used, as well as information for determining the operating system and browser used.  The event type values allow for the measurement of various respondent activities including: logins, devices used, operating systems used, languages used, and time spent in the diary. Furthermore, these paradata measures can be combined to better understand more nuanced respondent behaviors such as logins per diary day, time spent in the diary by device type, and average respondent logins by device type.

2.3 Purpose of Analysis

The purpose of this report is to examine CE online diary paradata from 2021, which was the first full year that the online mode was available in CE Production. This paper will leverage paradata generated through the use of the online diaries to analyze a variety of respondent behaviors and activities including user login activity, devices used, operating systems used, languages used, and time spent in the diary. These measures will be compared to paradata results from the LSF test of the online diaries in CE, to identify any changes in respondent behaviors that occurred due to updates in CE fielding protocols between the LSF and the implementation of the online diaries into CE production.[5] Estimates of respondent behavior using the CE paradata will also be broken down by months within 2021. The purpose of the latter is to identify any seasonal fluctuation in the online diary activity and see if measures improve over the first full calendar year of its availability. Along with examining respondent behavior, this analysis seeks to establish new quality metrics, using the CE paradata, that will be regularly monitored as changes are introduced into the CE Diary Survey instrument and fielding protocols. 

3. Results

3.1 Online Diary Use

In 2021 CE Production, CE respondents completed 12,067 weekly expenditure diaries, approximately 27 percent (3,263 total) of these weekly diaries were placed as online diaries (Armstrong et al., 2022).[6]  Typically, CE researchers analyze weekly expenditure diary data, where each weekly diary completed within the same Consumer Unit (CU) are treated independently, but in the online paradata, actions are recorded at the CU level.[7] Thus, the analysis of online diary paradata in this report was conducted at the CU level.

An examination of the 2021 paradata revealed that information was recorded for 1,877 unique CUs over the course of 2021. It should be noted that diary placement in CE follows a modified mode-choice protocol, which affords Census Field Representatives (FRs) working cases more flexibility to work with respondents. This protocol allows respondents and FRs input on the survey mode being used, given that they meet online diary eligibility requirements.[8]  Table 3.1.1 illustrates that of the 1,877 unique CUs identified in the online diary paradata, 76 never successfully logged into the online diary. The 1,801 CUs considered to have at least one successful login included both complete and incomplete diary cases.[9] Table 3.1.2 below shows the same data on online diary user login success, but here the results are broken up by the final coded outcome of the diary case. The data show that 1,529 CUs were associated with complete diary cases and had at least one successful login, while 43 did not have a successful login.[10]

Table 3.1.1 Login Success for All Online Diary Users
Login Success Number of CUs







Table 3.1.2 Login Success for Online Diary Users by Outcome
Login Success Complete Cases Type-A Cases


1,529 272


43 33


1,572 305
3.2 Login Activity

After sub-setting the data to consider only CUs associated with complete cases, and with at least one successful login, there were 1,529 unique CUs left to analyze. These 1,529 CUs accounted for 20,870 successful logins, with an average of 13.7 logins per user (and a median of 10), as displayed in Table 3.2.1 below. The other 272 CUs, associated with incomplete diary cases and at least one login, had an average of 4.5 logins per user (and a median of 2).

Table 3.2.1 Total CE Online Diary Successful Login Attempts by Case Outcome in 2021
Login Attempt Result Number of CUs Total Login Attempts Mean Median

Complete Cases with logins ≥ 1

1,529 20,870 13.7 10

Type – A Cases with logins ≥ 1

272 1,230 4.5 2

The higher average login activity for CUs associated with complete cases meets expectations, as incomplete diary cases often result from prematurely dropping out of the survey or providing insufficient detail in the spending diary. However, the difference in average login failures were not as pronounced when comparing rates by diary case outcomes. Table 3.2.2 below shows that CUs associated with complete cases, and with at least one successful login, had an average of 1.8 login failures, which is only just higher than the rate of 1.4 for CUs associated with incomplete diary cases.

Table 3.2.2 Total CE Online Diary Successful Login Attempts by Case Outcome in 2021
Failed Login Attempt Result Number of CUs Total Failed Login Attempts Mean Median

Complete Cases with logins ≥ 1

1,529 2,768 1.8 0

Type – A Cases with logins ≥ 1

272 392 1.4 0

Login activity paradata was also analyzed by day of diary keeping to assess CE respondent behavior over the diary keeping period, which is depicted in Figure 3.2.1 below.[11] This figure shows the percentage of login activity by respondents associated with complete diary cases in 2021 for each diary day.[12] The percentages in each line should be interpreted as the proportion of logins or login failures occurring on that diary day out of all logins and login failures recorded. Ideally, respondents would contemporaneously report expenditures in the diary, resulting in an even distribution of logins per diary day, but the data show that this was not the case.

Figure 3.2.1 above shows that the highest levels of login activity are concentrated around the beginning and end of the diary keeping period, with a small bump at Diary Day 8, presumably following a mid-week reminder. The data show that Day 0, the placement day, has the highest proportion of logins with roughly 9.0 percent. This is meets expectations as this is when the FR working the case “places” the expenditure diary and typically helps the respondent log in for the first time. Logins on Day 1 and 2 remained relatively high compared to the rest of the diary keeping period, with 7.8 and 7.3 percent of total logins, respectively, but dropped considerably over the rest of the first week. Following the small mid-week bump on Day 8, the percentage of total logins declined in the second week to a low of 4.5 percent on Day 10. The series experienced gradual increases in the percentage of total logins between Day 11 and Day 13 but remained relatively low until Day 14 when logins jumped up to 7.3 percent. This late bump in logins may have been due to last minute data entry by respondents or data entry that took place during the pickup interview. Figure 3.2.1 also depicts login failures, which experienced a sizeable drop off from Day 1 to Day 2 of the diary keeping period. Following Day 2, login failures follow a similar trend to logins through the first week of the diary keeping period but does not increase as sharply in the second week. This may be capturing the learning curve of online diary users who experience fewer login failures towards the end of the two-week process. The next figure examines login activity rates by month captured in the paradata. Figure 3.2.2 below shows average logins per user by month for 2021, with the blue and orange bars representing fluctuations in the mean and median of user logins, respectively.

The mean average of successful logins per user fluctuated between a low of 12.2 and a high of 15.0 over the course of 2021. Overall, average monthly logins per user did not stray far from the mean for the year (13.7), with the greatest difference being only about 1.5 logins. From month to month, the greatest difference in Figure 3.2.2 can be seen between November and December, where the average rose by 2.4 logins. Median logins per user fluctuated between 10 and 12 logins for most of the year, before falling to a low of 8 logins in November and then rising again to 12 in December. With respect to the average number of login failures per user, the differences between months were relatively low. As little could be deduced from these findings, the graph containing this data was placed in the appendix of this report.

The last set of figures in this section compares CE online diary login activity from 2021 against paradata estimates from the LSF Test of Online Diaries, which was fielded from October 2019 through March 2020. The data in Table 3.2.3 below shows the comparison of average logins per user captured in the two sets of paradata. It should be noted that this comparison includes CUs regardless of completion status, as the LSF analysis was not limited to complete cases.

Table 3.2.3 Online Diary Login Activity: 2021 CE Production vs LSF Test
Login Result Number of CUs Total Logins Mean Median

2021 CE Production

1,801 22,100 12.27 10

LSF Test (10/2019 – 03/2020)

372 2,825 7.59 5

The data show that the mean average of logins was higher by roughly 4.7 logins per user in the 2021 CE production paradata than in the LSF test paradata. This is an encouraging finding as it could indicate that CE online diary users are interacting with the diary more frequently than those sampled in the LSF test, possibly due to improved field staff training and diary mode placement. As the BLS utilizes a mode-choice protocol for CE production, this may have resulted in the online diary mode being placed with more suitable CUs. The final chart, Figure 3.2.3 below, examines the percentage of total login activity by diary day, and the data show that Day 0 had the highest percentage of logins for both samples.

Roughly 10.4 percent of 2021 CE diary user login activity occurred on Day 0 of the diary keeping period, compared to 15.7 percent in the LSF. This relatively high Day 0 login activity in the LSF paradata was likely influenced by differences in the LSF testing protocols and the fielding procedures in 2021 CE production. The LSF testing protocol specifically instructed FRs working LSF online diary cases “to help the respondent log in to the online diary if the respondent was willing.” In normal fielding for CE production, this practice is encouraged to help respondents get adjusted to the online diary, but it is not a strict protocol.

Related to logins, updates to login credentials by users (i.e., usernames and passwords) were also captured in the paradata, but our analysis revealed that there were only 28 instances of password updates and 4 instances of username updates in 2021. This low activity is likely due to the ability to save usernames and passwords in most modern browsers.[13]

3.3 Devices Used

In addition to user actions (e.g., login attempts, hyperlink clicks, and text box entries), the CE Diary paradata also capture user environment data on how the online diary web page was accessed upon login. User environment data includes information on the screen size and operating system used to view the webpage, which can in turn be used to determine the type of device used by the respondent for a particular login. For the purposes of this report, device-use by a given CU was broken down into three categories: Desktop Only; Mobile Only; and Mixed Device Use. A CU was determined to be a ‘Mixed Device Use’ case if they had login events that were associated with more than one device type in the paradata. Table 3.3.1 below shows the overall breakdown of device use by all online diary respondents captured in the 2021 paradata.

Table 3.3.1 Device Types used by CE Online Diary Respondents in 2021
Device Type Used Number of CUs Percentage of CUs

Desktop Only

1,162 64.5%

Mobile Only

387 21.5%

Mixed Device Use

252 14.0%



The data show that ‘Desktop Only’ use for accessing the diary was by far the most prevalent option for CUs in 2021, with 64.5 percent, followed by ‘Mobile Only’ and ‘Mixed Device Use’, with 21.5 percent and 14.0 percent, respectively. This device use data was further analyzed by the case outcome associated with the online diary user, the results of which are shown in Table 3.3.2. These results confirm that most online diary CUs in 2021 logged into the instrument strictly via desktop, but also reveal a disparity in the device use choices between those with complete diary cases and incomplete diary cases. The data in Table 3.3.2 show that the percentage of CUs associated with incomplete diary cases that used only a mobile device was over twice that of CUs associated with complete diary cases (40.4 percent compared to 18.1 percent), and conversely the rate of those using only a desktop was higher for CUs with complete diaries (66.9 percent compared to 51.1 percent). The use of multiple devices was also more prevalent for CUs with complete diaries (15 percent compared to 8.5 percent).

Table 3.3.2 Device Types used by CE Online Diary Respondents by Case Outcome in 2021
Device Type Used CUs with Complete Cases and logins ≥ 1 (N=1,529) CUs with Type – A Cases and logins ≥ 1 (N=272) All CUs (N=1,801)

Desktop Only

66.9% 51.1% 64.5%

Mobile Only

18.1% 40.4% 21.5%

Mixed Device Use

15.0% 8.5% 14.0%

This user environment data can be examined further to reveal the combination of operating systems users employed to access the diary. Figure 3.3.1 below shows the breakdown of 2021 operating system use, by the diary outcome associated with the CU. The data show that MS Windows was most commonly used by CUs logging into the diary for the first time. While this was the case for both groups, the level of Windows use for CUs associated with complete diary cases was noticeably higher than those associated with incomplete diaries (49.3 percent versus 39.7 percent, respectively). The data show a similar dynamic for the use of Mac iOS, with a higher proportion of CUs associated with complete diary cases using this operating system compared to those with incomplete cases (18.4 percent versus 11.8 percent, respectively). Conversely, CUs associated with incomplete cases had a higher proportion of iPhone and Android use than those associated with complete diary cases (27.9 versus 18.6 percent & 19.5 versus 10.3 percent, respectively), which was expected given the respondent device type results in Table 3.3.2 above. X11 and iPad also registered as operating systems used, but at very low rates.

While the breakdown of operating system use is interesting to note, we found in our exploration of the paradata that device type was a more meaningful grouping for user activity analysis. As such, estimates of the average number of logins per CU in Table 3.3.3 below, were grouped by device type used. The data in Table 3.3.3 show that, for both the mean and median estimates, CUs with mixed device use logged into the diary more frequently than desktop only or mobile only users. Mixed device users had an average login rate roughly 2.0 percentage points higher than the desktop group (15.3 compared to 13.3) and about 1.8 percentage points higher than the mobile group (15.3 compared to 13.5). When examining the median logins, the data show that desktop only users had, on average, two more logins than mobile only users (10 versus 8, respectively), despite the mean average being nearly equal. This would lead us to believe that some highly active mobile only users skewed the mean upward.

 Table 3.3.3 Average CE Online Diary Logins per CU in 2021 by Device Type
Device Type Used Number of CUs Mean Logins Median Logins

Desktop Only

1,023 13.3 10

Mobile Only

277 13.5 8

Mixed Device Use

229 15.3 12


1,529 13.7 10

Figure, 3.3.2 shows the monthly distribution of devices used by responding CUs in 2021. The data illustrate a similar picture to the results in Table 3.3.1, with the majority of CUs using only a desktop to access the CE Online Diary for every month in 2021. After initial variability in device use, the share of desktop usage leveled out after the first half of 2021. This trend can be seen in the last quarter where the proportion of “desktop only” users remained right around the total 2021 desktop share of 64.5 percent (shown in Table 3.3.1), after falling from 66.9 percent in September. The stabilization in the proportion of desktop only users in the final months of 2021 coincided with fluctuations in the rate of mixed device users and mobile only users among CE online diary respondents. Mixed device use in particular saw its largest three-month swing in the final quarter of 2021, when the share of users increased from 13.0 percent in October to 20.7 percent in December. A similar story was true for the share mobile only CUs from October to December, but in the opposite direction. In the final quarter of 2021, the share of mobile only users fell from 22.5 percent in October to 14.4 percent in December 2021.

Similar to the previous section, the last set of figures on device use compare CE online diary paradata results from 2021 against estimates from the LSF Test of Online Diaries. The first comparison, Table 3.3.4 below, examines the overall breakdown of device use by responding CUs in 2021 CE production and those in the LSF Test. The data show considerable differences in the proportions of device use, most notable is the decrease in mobile device use from roughly 36.0 percent during the LSF Test to 21.5 percent in 2021 CE Production. This decrease in overall mobile device use from the LSF Test to 2021 is also captured in the decrease of ‘Mixed Device Use’ CUs (20.4 percent to 14 percent) and coincides with a sizeable increase in the percentage of CUs using only a desktop to log in (43.6 percent to 64.5 percent).

 Table 3.3.4 Device Types used by Online Diary Respondents: 2021 CE Production vs LSF Test
Device Type Used Percentage of CUs in CE Production (N=1,801) Percentage of CUs in LSF Test (N=372)

Desktop Only

64.5% 43.6%

Mobile Only

21.5% 36.0%

Mixed Device Use

14.0% 20.4%


100% 100%

The next chart, Figure 3.3.3 below, compares the average number of logins per online diary user by device type, between 2021 CE production and the LSF Test. In this figure, the data show that the average number of logins per LSF diary user was lower than the login rates for online diary users in 2021, across all device types. Despite login averages being lower for each device type, the LSF logins followed a similar pattern in levels to 2021 CE production, with ‘Mixed Device’ users having the highest rate (9.05 logins) followed by ‘Desktop Only’ and ‘Mobile Only’ users (7.33 and 7.09, respectively).

The findings from this section show that login activity has increased from the time of the LSF test, and that ‘Mixed Device’ users are still associated with the highest login rates. With respect to the ‘Mobile Only’ user experience, results also show that there is still room for improvement as engagement is lagging relative to ‘Desktop Only’ users. The next section will cover the topic of how much time users spent in the online diary, examining the average time total online diary users spent interacting with the expenditure diary.

3.4 Time Spent in Diary

Unlike the findings in the previous sections, using the paradata to produce a meaningful estimation for total time spent in the online diary required a more nuanced approach. While the logic for calculating total time spent in diary was straightforward, summing the time differences between actions for a given user measured in seconds and dividing that total by 60 to get a total in minutes, this simple calculation was complicated by noise in the paradata. The goal of the metric is to capture time spent entering expenses into the diary as opposed to total time spent logged in. Thus, certain user actions within the diary (i.e., login, logout, failed login, entry, and exit) were excluded from the calculation to reduce this noise interference. Any user events that did not involve diary keeping activities like the login and post-login forms were also excluded. The exclusion of the post-login form from the total time in diary calculation also removes the possibility of overestimating time resulting from a user being timed out from inactivity.

Beyond affecting the total time estimates, the exclusion of user actions and diary forms impacted the number of CUs in the dataset eligible for total time calculation. After the previously described restrictions were applied, 1,440 CUs remained from the original set of 1,529. Table 3.4.1 below shows this new number of observations as well as the mean and median total time spent in diary for eligible CUs. In 2021, CUs using the CE Online Diary spent a mean average of 36.1 minutes in the diary, and a median of 26.6 minutes.

Table 3.4.1 Average Total Time Spent in Diary by CUs in 2021
Time Number of CUs Mean Median

Total Time Spent in Diary

1,440 36.1 26.6

After estimating the average total time for all online diary users in 2021, CUs were split up by the type of device used. The data in Table 3.4.2 show that ‘Desktop Only’ users spent the most time in the diary, with a mean of 38.9 minutes and a median of 29.7 minutes. ‘Mixed Device’ users trailed ‘Desktop Only’ users in both the mean and median estimates of total time, but only slightly with 37.2 and 29.0 minutes respectively, while ‘Mobile Only’ users lagged both by a larger amount. The mean of total time spent in the diary for ‘Mobile Only’ users was 22.8 minutes and the median just 13.2 minutes, roughly 16 minutes less than both the mean and median estimates for ‘Desktop Only’ users.

Table 3.4.2 Average Total Time Spent in Diary by Device Type in 2021
Device Type Used Number of CUs Mean Median

Desktop Only

982 38.9 29.7

Mobile Only

287 22.8 13.2

Mixed Device Use

240 37.2 29.0

Next, monthly average total time spent in diary by 2021 users was analyzed. Figure 3.4.1 below illustrates this variation over the course of 2021. The data show that average total time spent in the diary per user fluctuated between a low point in February (mean of 29.9 minutes and median of 20.6) and a high in September (mean of 41.4 minutes and median of 28.7). When observing the mean of total time spent in diary, the graph appears to trend upward in the latter half of the year, but this trend is less apparent when examining the median.

Estimates of median total time spent in the diary per user did not vary as much as the mean over the course of 2021, but did see a roughly 10 minute difference between the low point of 20.6 minutes in February and the high point of 30.9 minutes in July.  

Finally, median time spent in diary for 2021 CE Production was compared to estimates from the LSF Test of Online Diaries. Figure 3.4.2 below illustrates this comparison for all users from each sample, and by device type used. The data show that median total time spent in diary was higher for LSF online diary users than for 2021 CE production diary users, with LSF users spending about 3.4 minutes longer in the diary (27.6 minutes compared to 24.2). Median total time for LSF users was higher across all device types. The largest differences were in the “Mixed Device” use category (11.8 minutes) and the “Mobile Only” use category (6.3 minutes). In the case of “Desktop Only” users, which accounted for the largest share of diary users, the difference in median total time between the LSF and 2021 CE Production was less pronounced at just 2.6 minutes.

The next comparison between 2021 production and the LSF, shown below in Figure 3.4.3, examined time spent in the diary, but for each individual diary day. This measure summed the total time spent entering expenses in the diary by users on each diary day (Day 0 - Day 16). In Figure 3.4.3, the median of this total time per diary day measure is depicted at the CU level. The data show that diary time for users in both 2021 production and the LSF was higher in the second half of the two-week diary keeping period. It should be noted that time was only counted on days where a user interacted with the diary, so for example, a user with no time spent on Day 5 would not be counted as a 0 when calculating the Day 5 median time. The median time spent in the diary per day for 2021 production did not fluctuate as much as in the LSF, but generally followed the same upward trend. In both samples, the final days of the diary keeping period had the highest median times, which would lead us to believe that most respondents save the bulk of their reporting for the end of the period. One notable exception to this was the jump in median LSF user time from 4.9 minutes on Day 3 to 9.2 minutes on Day 4. This jump in the LSF line is most likely explained by the midweek reminder that took place on Day 3, as part of the LSF protocol. This protocol remained in the CE production fielding protocols, but the equivalent increase does not appear in the CE production data. This may have been due to the fact the FRs were not required to make a note of the reminder contacts after the conclusion of the LSF, leading to less follow-up.

The ‘Time Spent in Diary’ section results showed that within 2021 CE Production, users with higher overall login rates (e.g., “Desktop Only’ and ‘Mixed Device’ users shown in Table 3.4.2) translated to more time spent in the diary. This connection between login rates and time spent in the diary did not hold as true when measured by diary day. The differences between Figure 3.2.3 and Figure 3.4.3 illustrates that relatively high login activity in the first few days of diary keeping does not necessarily equate to higher median times spent in the diary. The inconsistency is likely due to online diary placement protocols, in which Census Field Representatives are encouraged to help respondents login for the first time at placement (Day 0), but typically no expenditure reporting occurs at this time. The two series do show some connectivity toward the end of the diary keeping period, where the median time spent increases nearly three minutes from Day 11 (6.7 minutes) to Day 13 (9.6 minutes). In the next section, more standard measures of data quality used in the CE Diary Survey will be looked at alongside some of the paradata measures from this report.

3.5 Paradata and CE Data Quality

To better understand the relationship between the paradata measures presented in this paper and data quality in the CE Diary Survey, the online diary paradata were analyzed alongside more traditional indicators of data quality used in the CE program. Specifically, expenditure totals and expenditure counts were analyzed by login activity, user device type, and time spent in the diary. In order to make this comparison, the online diary paradata first needed to be combined with the final 2021 CE production diary data. In this section, the sample of CUs examined is subset to only those associated with complete diaries, with at least one successful login, and with expenditure information. A total of 1,490 CUs from the 2021 online diary paradata met the previously described criteria and were successfully combined with the corresponding expenditure data.

The next set of figures examine the relationship between findings from the paradata and traditional indicators of CE Diary data quality. First, median total expenditures and expenditure counts were analyzed by the total number of logins for a given CU, using 10.5 for the group cutoff point as the median for total logins was 10 (see Table 3.2.1). The data in Table 3.5.1 below show that for CUs with 10.5 or greater logins, both median total expenditures and expenditure counts were higher. With respect to total expenditures, the median for CUs in the relatively high login group was roughly $279 higher than in the low login group. In the case of median expenditure counts, the high login group exceeded the low login group by 8.5 entries.

Table 3.5.1 CE Quality Indicators by Login Activity
Login Activity Number of CUs Median Total Expenditures (in Dollars) Median Expenditure Counts

Logins less than 10.5

756 $400.1 28.5

Logins greater than or equal to 10.5

734 $697.1 37.0



After looking at data quality measures by relative login activity, expenditure totals and counts were analyzed by user device type. The results, shown in Table 3.5.2 below, illustrate that ‘Desktop Only’ and ‘Mixed Device’ users were only separated by about $4 in median expenditure totals and 1 entry count, while median estimates for ‘Mobile Only’ users lagged both groups by over $155 and roughly 6 entries. Following the findings in the previous table on data quality and login activity, it is not surprising to find that median expenditure totals and counts were higher for ‘Desktop Only’ and ‘Mixed Device’ users, as these groups were associated with higher login activity when analyzed in section 3.3 of this paper.

Table 3.5.2 CE Quality Indicators by Device Type Used
Device Type Used Number of CUs Median Total Expenditure (in Dollars) Median Expenditure Count

Desktop Only

1,002 $555.3 34.0

Mixed Device

222 $559.0 33.0

Mobile Only

266 $400.5 27.5



Finally, the data quality measures were analyzed by time spent in the diary, which used a method similar to the login activity comparison. All CUs with a total time in diary greater than or equal to 27 minutes were sorted into the relatively high group, and those with less than 27 minutes spent in the diary were classified into the relatively low group. It should also be noted that the number of observations is slightly lower for this comparison (n=1,414) as not all CUs had sufficient activity in the paradata for a total time to be calculated.

Table 3.5.3 CE Quality Indicators by Time Spent in Diary
Total Time Spent in Diary Number of CUs Median Total Expenditure (in Dollars) Median Expenditure Count

Time less than 27 minutes

708 $365.1 24.0

Time greater than or equal to 27 minutes

706 $838.8 44.0



The data in Table 3.5.3 show that the estimates for median expenditure totals and median expenditure counts were much higher among those CUs with relatively greater time spent in the diary. In the 2021 data, median expenditure totals were roughly $474 higher for those in the group with the most time spent in diary, and the median expenditure counts were 20 entries higher. This tracts with the findings related to login activity and data quality, as high logins are associated with greater total time spent in the diary. Of the three paradata measures examined in this section, time spent in diary appears to be most strongly associated with higher expenditure totals and expenditure counts. It should be noted that the paradata does not allow data users to determine the specific cause of a given diary user’s time spent in the diary. In other words, paradata cannot be used to distinguish between respondents whose time spent in the diary is relatively higher due to having more expenditures to report, and those whose time spent in the diary is relatively higher due to them being a more conscientious respondent.

4. Conclusions & Next Steps

4.1 Conclusions

The evaluation of CE Online Diary paradata confirms that the addition of paradata into CE production greatly increases the programs’ ability to analyze CE Diary respondent behavior. The granular detail of user activities captured in the paradata allows researchers to look at diary keeping behavior from new angles (i.e., time spent in the diary and logins) that they otherwise would not have access to using only the paper form. Time spent in the diary per user and total logins per user were associated with higher expenditure totals and expenditure counts, showing that the paradata not only offers new information, but potential insights to diary data quality.

The device use information in the paradata also provide a unique view of how respondents interact with the CE Online Diary. The findings in this report suggest that respondents who incorporated non-mobile devices (i.e., desktop computers) into their diary keeping were associated with higher quality expenditure reporting than those only using a mobile device.

4.2 Next Steps

With these conclusions in mind, the authors of this report would recommend that average total logins per user and average total time spent in the diary per user be incorporated into the Data Quality Profile as standard metrics for the CE Diary Survey. The findings of this paper also support the inclusion of a metric for tracking the patterns of device use by CE Online Diary respondents, as desktop use was also associated with higher quality data. In addition to tracking device use, the authors recommend a deeper analysis of mobile-only users, which would seek to determine if variance in reporting behavior or actual differences in spending are responsible for the differing results by device group. If the measurable differences in data quality for mobile-only device users are in fact driven by sub optimal reporting behaviors, it may be beneficial to encourage respondents to use non-mobile devices for diary keeping, or to screen out users that only access the internet with a mobile device. This is of course not possible for all respondents, but the addition of a question prior to placement asking, “if they had a device other than their phone to fill out the diary with?” could improve our determination of who is ultimately offered the online diary mode.

4.3 Recommendations for Paradata Improvement

The authors also recommend that the calculation of ‘time spent in diary’ continues to be worked on for the most precise estimate. Currently, the paradata at the event level for hyperlink actions lacks sufficient detail for researchers to differentiate between all actions, meaning that some user actions are being excluded from the total time estimation that perhaps should not be. For example, a meaningful number of these hyperlink actions recorded in the paradata had a value label of “undefined button,” making it unclear whether these actions should be factored into the time total calculation.


[1] The Census Bureau collects data for many Federal agency's surveys, including the Bureau of Labor Statistics (BLS) Consumer Expenditure Surveys (CE). During the onset of the COVID-19 pandemic, the Census Bureau put a temporary hold on all in-person contacts for household interviews.

[2] The Gemini Project redesign plan was approved in 2013 to address issues of measurement error and respondent burden in the Consumer Expenditure (CE) surveys.

[3] The LSF Test of online diaries was administered between October 2019 and March 2020.

[4] The LSF Test resulted in a preliminary and a final report.

[5] It should be noted that the LSF test provided pre-paid incentives for half of its sample, which is not a part of normal CE fielding procedures.

[6] Diary placement refers to the process of contacting a sampled household, screening the them to determine their internet access and ability, providing them with either a paper or online expenditure diary, and walking them through diary keeping instructions. The diary mode given is based on the internet screening results, and in part, respondent preference. A respondent with suitable internet access and ability can choose to complete a paper diary, but a respondent with insufficient internet access and ability will not be given the online diary option.

[7] A Consumer Unit (CU) is the measurement unit collected for the eligible individuals represented in survey expenditure reports. The CU is defined as all members of a particular housing unit who are related by blood, marriage, adoption, or some other legal arrangement. This definition can also extend to a person living alone or sharing a household with others, or living as a roomer in a private home, lodging house, or in permanent living quarters in a hotel or motel, but who makes independent financial decisions; or two or more unrelated persons living together who pool their income to make joint expenditure decisions. More information can be found in the CE handbook of methods.

[8] Online diary eligibility is based on a responding CUs internet access and internet ability. Data on internet access and ability are collected by the FR prior to mode selection.

[9] Definitions of CE Survey outcomes (complete and incomplete) can be found in the CE Data Quality Profile Reference Guide.

[10] Unsuccessful logins are the result of either an incorrect username or password being entered on the online diary login page. It is possible that an unsuccessful login could be the result of a system error, but this is unlikely, and not something that can be corroborated with paradata analysis.

[11] Diary Days refer to the days following an expenditure diaries placement with a responding CU. The day of a diary’s placement is considered Day 0, and each of the following days are counted in sequence until the diary keeping period ends, and the diary is collected.

[12] Diary Days we restricted to Day 0 - Day 16 in the CUs expenditure diary keeping period. Some login events in the paradata were recorded outside the diary keeping period and were omitted from the analysis in Figure 3.2.1

[13] As findings for this category were limited, the corresponding table was placed in the appendix.