3.3 Devices Used

In addition to user actions (e.g., login attempts, hyperlink clicks, and text box entries), the CE Diary paradata also capture user environment data on how the online diary web page was accessed upon login. User environment data includes information on the screen size and operating system used to view the webpage, which can in turn be used to determine the type of device used by the respondent for a particular login. For the purposes of this report, device-use by a given CU was broken down into three categories: Desktop Only; Mobile Only; and Mixed Device Use. A CU was determined to be a ‘Mixed Device Use’ case if they had login events that were associated with more than one device type in the paradata. Table 3.3.1 below shows the overall breakdown of device use by all online diary respondents captured in the 2021 paradata.

The data show that ‘Desktop Only’ use for accessing the diary was by far the most prevalent option for CUs in 2021, with 64.5 percent, followed by ‘Mobile Only’ and ‘Mixed Device Use’, with 21.5 percent and 14.0 percent, respectively. This device use data was further analyzed by the case outcome associated with the online diary user, the results of which are shown in Table 3.3.2. These results confirm that most online diary CUs in 2021 logged into the instrument strictly via desktop, but also reveal a disparity in the device use choices between those with complete diary cases and incomplete diary cases. The data in Table 3.3.2 show that the percentage of CUs associated with incomplete diary cases that used only a mobile device was over twice that of CUs associated with complete diary cases (40.4 percent compared to 18.1 percent), and conversely the rate of those using only a desktop was higher for CUs with complete diaries (66.9 percent compared to 51.1 percent). The use of multiple devices was also more prevalent for CUs with complete diaries (15 percent compared to 8.5 percent).

This user environment data can be examined further to reveal the combination of operating systems users employed to access the diary. Figure 3.3.1 below shows the breakdown of 2021 operating system use, by the diary outcome associated with the CU. The data show that MS Windows was most commonly used by CUs logging into the diary for the first time. While this was the case for both groups, the level of Windows use for CUs associated with complete diary cases was noticeably higher than those associated with incomplete diaries (49.3 percent versus 39.7 percent, respectively). The data show a similar dynamic for the use of Mac iOS, with a higher proportion of CUs associated with complete diary cases using this operating system compared to those with incomplete cases (18.4 percent versus 11.8 percent, respectively). Conversely, CUs associated with incomplete cases had a higher proportion of iPhone and Android use than those associated with complete diary cases (27.9 versus 18.6 percent & 19.5 versus 10.3 percent, respectively), which was expected given the respondent device type results in Table 3.3.2 above. X11 and iPad also registered as operating systems used, but at very low rates.

While the breakdown of operating system use is interesting to note, we found in our exploration of the paradata that device type was a more meaningful grouping for user activity analysis. As such, estimates of the average number of logins per CU in Table 3.3.3 below, were grouped by device type used. The data in Table 3.3.3 show that, for both the mean and median estimates, CUs with mixed device use logged into the diary more frequently than desktop only or mobile only users. Mixed device users had an average login rate roughly 2.0 percentage points higher than the desktop group (15.3 compared to 13.3) and about 1.8 percentage points higher than the mobile group (15.3 compared to 13.5). When examining the median logins, the data show that desktop only users had, on average, two more logins than mobile only users (10 versus 8, respectively), despite the mean average being nearly equal. This would lead us to believe that some highly active mobile only users skewed the mean upward.

Figure, 3.3.2 shows the monthly distribution of devices used by responding CUs in 2021. The data illustrate a similar picture to the results in Table 3.3.1, with the majority of CUs using only a desktop to access the CE Online Diary for every month in 2021. After initial variability in device use, the share of desktop usage leveled out after the first half of 2021. This trend can be seen in the last quarter where the proportion of “desktop only” users remained right around the total 2021 desktop share of 64.5 percent (shown in Table 3.3.1), after falling from 66.9 percent in September. The stabilization in the proportion of desktop only users in the final months of 2021 coincided with fluctuations in the rate of mixed device users and mobile only users among CE online diary respondents. Mixed device use in particular saw its largest three-month swing in the final quarter of 2021, when the share of users increased from 13.0 percent in October to 20.7 percent in December. A similar story was true for the share mobile only CUs from October to December, but in the opposite direction. In the final quarter of 2021, the share of mobile only users fell from 22.5 percent in October to 14.4 percent in December 2021.

Similar to the previous section, the last set of figures on device use compare CE online diary paradata results from 2021 against estimates from the LSF Test of Online Diaries. The first comparison, Table 3.3.4 below, examines the overall breakdown of device use by responding CUs in 2021 CE production and those in the LSF Test. The data show considerable differences in the proportions of device use, most notable is the decrease in mobile device use from roughly 36.0 percent during the LSF Test to 21.5 percent in 2021 CE Production. This decrease in overall mobile device use from the LSF Test to 2021 is also captured in the decrease of ‘Mixed Device Use’ CUs (20.4 percent to 14 percent) and coincides with a sizeable increase in the percentage of CUs using only a desktop to log in (43.6 percent to 64.5 percent).

The next chart, Figure 3.3.3 below, compares the average number of logins per online diary user by device type, between 2021 CE production and the LSF Test. In this figure, the data show that the average number of logins per LSF diary user was lower than the login rates for online diary users in 2021, across all device types. Despite login averages being lower for each device type, the LSF logins followed a similar pattern in levels to 2021 CE production, with ‘Mixed Device’ users having the highest rate (9.05 logins) followed by ‘Desktop Only’ and ‘Mobile Only’ users (7.33 and 7.09, respectively).

The findings from this section show that login activity has increased from the time of the LSF test, and that ‘Mixed Device’ users are still associated with the highest login rates. With respect to the ‘Mobile Only’ user experience, results also show that there is still room for improvement as engagement is lagging relative to ‘Desktop Only’ users. The next section will cover the topic of how much time users spent in the online diary, examining the average time total online diary users spent interacting with the expenditure diary.

3.4 Time Spent in Diary

Unlike the findings in the previous sections, using the paradata to produce a meaningful estimation for total time spent in the online diary required a more nuanced approach. While the logic for calculating total time spent in diary was straightforward, summing the time differences between actions for a given user measured in seconds and dividing that total by 60 to get a total in minutes, this simple calculation was complicated by noise in the paradata. The goal of the metric is to capture time spent entering expenses into the diary as opposed to total time spent logged in. Thus, certain user actions within the diary (i.e., login, logout, failed login, entry, and exit) were excluded from the calculation to reduce this noise interference. Any user events that did not involve diary keeping activities like the login and post-login forms were also excluded. The exclusion of the post-login form from the total time in diary calculation also removes the possibility of overestimating time resulting from a user being timed out from inactivity.

Beyond affecting the total time estimates, the exclusion of user actions and diary forms impacted the number of CUs in the dataset eligible for total time calculation. After the previously described restrictions were applied, 1,440 CUs remained from the original set of 1,529. Table 3.4.1 below shows this new number of observations as well as the mean and median total time spent in diary for eligible CUs. In 2021, CUs using the CE Online Diary spent a mean average of 36.1 minutes in the diary, and a median of 26.6 minutes.

After estimating the average total time for all online diary users in 2021, CUs were split up by the type of device used. The data in Table 3.4.2 show that ‘Desktop Only’ users spent the most time in the diary, with a mean of 38.9 minutes and a median of 29.7 minutes. ‘Mixed Device’ users trailed ‘Desktop Only’ users in both the mean and median estimates of total time, but only slightly with 37.2 and 29.0 minutes respectively, while ‘Mobile Only’ users lagged both by a larger amount. The mean of total time spent in the diary for ‘Mobile Only’ users was 22.8 minutes and the median just 13.2 minutes, roughly 16 minutes less than both the mean and median estimates for ‘Desktop Only’ users.

Next, monthly average total time spent in diary by 2021 users was analyzed. Figure 3.4.1 below illustrates this variation over the course of 2021. The data show that average total time spent in the diary per user fluctuated between a low point in February (mean of 29.9 minutes and median of 20.6) and a high in September (mean of 41.4 minutes and median of 28.7). When observing the mean of total time spent in diary, the graph appears to trend upward in the latter half of the year, but this trend is less apparent when examining the median.

Estimates of median total time spent in the diary per user did not vary as much as the mean over the course of 2021, but did see a roughly 10 minute difference between the low point of 20.6 minutes in February and the high point of 30.9 minutes in July.  

Finally, median time spent in diary for 2021 CE Production was compared to estimates from the LSF Test of Online Diaries. Figure 3.4.2 below illustrates this comparison for all users from each sample, and by device type used. The data show that median total time spent in diary was higher for LSF online diary users than for 2021 CE production diary users, with LSF users spending about 3.4 minutes longer in the diary (27.6 minutes compared to 24.2). Median total time for LSF users was higher across all device types. The largest differences were in the “Mixed Device” use category (11.8 minutes) and the “Mobile Only” use category (6.3 minutes). In the case of “Desktop Only” users, which accounted for the largest share of diary users, the difference in median total time between the LSF and 2021 CE Production was less pronounced at just 2.6 minutes.

The next comparison between 2021 production and the LSF, shown below in Figure 3.4.3, examined time spent in the diary, but for each individual diary day. This measure summed the total time spent entering expenses in the diary by users on each diary day (Day 0 - Day 16). In Figure 3.4.3, the median of this total time per diary day measure is depicted at the CU level. The data show that diary time for users in both 2021 production and the LSF was higher in the second half of the two-week diary keeping period. It should be noted that time was only counted on days where a user interacted with the diary, so for example, a user with no time spent on Day 5 would not be counted as a 0 when calculating the Day 5 median time. The median time spent in the diary per day for 2021 production did not fluctuate as much as in the LSF, but generally followed the same upward trend. In both samples, the final days of the diary keeping period had the highest median times, which would lead us to believe that most respondents save the bulk of their reporting for the end of the period. One notable exception to this was the jump in median LSF user time from 4.9 minutes on Day 3 to 9.2 minutes on Day 4. This jump in the LSF line is most likely explained by the midweek reminder that took place on Day 3, as part of the LSF protocol. This protocol remained in the CE production fielding protocols, but the equivalent increase does not appear in the CE production data. This may have been due to the fact the FRs were not required to make a note of the reminder contacts after the conclusion of the LSF, leading to less follow-up.

The ‘Time Spent in Diary’ section results showed that within 2021 CE Production, users with higher overall login rates (e.g., “Desktop Only’ and ‘Mixed Device’ users shown in Table 3.4.2) translated to more time spent in the diary. This connection between login rates and time spent in the diary did not hold as true when measured by diary day. The differences between Figure 3.2.3 and Figure 3.4.3 illustrates that relatively high login activity in the first few days of diary keeping does not necessarily equate to higher median times spent in the diary. The inconsistency is likely due to online diary placement protocols, in which Census Field Representatives are encouraged to help respondents login for the first time at placement (Day 0), but typically no expenditure reporting occurs at this time. The two series do show some connectivity toward the end of the diary keeping period, where the median time spent increases nearly three minutes from Day 11 (6.7 minutes) to Day 13 (9.6 minutes). In the next section, more standard measures of data quality used in the CE Diary Survey will be looked at alongside some of the paradata measures from this report.

3.5 Paradata and CE Data Quality

To better understand the relationship between the paradata measures presented in this paper and data quality in the CE Diary Survey, the online diary paradata were analyzed alongside more traditional indicators of data quality used in the CE program. Specifically, expenditure totals and expenditure counts were analyzed by login activity, user device type, and time spent in the diary. In order to make this comparison, the online diary paradata first needed to be combined with the final 2021 CE production diary data. In this section, the sample of CUs examined is subset to only those associated with complete diaries, with at least one successful login, and with expenditure information. A total of 1,490 CUs from the 2021 online diary paradata met the previously described criteria and were successfully combined with the corresponding expenditure data.

The next set of figures examine the relationship between findings from the paradata and traditional indicators of CE Diary data quality. First, median total expenditures and expenditure counts were analyzed by the total number of logins for a given CU, using 10.5 for the group cutoff point as the median for total logins was 10 (see Table 3.2.1). The data in Table 3.5.1 below show that for CUs with 10.5 or greater logins, both median total expenditures and expenditure counts were higher. With respect to total expenditures, the median for CUs in the relatively high login group was roughly $279 higher than in the low login group. In the case of median expenditure counts, the high login group exceeded the low login group by 8.5 entries.

After looking at data quality measures by relative login activity, expenditure totals and counts were analyzed by user device type. The results, shown in Table 3.5.2 below, illustrate that ‘Desktop Only’ and ‘Mixed Device’ users were only separated by about $4 in median expenditure totals and 1 entry count, while median estimates for ‘Mobile Only’ users lagged both groups by over $155 and roughly 6 entries. Following the findings in the previous table on data quality and login activity, it is not surprising to find that median expenditure totals and counts were higher for ‘Desktop Only’ and ‘Mixed Device’ users, as these groups were associated with higher login activity when analyzed in section 3.3 of this paper.

Finally, the data quality measures were analyzed by time spent in the diary, which used a method similar to the login activity comparison. All CUs with a total time in diary greater than or equal to 27 minutes were sorted into the relatively high group, and those with less than 27 minutes spent in the diary were classified into the relatively low group. It should also be noted that the number of observations is slightly lower for this comparison (n=1,414) as not all CUs had sufficient activity in the paradata for a total time to be calculated.

The data in Table 3.5.3 show that the estimates for median expenditure totals and median expenditure counts were much higher among those CUs with relatively greater time spent in the diary. In the 2021 data, median expenditure totals were roughly $474 higher for those in the group with the most time spent in diary, and the median expenditure counts were 20 entries higher. This tracts with the findings related to login activity and data quality, as high logins are associated with greater total time spent in the diary. Of the three paradata measures examined in this section, time spent in diary appears to be most strongly associated with higher expenditure totals and expenditure counts. It should be noted that the paradata does not allow data users to determine the specific cause of a given diary user’s time spent in the diary. In other words, paradata cannot be used to distinguish between respondents whose time spent in the diary is relatively higher due to having more expenditures to report, and those whose time spent in the diary is relatively higher due to them being a more conscientious respondent.

4. Conclusions & Next Steps

4.1 Conclusions

The evaluation of CE Online Diary paradata confirms that the addition of paradata into CE production greatly increases the programs’ ability to analyze CE Diary respondent behavior. The granular detail of user activities captured in the paradata allows researchers to look at diary keeping behavior from new angles (i.e., time spent in the diary and logins) that they otherwise would not have access to using only the paper form. Time spent in the diary per user and total logins per user were associated with higher expenditure totals and expenditure counts, showing that the paradata not only offers new information, but potential insights to diary data quality.

The device use information in the paradata also provide a unique view of how respondents interact with the CE Online Diary. The findings in this report suggest that respondents who incorporated non-mobile devices (i.e., desktop computers) into their diary keeping were associated with higher quality expenditure reporting than those only using a mobile device.

4.2 Next Steps

With these conclusions in mind, the authors of this report would recommend that average total logins per user and average total time spent in the diary per user be incorporated into the Data Quality Profile as standard metrics for the CE Diary Survey. The findings of this paper also support the inclusion of a metric for tracking the patterns of device use by CE Online Diary respondents, as desktop use was also associated with higher quality data. In addition to tracking device use, the authors recommend a deeper analysis of mobile-only users, which would seek to determine if variance in reporting behavior or actual differences in spending are responsible for the differing results by device group. If the measurable differences in data quality for mobile-only device users are in fact driven by sub optimal reporting behaviors, it may be beneficial to encourage respondents to use non-mobile devices for diary keeping, or to screen out users that only access the internet with a mobile device. This is of course not possible for all respondents, but the addition of a question prior to placement asking, “if they had a device other than their phone to fill out the diary with?” could improve our determination of who is ultimately offered the online diary mode.

4.3 Recommendations for Paradata Improvement

The authors also recommend that the calculation of ‘time spent in diary’ continues to be worked on for the most precise estimate. Currently, the paradata at the event level for hyperlink actions lacks sufficient detail for researchers to differentiate between all actions, meaning that some user actions are being excluded from the total time estimation that perhaps should not be. For example, a meaningful number of these hyperlink actions recorded in the paradata had a value label of “undefined button,” making it unclear whether these actions should be factored into the time total calculation.

[1] The Census Bureau collects data for many Federal agency's surveys, including the Bureau of Labor Statistics (BLS) Consumer Expenditure Surveys (CE). During the onset of the COVID-19 pandemic, the Census Bureau put a temporary hold on all in-person contacts for household interviews.

[2] The Gemini Project redesign plan was approved in 2013 to address issues of measurement error and respondent burden in the Consumer Expenditure (CE) surveys.

[3] The LSF Test of online diaries was administered between October 2019 and March 2020.

[4] The LSF Test resulted in a preliminary and a final report.

[11] Diary Days refer to the days following an expenditure diaries placement with a responding CU. The day of a diary’s placement is considered Day 0, and each of the following days are counted in sequence until the diary keeping period ends, and the diary is collected.

[12] Diary Days we restricted to Day 0 - Day 16 in the CUs expenditure diary keeping period. Some login events in the paradata were recorded outside the diary keeping period and were omitted from the analysis in Figure 3.2.1