Department of Labor Logo United States Department of Labor
Dot gov

The .gov means it's official.
Federal government websites often end in .gov or .mil. Before sharing sensitive information, make sure you're on a federal government site.

Https

The site is secure.
The https:// ensures that you are connecting to the official website and that any information you provide is encrypted and transmitted securely.

Bureau of Labor Statistics > Office of Survey Methods and Research > Publications > Browse Research Papers

What Is Gained by Asking Retrospective Probes after an Online, Think-Aloud

Abstract

Recent research (Behr, 2012; Meitinger and Behr, 2016; Lenzer and Neuert, 2017) has demonstrated that cognitive testing can be moved online through the use of web-based probing. Researchers note that the benefits of web probing include faster and less expensive recruiting; access to more diverse and larger sample sizes that can lead to an estimate of measurement error; and by using a self-administered mode, the control of interviewer effects, thereby increasing the reliability and comparability of results. Additionally, findings suggest that web probing and cognitive interviewing uncover similar problems that ultimately lead to the same question revisions. However, in discussing limitations of their study, Lenzer and Neuert (2017) mention that, of several possible cognitive interviewing techniques, they applied only one technique: verbal probing. They also suggest that given the technical feasibility of creating an audio and screen recording of a web respondent’s answering process, future studies should look into whether web respondents can be motivated to perform think-aloud tasks while answering an online questionnaire.

Fortunately, online platforms currently exist that can provide the functionality described by Lenzer and Neuert. These platforms provide an audio and screen recording of a participant completing online tasks, and have been used successfully to conduct think-aloud, cognitive interviews (Mockovak and Kaplan, 2016). This study demonstrates how one of these platforms was used to conduct a think-aloud test of survey questions that asked about the cognitive demands of occupations. The study also expanded the research objective by demonstrating the additional benefits of asking retrospective probe questions after the think-aloud procedure. This objective ties into the greater challenge of identifying best practices when testing questions online.

Twenty five cognitive interviews were successfully conducted. A total of 41 potential problems were uncovered, with 78% identified in the think-aloud section, and an additional 22% problems identified in the retrospective, web-based probing section. The types of problems identified dealt mostly with comprehension and response-selection issues. Findings agreed with results obtained in a field test of the interviewer-administered questions, with results from both studies being used to revise the survey questions.