An official website of the United States government
United States Department of Labor
United States Department of Labor
The .gov means it's official.
Federal government websites often end in .gov or .mil. Before sharing sensitive information,
make sure you're on a federal government site.
The site is secure.
The
https:// ensures that you are connecting to the official website and that any
information you provide is encrypted and transmitted securely.
Office of Survey Methods Research
SHARE ON:
More Information
Published Year: 2020
Survey/Program(s): General Methodology
Catalog: Statistical Survey Paper
Download PDF
What Is Gained by Asking Retrospective Probes after an Online, Think-Aloud
Abstract
Recent research (Behr, 2012; Meitinger and Behr, 2016; Lenzer and Neuert, 2017) has demonstrated that cognitive testing can be moved online through the use of web-based probing. Researchers note that the benefits of web probing include faster and less expensive recruiting; access to more diverse and larger sample sizes that can lead to an estimate of measurement error; and by using a self-administered mode, the control of interviewer effects, thereby increasing the reliability and comparability of results. Additionally, findings suggest that web probing and cognitive interviewing uncover similar problems that ultimately lead to the same question revisions. However, in discussing limitations of their study, Lenzer and Neuert (2017) mention that, of several possible cognitive interviewing techniques, they applied only one technique: verbal probing. They also suggest that given the technical feasibility of creating an audio and screen recording of a web respondent’s answering process, future studies should look into whether web respondents can be motivated to perform think-aloud tasks while answering an online questionnaire.
Fortunately, online platforms currently exist that can provide the functionality described by Lenzer and Neuert. These platforms provide an audio and screen recording of a participant completing online tasks, and have been used successfully to conduct think-aloud, cognitive interviews (Mockovak and Kaplan, 2016). This study demonstrates how one of these platforms was used to conduct a think-aloud test of survey questions that asked about the cognitive demands of occupations. The study also expanded the research objective by demonstrating the additional benefits of asking retrospective probe questions after the think-aloud procedure. This objective ties into the greater challenge of identifying best practices when testing questions online.
Twenty five cognitive interviews were successfully conducted. A total of 41 potential problems were uncovered, with 78% identified in the think-aloud section, and an additional 22% problems identified in the retrospective, web-based probing section. The types of problems identified dealt mostly with comprehension and response-selection issues. Findings agreed with results obtained in a field test of the interviewer-administered questions, with results from both studies being used to revise the survey questions.