Horizontal vs. Vertical Scales vs. Use of a Grid in Online Data Collection: Which Is Better?

William Mockovak

Abstract

When designing online questionnaires, survey designers often have the option of choosing between the use of horizontal or vertical rating scales, or possibly a grid, if several survey questions use a common response scale. Previous research has investigated use of these alternative scales, but with inconsistent findings. As a result, some researchers advocate use of horizontal scales, others recommend the use of vertical scales,1 whereas the use of grids has been generally discouraged. Although the use of alternative scales has been heavily researched, the familiarity and expertise of online respondents with different question formats have continued to increase, so studies that were done years ago may no longer be relevant. Therefore, this study revisited the topic of question format and its impact on data quality. Using online instruments that presented the exact same question order, content, and scale direction (from positive to negative), unipolar horizontal and vertical scales were compared across two questions, and with the use of a grid in four additional questions. All individual questions and the grid appeared on separate web pages. Participants were recruited through Amazons Mechanical Turk and saw only one version of the questions. Across the six questions that compared horizontal or vertical scales (N=193 and 229, respectively), no significant differences were found on five of the questions. On one question, the use of a horizontal scale led to significantly higher ratings. In the comparison of the 4-question grid (N=279) to the same questions presented using horizontal or vertical scales, three of the questions showed no significant differences among any of the contrasts (vertical vs. horizontal vs. grid). The only significant difference occurred on one question where the mean of the grid question was significantly higher than the mean for the horizontal scale, but not significantly higher than the vertical scale. Data quality comparisons among the grid, horizontal, and vertical scales were also explored, but no significant differences were found in scale reliability (measured with Chronbachs alpha), in the amount of straight-lining that occurred, in item non-response, or in the resulting factor structure. In summary, the question formats studied did not consistently affect the results or associated measures of data quality for the questions in this study.e