By Paul O'Donohoe, Manager Health Outcomes, CRF Health
There are many things to consider when developing a questionnaire – the concepts of interest, wording of questions, recall period, response options, a Likert scale versus a visual analog scale versus a numeric rating scale, etc. However, an element that is often overlooked is the actual layout of the questionnaire, i.e., how the questions and responses that the respondent will see and interact with are actually going to look on the screen or page. For sure, great importance is given (or should be given) to ensuring that the questionnaire is easy on the eye and intuitive to follow; but how do these decisions impact the perception of the user?
A paper from Arizona State University published earlier this year (abstract and summary) highlights the power of layout on how users respond to questions. Researchers presented participants with two lists of symptoms; one list was made up of real symptoms associated with a type of cancer, while the second list had symptoms for a fictional type of thyroid cancer. All participants were presented with the same list of symptoms; however, they differed in how they were laid out. Some participants were presented with lists in which “common” symptoms (i.e. those we all suffer from, such as fatigue, difficulty concentrating etc.) were clumped together, while other participants were presented lists in which these common symptoms were interspersed with more unique symptoms such as “lump in neck”, etc.
When participants were presented with the “clumped” lists they were more likely to diagnose themselves with the disease being described. The researchers surmised that this is due to the fact that participants got a “run” of positive hits, making it appear more meaningful compared to a mix of positive and negative hits. “Identifying symptoms in “streaks” – sequences of consecutive items on a list that are either general or specific – prompted people to perceive higher disease risk than symptoms that were not identified in an uninterrupted series.”
Researchers also found that the length of the list of symptoms presented had an impact, with participants less likely to self-diagnose with a disease when positive hits were part of a longer list. It was concluded that the effect was “diluted” when positive hits were mixed in with a longer list of negatives.
That the layout of a questionnaire can have a statistical impact on the data captured is not a newly recognized phenomenon. Two papers, Christian & Dillman (2004) and Tourangeau et al. (2004), neatly summarize some of the effects that can be produced by simply altering how questions look. Some of the highlights of their findings include the following: