The Online Research Quality Council: Key Take Aways

A  study of 17 panels and over 100,000 interviews by the Advertising Research Foundation’s¹ Online Research Quality Council² conducted in June, 2009 provides, in some cases, counter-intuitive evidence to some of the classic pitfalls associated with panel based online research.

Some of the findings also fall in-line with findings from a white paper entitled: Survey Lockouts: Are we too cautious?³ released jointly by Ipsos and SSI in 2008.

Here are some of the key take aways from their efforts⁴:

• Online Panels – the percentage of overlap is around 16%
• Duplication – or evidence of a respondent taking a survey more than once in the same study – also exists.
• On a typical per study basis, when another source of sampling is needed the duplicate percentages found are generally in the low single digits.
• Taking fewer surveys a month increases the odds of exhibiting ‘bad behavior’; but taking more actually lowers the odds slightly
• Being a member of multiple panels actually lowers the odds of bad behavior occurring by as much as 32%
• Recent academic research on ‘speeding’ (elapsed survey completion time) suggests that ‘total interview time’ may be a biased indicator of quality because it doesn’t take into account endogenous variables such as diligence, accessability of attitudes, and speed of thinking (which may vary by age).
• Multiple panel membership is not a significant driver of survey response quality
• Sample balancing (weighting) survey data to known Census targets, minimally on age within gender, education, income, and region removed variance, but did not completely eliminate it.
• The findings suggest strongly that panels are not interchangeable. Guidelines and transparency about sourcing is needed, when blending or multiple panels are used to fulfill sample requirements.