When Randomization Actually Works: Reducing Position Bias in Mobile Surveys
Enia Xhakaj
Data Scientist
,
Fairing
Survey design is full of tradeoffs, especially on mobile devices where screen space is limited. One consistent challenge is response position bias—the tendency for respondents to favor answer options that are visible without scrolling. Research shows that once scrolling is required, responses below the threshold are selected 40% less often. By randomizing the order, the intent is to distribute this bias evenly across all potential answers.
But that raises an important question: Does randomization always reduce response bias, or are there conditions where it falls short?
The Experiment
To find out, we ran a Monte Carlo simulation to measure when randomization becomes an effective tool.
Sample:
Surveys with varying response option counts (up to 30)
Surveys with 50 to 10,000 respondents
Output: Effectiveness percentage i.e. how much randomization improves fairness compared to a fixed-order survey
Key Insights
Sample Size is Critical
The data illustrates a clear relationship between sample size and randomization effectiveness. Without an adequate sample size, randomization may fail to distribute position bias evenly.
Our research identified the following sample size thresholds:
Small samples (<500 respondents): Minimally effective. The statistical benefits of randomization are not fully realized, meaning that randomization helps, but the effect isn’t strong enough at this scale to justify relying on it alone.
Medium samples (500-2,000 respondents): Moderately effective. Within this range, randomization begins to yield a tangible, moderate improvement in data quality.
Large samples(>2,000 respondents): Highly effective. For surveys of this scale, randomization is a powerful and highly effective method for enhancing the accuracy and validity of the results.
More Response Options = More Complexity
The number of response options also influences the sample size required for randomization to work.
Our research identified the following response option thresholds:
0-10 options: Randomization can achieve effective results more readily within this range.
10+ options: A significantly larger sample size is required for randomization to be effective. For lists exceeding 10 options, there may be diminishing returns even with a large respondent pool.
What This Means for Survey Implementation
For many survey designers, randomization is treated as a default best practice. But our analysis shows that both sample size and response option count matter.
Small surveys (<500 respondents): Randomization helps, but the effect isn’t strong enough to justify relying on it alone. The better strategy is to design mobile-friendly surveys with fewer response options (7-10) to avoid position bias from the start.
Medium surveys (500-2,000 respondents): Randomization is helpful and is recommended. This is the threshold at which randomization becomes a worthwhile strategy, providing a meaningful reduction in position bias.
Large surveys (>2,000 respondents): Randomization is critical and strongly recommended. For large-scale surveys, this technique is a highly effective method for ensuring equitable option presentation and significantly improving data quality.
Randomization isn’t one-size-fits-all. Its value depends on both the scale of your survey and the number of response options presented.
The Bottom Line
Survey randomization on mobile works, but the decision to randomize response options should not be an automatic one. With sufficient sample size and a manageable number of response options, randomization is a powerful tool for mitigating position bias. But when surveys are small or overloaded with responses, its benefits taper off.
As always in survey design, context is key.
Have additional questions about randomization in survey design? We’d love to chat – book time with us here.
