Try Felix For Free. No Credit Card Required. Get Started.

What’s in a Question?: Five Mistakes to Consider in Survey Design


What’s in a Question?: Five Mistakes to Consider in Survey Design

At New Story, we use the Lean Participatory Design approach to plan affordable housing communities in Latin America. This approach is grounded in continuous community surveying and data collection, focusing on empathy and humility.

When working with local community members, New Story uses Felix to optimize survey design and overall data collection. We have learned the best practices and strategies for a more effective survey design through trial and error.

Five factors to consider with survey design

Review your word choices

Always contextualize the wording in your questions to your audience by taking into account local customs and culture. Word choice will be dependent on the community and geographical location of who you are surveying. 

For example, one survey may look completely different in the U.S. versus in El Salvador. That same survey may look different in two Spanish-speaking countries, such as El Salvador and Mexico.

Due to the various social and cultural meanings of words and different grammar structures, it is incredibly important to be mindful of the communities you are interviewing. In these settings, Felix’s multilingual capabilities helps foster inclusive surveying. 

Avoid leading questions

Leading questions are a common mistake found in survey design. Such questions insinuate the participant to answer in a particular manner or the ‘expected’ manner. To avoid leading questions, you must sim for neutral wording. At first, leading questions are hard to detect; however, the main component to be aware of is value-based words, such as ‘new’ or ‘better.’ 

Here is an example of a leading question: Tell us why your current housing situation is not the best? An alternative to this leading question is: How is your current housing situation? 

It is often better for questions to be as open-ended as possible to avoid biases. Hearing people’s stories in their own words also builds mutual-trust and a better understanding of the participant’s and community’s needs. 

Steer clear from loaded questions

Loaded questions are also quite common in survey design. Like leading questions, loaded questions push the participant in a specific direction, to answer in a possibly less natural way. 

For example, a loaded question may be: What kind of sports do you play? 

However, what if the person answering does not play sports? Alternatively, you could ask: What do you do in your free time? Or What are some of your interests? Again, neutral and open-ended questions are the most useful for surveys when trying to get honest responses. 

Be mindful of answer options

The exciting thing about surveys is that there are many ways to answer questions, whether through discrete yes/no questions or rating-scale questions. Such options can create trade-offs; however, there are methods for combining them via conditional logic frameworks.

This method not only can save time, but also help reduce non-relevant data. Felix can depict conditional logic questions within its survey builder so you only get the data you want and need. This also helps reduce survey fatigue if not all questions apply to your recipients.

Here is an example: Do you have clean drinking water? Yes/No. 

Simplify your questions

Double-barreled questions are when two (or more) questions are asked within one larger question. To avoid such questions, you can split the larger one into separate, more concise, direct questions. 

For example, a double-barreled question may be: How long do you walk to the well, and do you have access to drinking water? 

In this case, we want to understand the community’s current water situation. However, this complex question may lead to a long-winded answer with not all of the data you need. 

Instead, you can ask: 

  • Do you have access to drinking water in your community? Possible answers can include yes, inside my home/yes, at the town well/no.
    • If you have a town well, how long do you walk to get water? Answers can be different options for time — from 10 minutes to 3 hours.

In the end, survey design is an iterative process. For nonprofit and public service organizations to reach their impact goals, it is vital to be aware of the subtle biases we may include in our survey designs. Being mindful of the five factors above can lead to more honest answers from your aid recipients and help you meet your goals faster.

When you onboard with Felix, you get our expertise in surveying too. Interested in learning more about how Felix can help your data strategy and analysis? Request a demo.

Written by Minah Kwon