Much like the old adage “it takes a village to raise a child,” it also takes a village to create a successful survey. Wiley’s annual Society Membership Survey is written with input from colleagues across the business sharing their expertise and insights. In order to succeed, the final survey needs to be easy to understand, unbiased, and as streamlined as possible. It’s trickier than it sounds. I sat down with my colleague Trina Cody, Senior Manager, Customer Experience to find out what makes a survey successful and how to get results that will help you make decisions.

Q: So let’s start at the beginning. With surveys, where do you start?

TC: Always start with the goal of the survey. Think: What is the question you are trying to answer? It’s important for us to be very clear about this, so you don’t end up with “question creep,” which is the scenario when you start to add in everything you can think of. This makes for a terrible survey experience for the respondent, and will lower the completion rate for the survey. It’s also important to think about why you want this information. There should always be a purpose to the survey, and actions that you take from the results. For us, a general “what do our members want?” survey isn’t good enough if we aren’t willing to act on that information.

Once there’s a clear purpose to the survey, it becomes much easier to write. When I write questions, I always ask myself: what am I going to do with the answer? How am I going to make changes based on these results? If I can’t answer that question, then I know I need to take it out of the survey. It's also worth noting that for surveys like the membership survey, we often use outside market research firms to perform the statistical analysis. Their experience really can't be beat. For the membership survey, we used a company called Broadview Analytics.

Q: That makes sense. Focusing on the action means that the survey isn’t a waste of time for us, but more importantly it isn’t a waste of time for the respondent. I think for me, one of the hardest parts of writing a survey is avoiding bias in the questions. We have a lot of experience working with and supporting members, so how can we remove any assumptions when we’re writing the survey?

TC: Language is key. We always try to use simple, non-leading language in the questions. We never want to force someone to pick an answer that doesn’t represent their thinking because it makes your results lose validity. I always make sure to include options like “don’t know,” or “unsure.” Even better, use “other” with the option to write in an answer. They might be more time consuming to analyze, but written responses should be included to capture the options that our team didn’t think of.

For the membership survey, we also asked several people who weren’t involved in writing the survey to test it. We picked people who weren’t involved with membership, so that they wouldn’t bring the same biases to the table. Doing this let us spot check for any areas where we might have made assumptions.

Q: It really does take a village! When I’ve written surveys, I always try to put myself in the shoes of a respondent who has no context for the questions, but it’s hard. I think asking others to test and review is a great idea.

Our membership survey is a multi-year study. Can you talk a little bit about the benefits of annual testing, or re-testing in general?

TC: Things change. People’s attitudes, opinions and behaviors change over time. They grow older, change jobs, move to a new country. I like to think of surveys as a snapshot in time while real life keeps moving.

With the membership survey, we’re surveying both members and non-members annually so that we can uncover trends and changes in attitudes. This is going to help our society partners with their membership goals. If we didn’t survey every year, we might miss major shifts in thinking, and we would be making strategic decisions based on old data.

Q: It is important to stay current, and to potentially start capturing insights from new members of the research community as they start their careers.

It seems like another benefit of a multi-year study is the opportunity to improve it over time. With the membership survey, were there questions that didn’t turn out the way you thought they would? Did you have to edit or change anything?

TC: Whenever you run a survey, there’s always at least one question that doesn’t turn out like you wanted. In the first membership survey, we asked about engagement with members but we didn’t specify if we meant the level of engagement the society has with its members or the engagement members have with the society.

This year, we asked the question both ways. As it turns out the responses were pretty similar (59% were satisfied with the level of engagement their society has with them, and 60% were satisfied with the level of engagement they have with their society).

I always review the previous survey when I’m writing a new survey, or updating it. I look to see what I can improve upon this time around.

Q: My last question is about demographic information. At the beginning, you talked about always making sure there was something we could take action on with each question we asked. What do we do with demographic information?

TC: For this year’s survey, we were focused on needs-based segmentation. Basically, we wanted to use quantitative, statistical analysis to group respondents based on the benefits they seek from a society, regardless of age, geography, etc. We did this so that we could cut through all of those demographics to get at the core of what the person is trying to accomplish and the job they’re trying to get done. We also ask demographic-based questions so that we can complete the picture of who our participants are. For example, asking a geographic question lets us know where members are most highly concentrated. This year, we learned that countries in central Asia have the lowest percentage of members, but they are also the most likely to join a society in the next 12 months. Those insights can be important for targeting communications and outreach strategies.

For a high-level look at insights from Wiley’s Society Membership survey, you can download a free infographic here.

Image credit: Ryhor Bruyeu/iStockphoto