marketing consultants for associations and other nonprofit organizations

FREE MARKETING RESOURCES

MARKETING ARTICLES

Comparison of Marketing Research Methods: The Pros and Cons of Research Methodologies Used by Associations and Other Nonprofit Organizations

by John Gunn, CEO, John Gunn Marketing Partners

A variety of market and consumer research methods can help managers of associations and other nonprofit organizations better understand the interests, needs, perceptions and expectations of their members, customers and other publics. This comparison outlines the advantages and disadvantages of telephone surveys, online surveys, focus groups, mail surveys and other marketing research methods used to support strategic marketing decision-making.

Telephone Surveys
Online Surveys
Focus Groups
Mail Surveys
Face-to-Face Interviews
Other Research Methods

Telephone Surveys

Telephone surveys are structured interviews conducted by qualified interviewers using a telephone script and are typically the best method when researchers need to interact with respondents and projectable findings are desired.

Advantages
Telephone surveys offer researchers opportunities to garner a high volume of information from a predetermined number of respondents within a specific and, often, very short timeframe. With some types of audiences, the personal contact aspect of telephone surveys encourages higher rates of response than do other methods. Telephone surveys provide opportunities to engage respondents and probe for complete responses, explain questions or clarify respondents’ answers. Similarly, information can be retrieved from respondents when researchers do not know likely response options, such as questions designed to explore the potential for new product or service offerings or formats.

Pre-testing telephone survey instruments can occur quickly, making it easy to identify and resolve problem areas in survey scripts and specific questions and/or response options that must be adjusted to capture the information needed. Respondents are also more likely to offer frank opinions and possibly embarrassing pieces of information to a faceless interviewer. Appointments and multiple callbacks enable interviewers to reach the individuals in target audiences, and telephone surveys conducted by field service callers can be monitored to ensure uniformity of technique and increase levels of control. Telephone surveys also provide opportunities to collect updated phone numbers, street and e-mail addresses and to note changes in employment.

Disadvantages
Advance notice announcements educating potential respondents about upcoming surveys are needed to raise awareness and break through gatekeepers. Researchers must understand their target audiences’ communication preferences, as telephone surveys may be perceived as intrusive telemarketing. Telephone surveys lack opportunities to use visual props, such as product packaging or marketing communications collateral, and it can be difficult to establish a rapport with the respondents who are free to simply hang up.

Online Surveys

Online surveying is typically quantitative research conducted via the Internet using interactive survey instruments hosted by a service provider or posted on the sponsor’s web site. Qualitative research can also be conducted online using focus groups and bulletin boards.

Advantages
The primary advantages of online surveys are the lower overall costs, the potential ease with which respondents can complete and submit surveys, and time saved due to online databases that tabulate results instantaneously. Survey results can be examined in real-time, 24/7, and quickly and easily disseminated electronically to decision-makers and users of marketing research. Researchers can also ensure responses to specific questions by making key fields required in order to submit questionnaires.

Online research is also useful in testing audio and visual materials, such as web sites or television ads, as visual stimuli that can be presented to respondents. Also, real-time evaluations of products and web sites allow continual feedback for ongoing improvements. It is estimated that as many as 90 percent of people with Internet access say they would rather complete surveys online.

Disadvantages
When conducting online surveys, researchers cannot guarantee the respondents are actually the intended decision-makers the survey was designed to reach. There are also potential confidentiality and security risks when new product concepts and designs are presented to survey respondents via online means. The most effective online surveys are short; therefore, the volume of information collected is limited. Researchers are not able to probe respondents for clarity and open-ended responses are often left blank or incomplete. “Don’t know” options in online surveys tend to be selected at a higher frequency than on telephone surveys where “don’t know” is not a response option that is verbally communicated to respondents.

E-mail and web-based surveys often garner low response rates when the sender is unknown to recipients and e-mail messages may be viewed as SPAM. Similarly, e-mails with links and attachments may be viewed negatively or even discarded by filtering and security software. It is also difficult to establish whether respondents are representative of the larger population, and resistance to online surveys is now near par with resistance to traditional surveys as the novelty of this technology wears off.

Surveys on web sites featuring “Take Our Poll” buttons often suffer from intense self-selection bias and may misrepresent the population(s) being studied. While most Americans now have access to the Internet, some do not and/or researchers do not possess e-mail addresses for all members, and offline research methods may be needed to ensure findings are projectable. Also, people who are willing and able to complete a web questionnaire are likely to be more sophisticated users of technology than are all people with access to the Internet, creating more bias.

Focus Groups

Focus groups are qualitative approximately 60- to 90-minute discussions led by a trained moderator with 8 to 12 relatively homogenous but unacquainted individuals who are brought together to discuss a specific research topic.

Advantages
Focus groups present opportunities for moderators to probe below the surface and learn why respondents think the way they do. They are particularly appropriate for in-depth exploration of new ideas, opinions, perceptions, and reactions to concepts and messaging. Focus groups often serve as exploratory research to assist survey design of subsequent quantitative research methods. Similarly, they can also be useful in validating and/or clarifying results garnered from previous quantitative research and can be used to elicit “in their own words” descriptions of products, services or issues being discussed.

Conferences and other events present relatively low-cost opportunities to conduct focus groups with target audiences. There tend to be few interviewer effects on dialog because individuals tend to be influenced more by the group discussion than by the moderator. Participants are usually enthusiastic and spontaneous in their responses and groups tend naturally to cover more questions, opinions and comments than researchers could have anticipated.

Disadvantages
Results from focus groups are qualitative and nonprojectable to larger populations. They require well-trained moderators to manage discussions, maintain focus, and minimize affects of the personalities and behaviors of individual participants on others and/or the entire group. Logistical and cost considerations include room and food set-up, and incentives are typically needed to encourage response.

Mail Surveys

Mail surveys are conducted by mailing printed surveys, enclosed with reply envelopes, to target audiences using an in-house or commercial mailing list. Mail surveys are best suited to studies requiring respondents to carefully consider their responses, gather information to be reported, or when the specific audience(s) being surveyed is most likely to respond to a mail format.

Advantages
Mail surveys can be inexpensive relative to other methods, particularly when in-house lists are used and an organization’s staff reproduces, distributes and gathers questionnaires. The print survey format allows respondents time to form well-considered answers and gather specific information requested in the survey. Print surveys also provide opportunities to gain qualitative information by asking open-ended questions to illicit personal perceptions and opinions.

Generally speaking, mail surveys can be longer in length than telephone surveys, and respondents are not influenced by the physical appearance, body language or other potential biases introduced by the presence of interviewers and/or focus group participants. Mail surveys also provide opportunities to easily collect change-of-address information to update member or donor records.

Disadvantages
Significant biases, such as sampling and nonresponse bias, may prevent researchers from accurately assessing the current situation. Similarly, mail surveys may require an extended period of time and multiple repeat mailings to achieve response rates suitable for projecting results to larger populations. In addition, respondents to mail surveys may be someone other than the intended recipient (e.g., administrative staff assigned the duty of completing surveys).

Respondents also have unlimited opportunities to change their responses before returning surveys; thus, mail surveys are not advised for studies requiring “top-of-mind” reactions, such as those needed during research studying brand awareness and brand attributes. Mail surveys typically require introductions and repeat follow-up mailings, and many associations find it difficult to rise above other competing print materials mailed to busy members.

Face-to-Face Interviewing

One-on-one, in-depth interviews conducted by a qualified interviewer using a survey script is a method most often used when sensitive issues need to be discussed and materials must be shown or handled. They are also appropriate when respondents are of a “high status” and/or when samples sizes are small and local.

Advantages
Individuals who agree to participate in face-to-face interviews are typically highly motivated to cooperate. Stimuli can be shown to and reviewed by respondents in particular sequences, and interviewers can report observations in addition to question responses. Questions can be explained when necessary, and researchers can probe for more complete responses during face-to-face interviews.

Disadvantages
By their nature, face-to-face interviews typically reach a limited audience of potential respondents and render nonprojectable, anecdotal findings. Costs for travel, meeting locations and incentives can be high; researchers run the risk of encountering “no shows” and significant delays between interviews may be necessary. Interview scripts and questions must be carefully matched to potential respondents’ characteristics to avoid introducing biases, and other biases may occur when respondents distort their responses trying to impress the interviewer or conceal information. It is also difficult to monitor the performance of an interviewer when interviews occur away from a central office.

Other Research Methods Used by Associations and Nonprofit
Organizations

Analyzing Secondary Data – This analysis includes review of internal marketing information, past market research findings, internal membership or donor statistics and demographics, as well as research and statistics compiled by external sources.

Diads/Triads – Discussions among groups of two or three individuals are often a more efficient (time and budget) means of obtaining quick “reads” to satisfy some types of research objectives.

Systematic Observation – Counting, measuring and other natural and mechanical methods of observation to identify behavioral patterns, such as traffic through bookstores at annual conferences.

Experimentation – Ranging in complexity, experimentation helps researchers conduct tests against a control, such as direct mail packages featuring different offers, levels of personalization, inserts, discount incentives and other variables marketers wish to test against a control package.

Web Site Tracking and Behavioral Research – Involves using log-file analysis and web tracking reports to observe how people move throughout an organization’s web site. Web behavior tracking also includes usability studies designed to test web sites before an official launch by observing the ways participants navigate pages.

Mall Intercepts – Mall intercepts are a popular survey method typically used in commercial research, where interviewers randomly approach shoppers and encourage them to complete a, typically verbal, questionnaire. However, the concept can be successfully applied to associations and other nonprofit organizations when the surveys are conducted at the annual meetings and other gatherings of individuals in target audiences being studied.

John Gunn is the CEO of John Gunn Marketing Partners, LLC,
specialists in marketing assessments, research, strategy and plans for associations. For more information, please contact the author at (703) 299-0774, jg@GunnMarketingPartners.com.

Home | Services | About | Contact
Resources | Marketing Links

John Gunn Marketing Partners, LLC
Alexandria, Virginia
Phone: (703) 299-0774 Fax: (703) 299-1106
info@GunnMarketingPartners.com