Getting Smart About Research

This work is licensed under a Creative Commons Attribution-NonCommercial 2.5 License.

This guest blog post is by Polly Karpowicz, MBA, CAE, and Elizabeth Weaver Engel, M.A., CAE.

Association executives use research all the time. We run surveys and focus groups to better understand our members’ most important goals and most pressing problems so we can make good decisions about the programs, products, and services we offer to meet those needs. We sponsor industry studies to help our members and other audiences make good decisions about investing their resources wisely. We consult third-party (university, government, NGO) data sets and studies to understand the STEEP (social, technological, environmental, economic, and political) trends that influence the operating environments of the professions and industries we serve. 

The problem is that most of us lack formal training in research methods.

How can you be sure that the research you’re using or sponsoring is giving you the insight you need to make good decisions? How can you protect your association’s reputation as a trusted source of unbiased information for the profession or industry you serve?

You need to build your information literacy muscles.

That involves asking questions like:

• How was the data the study uses collected? 
• Who was included in the research? Who was excluded?
• Who collected the data? Are those sources reputable? 
• Does the study draw conclusions without data to support them? 

It also involves familiarizing yourself with the terminology of research (sampling methods, margin of error, confidence interview, statistical significance, data hygiene, different types of research design bias) and with your options for collecting and interpreting data and the pros and cons of each, for instance:

Quantitative research, which is designed to ask questions and collect responses numerically, with the goal of coming to the kinds of answers that can be described by levels of statistical significance and degrees of confidence. Surveys are the most common form of quantitative research, and they provide reassuringly specific answers. However, surveys are particularly susceptible to design flaws in either the questions, the answer options, or both that can bias the results.

Qualitative research, which is designed to gather data through conversation. Interviews and focus groups are the most common types of qualitative research. While qualitative results are necessarily impressionistic and anecdotal, they allow you to delve into subjects’ motivations, the why behind what they think, prefer, and do. However, interviewers walk a fine line between creating a high level of trust with interview subjects, which is necessary for success, and inappropriately influencing their responses which can bias results.

Does this all sound complicated? 

It’s important for associations to get this right, both so that association executives have the best possible chance of making good decisions about how to invest limited association resources to generate the best return for members, and because associations are viewed as trusted, unbiased sources of information for the members and other audiences we serve. It’s incumbent on us to provide quality research products so we remain worthy of that trust. 

You can learn more about these and other critical research concepts and read stories of associations doing great work in their research programs to serve their members and the professions and industries they serve in our new whitepaper, Caveat Emptor: Becoming a Responsible Consumer of Research, in which we tackle the sometimes thorny issue of what you need to know and do to be a savvy consumer and sponsor of research even if you DON’T have a formal background.

The monograph also includes:

• An interview with Dr. Sharon E. Moss, co-editor (with Sarah C. Slater) of The Informed Association: A Practical Guide to Using Research for Results, on ethical practices in research.
• An interview with Dr. Joyce E. A. Russell, The Helen and William O’Toole Dean at Villanova School of Business, on developing discernment in assessing research.
• An interview with Jeff Tenenbaum, Managing Partner at Tenenbaum Law Group PLLC, on avoiding antitrust liability.
• Case studies with the American Association of Colleges of Pharmacy, the Association of American Medical Colleges, the Casualty Actuarial Society, and IEEE.
• A plain English review of key research terms, and a brief explanation of the rules of formal logic (and how they affect research work).
• Recommendations for books, articles, websites, podcasts, and courses you can use to improve your research skills.
• A series of thought questions for you to use to spark discussion with your team.
• An extensive list of resources in case you want to dig deeper on any of the topics addressed.