Survey Says: Revealing the Intricacies of Survey Crafting
Welcome to the July 2023 edition of the Improved Insights Newsletter: a newsletter for folks in informal STEM learning spaces who want to learn more about how to measure and communicate the impacts of their work.
I’m Sarah M. Dunifon - a long-time STEM educator, evaluator, and equity advocate. After spending time working in museums, nonprofits, and higher education, I now work on cutting-edge informal STEM learning research and evaluation with my team at Improved Insights. We specialize in informal STEM learning and youth programs.
This monthly newsletter includes tips, resources, and news on informal STEM learning evaluation. We hope you find it valuable! Now, let’s get started —
Survey Says: Revealing the Intricacies of Survey Crafting
Have you ever whipped up a survey at the last minute to collect data, only to later discover that the data you collected wasn’t very useful? You know what they say - garbage in, garbage out.
In this month's newsletter, we'll talk about the complicated process of developing a survey, exploring the development through an evaluator’s lens. To read the full article, check it out in our Insights. In the meantime, here’s a taste:
Professional evaluators know how complicated the survey-building process is and typically will budget enough time to carefully consider all aspects of their design. But what does the crafting process look like? The following are a few of the steps that I as an evaluator take when I am constructing a survey for a client:
I usually start with conversations with various users about what they’re interested in measuring and how the data might be used. A few examples of the people I typically interview are program managers, coordinators, directors, funders, developmental departments, and occasionally program participants.
Then, I might follow with a literature review of similar studies or programs to identify if any established measures are available, how other researchers or evaluators have approached this topic, best practices for designing with a certain audience in mind, etc.
Next, there will be a period of thinking and alignment where I (the evaluator) connect the client’s goals with new or established tools, adapting or creating questions when needed. This stage is also when I start to think about the best way to ask a survey question. Do scale-based questions make the most sense for this client? Or do open-ended questions present a better method of obtaining the information needed? There are many different types of questions that could reflect partial or similar answers to what a client is looking for, but understanding the nuance of the client’s goals is what ultimately makes the difference between a good instrument and a great instrument.
The next step is to figure out logistics, like when data needs to be collected, how much, and through what medium.
Finally, once I have a grasp on the content of the survey and how I want to approach each data point, I will build out the instrument. I’m careful to avoid common survey mistakes like double-barreled questions (more on that later), and to take my time to ensure that questions are clear and specific. I pay attention to the length of the instrument, and the intro and outro language that explains the study and provides the participants with information to inform their consent.
60-Second Suggestions
Here are a few things on my radar this month. These usually pertain to informal STEM education and evaluation, but occasionally I’ll highlight some fun personal stuff, too.
This recorded session from the American Alliance of Museums addresses the Museum Social Impact Study (MOMSI) project, exploring results of the project’s mission to validate a social impact instrument, and how the study’s goals have been achieved.
This report published by the Georgetown University Center on Education and the workforce addresses diversity and equity within engineering occupations by examining demographic data, finding that a mere 3% of engineers working in the field in 2019 were either Black or Latinx women.
The FIFA Women’s World Cup is going on now, taking place in Australia and New Zealand. As a former soccer player myself (through the very impressive 5th-grade league), I’ve been enjoying watching the incredible athleticism of the U.S. team.
The piece highlighted above is the first in a two-part series on survey crafting that will be published on our Insights page. Next month, we will take a closer look at the content-development process, and talk more specifically about how I assess a client’s goals in order to create an effective tool for them.
Until next time - thanks!