Survey Says: Content Crafting and Priorities
Welcome to the August 2023 edition of the Improved Insights Newsletter: a newsletter for folks in informal STEM learning spaces who want to learn more about how to measure and communicate the impacts of their work.
I’m Sarah M. Dunifon - a long-time STEM educator, evaluator, and equity advocate. After spending time working in museums, nonprofits, and higher education, I now work on cutting-edge informal STEM learning research and evaluation with my team at Improved Insights. We specialize in informal STEM learning and youth programs.
This monthly newsletter includes tips, resources, and news on informal STEM learning evaluation. We hope you find it valuable! Now, let’s get started —
Survey Says: Content Crafting and Priorities
This is the second installment in our series on the complexity of survey crafting. In this month's newsletter, we'll talk more about content development, and some key areas that contribute to the quality of a survey. To read the full article, check it out in our Insights. In the meantime, here’s a taste:
Welcome back to part two of our series on survey crafting! Last month, we looked at the logistics of the survey-building process. We talked about some of the initial steps an evaluator might take when creating a survey for a client, and a few important factors that I keep in mind as I’m developing an instrument.
This month, we will dive deeper into crafting an effective survey or questionnaire. Let’s look at a few key areas that contribute to the final design of a survey:
[...]
Audience: Our audiences play a major role in the design of a survey. Separate from considering the priorities of the interested parties (e.g., the program manager, the educator, the community, the funder, the development team, etc.), we must consider the audience we’re working with (e.g., the students in the program, their families, program participants, etc.). Which questions are asked and how they are asked, will look different depending on a number of factors. For example, consider the differences in designing for a group of middle school students versus a group of adults. Evaluators think about all sorts of factors, like developmentally appropriate questions and constructs, literacy level, and accessibility. Other factors to consider are the intended audience’s comfort level with technology (which may impact the design and administration of the instrument - e.g., paper or tablet/computer), and whether certain jargon or references will be familiar to them (which can vary widely depending on language background, cultural context, educational background, and more).
Context: There are always inherent differences in any program, and there is not a one-size-fits-all model that can effectively capture that variance. Developing a survey that considers the unique structure of each program is essential to generating useful data. A few questions I ask myself are:
What does the program design look like?
How long are the engagements?
What is realistic to expect to see changes in, based on the context?
Logistically, how much time and attention do we have from our audience for the survey?
How many people could feasibly complete the survey?
60-Second Suggestions
Here are a few things that caught our eye this month, usually pertaining to informal STEM education and evaluation, but occasionally some fun personal stuff, too.
The Afterschool Alliance’s funding database is a comprehensive repository of informal education funding opportunities available across the United States. Narrow your search by state, grade level, or program area to find a grant that is perfect for you!
Intersections in Assessment Conference: Exploring Informal Science Learning Assessment Methods to Improve Undergraduate STEM Education, a report generated by an NSF IUSE-funded project through the Institute for Learning Innovation, examines the potential that evaluation methods used in Informal Science learning (ISL) have in informing new approaches to Higher Education assessments.
I recently received the Apexel MS008 Portable Microscope as a gift, and have had a blast checking out insects, plant matter, sand, and other small items. This little microscope has surprisingly good zoom capabilities and even has photo capability. It would make an awesome present for someone in your life (or even you!) who is interested in science.
Thanks for tuning in to our two-part series on survey building! Our aim was to provide you with an inside look into the intricacies of developing these kinds of instruments and capture the multi-faceted nature of this work. We all interact with surveys of various types on a frequent basis. Next time your favorite retailer sends you an email requesting your feedback, take five minutes to peruse their questions. You will likely learn a lot about their priorities, and find that you are now a more informed consumer!
Until next time - thanks!