RMME PhD student, Amanda Sutter, discusses “The Story Behind an Evaluation Practice Survey: Insights from Cognitive Interviews” at the 2023 AEA Conference. Congratulations on this wonderful presentation, from the Research Methods, Measurement, & Evaluation Community!

Author and Presenter: Amanda Sutter

Title: The Story Behind an Evaluation Practice Survey: Insights from Cognitive Interviews

Abstract: Abstract Information: What do evaluators think about when they are asked questions about evaluation practice? What narratives are triggered by what words? This research study focused on understanding the thought processes and beliefs of evaluators as they reflect on their practice through cognitive interviews. Cognitive interviews are a powerful instrument design strategy that helps reveal the narratives and meanings behind survey responses. Building on last year’s field pilot of a new evaluation practice instrument, this research on evaluation study involved cognitive interviews with 20 evaluators recruited through purposive nonprobability sampling to gather diverse perspectives on the evaluation practice instrument. The semi-structured interview process followed the Willis Method, offering participants an easy process to explore their responses in-depth. This paper shares findings from the interviews and how the data will be used to improve the instrument, particularly to ensure that the phrasing of questions and subsequent interpretations align with the intended construct definitions and narrative of the instrument. The paper will be of interest to a variety of audiences. Practitioners and commissions will have opportunities to think about their own surveys and how cognitive interviews may be a useful tool for evaluation studies. Other researchers will also benefit from these opportunities, and they will learn about a potential instrument that may be useful in their own scholarly work.

RMME Faculty member, Dr. Bianca Montrosse-Moorhead, and RMME PhD student, Amanda Sutter, present their workshop, “Building Better Surveys” at the AEA 2023 Conference. Congratulations on this successful presentation, from the Research Methods, Measurement, & Evaluation Community!

Authors: Bianca Montrosse-Moorhead and Amanda Sutter

Presentation Title: Building Better Surveys

Abstract: This skill-building workshop is back by popular demand! Have you ever wondered what strategies you can use to improve your surveys? Or, wanted to create or adapt an instrument that better captures difficult to measure concepts well? Or, wanted to make sure that your survey instrument is able to accurately shed light on the story you are trying to understand? This session will teach you an easy-to-implement mixed-method feedback process to build better surveys. The session will begin with an overview of the process, which includes both a review and interview component. Throughout the session, presenters will focus on the value and use of the process, walk through examples from real-life evaluation studies, and share resources for further learning. Attendees will also have the opportunity to practice using this process through hands-on activities. Attendees will leave the session with templates they can adapt for their own use and guidance to help them feel prepared to take action. Throughout the session, presenters will call attention to ways in which equity is and can be centered in the mixed-method feedback process.

RMME Master’s alumna, Xueying Gao, presents her poster, “Validation of the VI-SPDAT (version 3) instrument: a confirmatory factor analysis” at the 2023 AEA conference. Congratulations on this fantastic poster presentation, from the Research Methods, Measurement, & Evaluation Community!

Authors: Xueying Gao and Brad Richardson

Presenter: Xueying Gao

Abstract: The Vulnerability Index-Service Prioritization Decision Assistance Tool (VI-SPDAT version 3; Community Solutions & OrgCode Consulting, Inc., 2020) is the primary assessment tool to scale individuals’ vulnerability (or self-sufficiency). The Iowa’s Treatment for Individuals Experiencing Homelessness (IA-TIEH) project was conducted to advance an informed, integrated program for homeless and at-risk adults who experience co-occurring disorders, in which VI-SPDAT was applied to the participants at intake assessment and 6-month reassessment to measure their level of vulnerability. The current study adopted the sample from 2020 to 2023 (N=539) to examine the construct validity of VI-SPDAT (version 3) with two factor models: single factor model and a three-factor model. Results suggested that the single factor model did not demonstrate adequate fit (CFI = 0.467, TLI = 0.445, RMSEA = 0.111, SRMR = 0.114), while the hierarchical CFA model demonstrated better fit (CFI = 0.856, TLI = 0.848, SRMR = 0.071, RMSEA = 0.061), suggesting its limitation in measuring individuals’ vulnerability and other outcome parameters in research and clinical practice. Some items were not associated with the global factor or sub-factor. The VI-SPDAT has substantial weaknesses in its theoretical alignment, item performance, and psychometric properties. We recommend the enhancement of a new multidimensional scale of vulnerability with a rigorous measurement development protocol.

RMME student, Jennifer Richardson, successfully defended her doctoral dissertation on September 6, 2023. Her work is entitled, “Mixed Effects Location-Scale Models: Methodological and Substantive Considerations.” Congratulations, Dr. Richardson, from the Research Methods, Measurement, & Evaluation Community!

UConn’s RMME Programs are thrilled to celebrate our newest graduates from RMME Master’s degree program and RMME’s Graduate Certificate in Program Evaluation program! We cannot wait to see all of your outstanding accomplishments as a new RMME grad! Congratulations, Anthony!

RMME student, Alexandra Perez, successfully defended her doctoral dissertation this month. Her work is entitled, “Validity Threats When Comparing Results from Randomized Control Trials and Regression Discontinuity Designs.” Congratulations, Dr. Perez!