RMME/STAT Joint Colloquium

Applications of Artificial Intelligence and Natural Language Processing in Educational Measurement

Dr. Matthias von Davier
Boston College

Friday, April 21, at 11AM ET

https://tinyurl.com/rmme-vonDavier

This talk will provide an overview of the applications of Artificial Intelligence (AI) and Natural Language Processing (NLP) in educational measurement, focusing on automated item generation, automated scoring, and test assembly in multilingual assessments. We will discuss the potential benefits of AI and NLP for educational measurement, including increased efficiency, improved accuracy and reliability of assessment, and increased access to assessment technology for low-resource languages. We will examine the current state of the technology, including challenges associated with developing and deploying AI and NLP-based educational assessment systems. We will also discuss future directions for research and development in this area, including the development of methods for assessing and validating AI- and NLP-based systems and the potential for AI and NLP to improve assessment fairness and reduce assessment bias.

Loading...

Taking too long?

Reload document

| Open in new tab

RMME/STAT Joint Colloquium

A Bayesian Nonparametric Approach to Geographic and two-Dimensional Regression Discontinuity Designs

Dr. Luke Miratrix
Harvard University

Friday, April 7, at 11AM ET

https://tinyurl.com/rmme-Miratrix

Geographical and two-dimensional regression discontinuity designs (RDDs) extend the classic, univariate RDD to multivariate, spatial contexts. We propose a framework for analyzing such designs with Gaussian process regression. This yields a Bayesian posterior distribution of the treatment effect at every point along the border, allowing for impact heterogeneity. We can then aggregate along the border to obtain an overall local average treatment effect (LATE) estimate. We address nuances of having a functional estimand defined on a border with potentially intricate topology, particularly with respect to defining the target estimand of interest. The Bayesian estimate of the LATE can also be used as a test statistic in a hypothesis test with good frequentist properties, which we validate using simulations and placebo tests. We demonstrate our methodology with a dataset of property sales in New York City, to assess whether there is a discontinuity in housing prices at the border between school districts. We also discuss application of this method to the context of treatment as a function of two forcing variables, such as falling below a threshold for either a reading or math test.

Loading...

Taking too long?

Reload document

| Open in new tab

RMME/STAT Joint Colloquium

Modeling Coarsened Categorical Variables: Techniques and Software

Dr. Joseph L. Schafer
U.S. Census Bureau

Friday, March 24, at 11AM ET

https://tinyurl.com/rmme-Schafer

Coarsened data can express intermediate states of knowledge between fully observed and fully missing. For example, when classifying survey respondents by cigarette smoking behavior as 1=never smoked, 2=former smoker, or 3=current smoker, we may encounter some who reported having smoked in the past but whose current activity is unknown (either 2 or 3, but not 1). Software for categorical data modeling typically provides codes for missing values but lacks convenient ways to convey states of partial  knowledge. A new R package cvam: Coarsened Variable Modeling, extends R’s implementation of categorical variables (factors) and fits log-linear and latent-class models to incomplete datasets containing coarsened and missing values. Methods include maximum likelihood estimation using an expectation-maximization algorithm, approximate Bayesian and Bayesian inference via Markov chain Monte Carlo. Functions are also provided for comparing models, predicting missing values, creating multiple imputations, and generating partially or fully synthetic data. In the first major application of this software, data from the U.S. Decennial Census and administrative records were combined to predict citizenship status for 309 million residents of the United States.

Loading...

Taking too long?

Reload document

| Open in new tab

RMME Evaluation Colloquium

The Health Profession Opportunity Grant (HPOG) Impact Study: A Behind-the-Scenes Look at Experimental Evaluation in Practice

Dr. Laura Peck
Abt Associates

Friday, March 10, at 11AM ET

https://tinyurl.com/eval-Peck

In 2010, the U.S. Department of Health and Human Services’ Administration for Children and Families awarded Health Profession Opportunity Grants (HPOG 1.0) to 32 organizations in 23 states. The purpose of the HPOG Program is to provide education and training to Temporary Assistance for Needy Families (TANF) recipients and other low-income individuals for occupations in the healthcare field that pay well and aim to meet local areas’ healthcare sector labor shortages. To assess its effectiveness, an experimental evaluation design assigned eligible program applicants at random to a “treatment” group that could access the program or a “control” group that could not. Beyond the impact analysis, the evaluation also probed questions about what drove program impacts, using various strategies. This colloquium will discuss how the HPOG 1.0 impact study was designed/implemented and introduce attendees to various design and analysis choices used by investigators, in partnership with the government funder, to address research questions. Specific topics will include: experimental design, multi-armed experimental design, experimental impact analysis, planned variation, natural variation, endogenous subgroup analysis, evaluation in practice.

Loading...

Taking too long?

Reload document

| Open in new tab

RMME/STAT Joint Colloquium

Bookmaking for Binary Outcomes: Prediction, Profits, and the IMV

Dr. Ben Domingue
Stanford University

Friday, February 24, at 11AM ET

https://tinyurl.com/rmme-Domingue

Understanding the “fit” of models designed to predict binary outcomes is a long-standing problem. We propose a flexible, portable, and intuitive metric for such scenarios: the InterModel Vigorish (IMV). The IMV is based on a series of bets involving weighted coins, well-characterized physical systems with tractable probabilities. The IMV has a number of desirable properties including an interpretable and portable scale and an appropriate sensitivity to outcome prevalence. We showcase its flexibility across examples spanning the social, biomedical, and physical sciences. We demonstrate how it can be used to provide straightforward interpretation of logistic regression coefficients and to provide insights about the value of different types of item response theory (IRT) models. The IMV allows for precise answers to questions about changes in model fit in a variety of settings in a manner that will be useful for furthering research with binary outcomes.

Loading...

Taking too long?

Reload document

| Open in new tab