Meta Science

Advancing Behavioral Science With Open Large Language Models
Addressing the Zoo of Psychological Measures and Constructs With Large Language Models
From a Zoo of Measures to a Single Trait: The Structure and Assessment of Impulsivity
Policy Science

Advancing Behavioral Science With Open Large Language Models

The rapid advancement of large language models (LLMs) is transforming scientific practice across disciplines, including the behavioral sciences. These models, trained on vast text corpora, have demonstrated remarkable capabilities that range from text generation to complex reasoning tasks. As the scientific community grapples with how to responsibly integrate these tools into research workflows, important questions have emerged about their role as collaborators, tools, or something in between (Binz, Alaniz, et al., 2025). To help behavioral researchers harness these developments while maintaining scientific rigor, Hussain et al. (2024) developed a tutorial on open-source LLMs using the Hugging Face ecosystem.

The tutorial highlights several applications that are directly relevant to behavioral science. One application demonstrates how embeddings extracted from LLMs can capture semantic relationships among personality constructs, recover the structure of the Big Five personality factors, and predict empirical item correlations. Another application demonstrates how LLM-derived features can predict human choices in repeated decision-making tasks, such as the horizon task paradigm used to study exploration–exploitation trade-offs. Using features extracted from a 13-billion parameter Llama model, the approach achieved prediction accuracy comparable to handcrafted psychological models, without requiring domain-specific feature engineering (Figure 1). This capacity for LLMs to serve as general-purpose feature extractors for behavioral prediction is a significant methodological advance. Building on this, researchers have recently built foundational models of human cognition by finetuning LLMs on large-scale behavioral datasets (Binz, Akata, et al., 2025). Such models predict behavior across diverse experimental paradigms and even align with human neural activity.

The tutorial emphasizes open-source models because they address critical concerns about reproducibility and transparency in science. Proprietary models can change without notice, undermining the ability to replicate findings. Open-source alternatives offer full control over model versions and can be run locally, ensuring that analyses remain reproducible. These open models now achieve performance comparable to proprietary alternatives across many behavioral science applications.

Open-source LLMs hold transformative potential for behavioral science, offering researchers powerful tools for feature extraction, prediction, and even text generation while preserving the openness and reproducibility that rigorous science demands. As these models continue to improve, they promise accelerated advances across the behavioral sciences.

Key Reference

Addressing the Zoo of Psychological Measures and Constructs With Large Language Models

Psychology faces a persistent challenge in managing its conceptual landscape. The field has accumulated tens of thousands of constructs and measures, most used only infrequently, leading to fragmentation rather than cumulative knowledge (Wulff & Mata, 2026). This proliferation produces "jingle-jangle fallacies": situations in which different measures share the same label (jingle) or the same measure receives different labels (jangle). Unlike chemistry, which developed systematic methods for refining its periodic table, psychology has lacked scalable tools for conceptual engineering, leaving researchers struggling to select appropriate measures, consortia facing difficulties integrating data across inconsistent definitions, and the discipline crowded with redundant or ill-defined constructs.

Large language models (LLMs) offer a solution through semantic embeddings—high-dimensional vector representations that capture meaning based on contextual relationships. By placing psychological measures and constructs into a shared semantic space (Figure 2), researchers can apply standard analytical methods to examine conceptual overlap. This approach enables three classes of applications: mapping relations between existing measures through clustering and dimensionality reduction; clarifying taxonomies by detecting fallacies and reassigning construct labels; and generating new measures that fill conceptual gaps without duplicating existing content.

Wulff and Mata (2025) demonstrated that semantic embeddings from a finetuned language model accurately capture key psychometric properties of personality measures, including their internal consistency (r = 0.61–0.75 with Cronbach's α), structural fidelity, and patterns of convergent and divergent validity across major inventories. Using these validated embeddings, the authors developed automated methods for detecting jingle-jangle fallacies across 459 scales and 277 constructs from the International Personality Item Pool. The clustering and relabeling approach produced taxonomies with 58% fewer fallacies while requiring 75% fewer unique construct labels, demonstrating that substantially more parsimonious representations can be achieved without sacrificing conceptual granularity.

These findings establish LLMs as powerful tools for navigating psychology's conceptual zoo. However, these methods should support, not replace, expert judgment. LLMs do not offer value-neutral representations—their outputs reflect biases in training data, and construct diversity often serves legitimate theoretical purposes across research communities. Meaningful progress toward conceptual clarity will ultimately require coordinated, community-wide efforts that combine computational advances with structured deliberation among experts who must collectively decide which constructs to retain, merge, or eliminate.

Key References

From a Zoo of Measures to a Single Trait: The Structure and Assessment of Impulsivity

Impulsivity is widely invoked to explain and predict important economic and psychological behaviors across clinical and nonclinical contexts. Yet inconsistencies in its definition and measurement (notably between self-report and behavioral tasks) have led to serious doubts about the validity of impulsivity as a psychological construct and personality trait. To assess whether impulsivity is a coherent, salvageable trait, Huang et al. (2024) conducted a preregistered, large-scale study (N = 1,676) administering 48 distinct measures of impulsivity from 10 self-report scales and 10 behavioral tasks, along with self-reported frequencies of seven impulsivity-related behaviors (e.g., impulsive buying, social media use). A subsample (N = 196) completed a retest 3 months later.

Correlations were substantially higher among self-report measures than among behavioral tasks or across these two classes of measurement tools (Figure 3), replicating similar divergences observed for risk preference (Frey et al., 2017). A bifactor analysis yielded one general factor of impulsivity, I, (analogous to the general intelligence factor g) and six specific factors. Factor I loaded primarily on self-reports, showed high test—retest reliability, and predicted real-world impulsivity-related behaviors better than existing measures and composites. These findings indicate that individuals differ in trait impulsivity and that these differences are stable, measurable, and behaviorally consequential, suggesting that impulsivity is a valid psychological construct.

To operationalize factor I, the authors used machine learning algorithms to develop the Adjustable Impulsivity Scale. The scale exhibits excellent psychometric properties, retains performance in shorter versions, and predicts impulsivity-related behaviors as well as factor I. The bifactor structure helps reconcile discrepancies between self-report and task-based indices by distinguishing a broad, stable disposition from more specific, context-bound processes often captured by laboratory tasks.

Together, the results provide a coherent framework and practical tool for research and assessment, support impulsivity as a robust personality trait with predictive utility, and converge with evidence on other foundational preferences (e.g., risk; Frey et al., 2017). They encourage multi-method measurement—including self-reports—and theory building that integrates general and specific components to inform interventions across health, finance, and digital behavior.

Key References

Policy Science

Scaling the Aggregation of Distributed Knowledge for Policy and Science Communication

Tackling societal challenges—from pandemics to climate change—requires tapping into knowledge scattered across citizens, scientists, and other stakeholders. Yet standard tools force a trade-off: Surveys can reach many people quickly but squeeze answers into predetermined response categories. Qualitative methods like interviews preserve nuance but are slow and costly to scale. 

Lopez-Lopez and Herzog (2025) developed a method that uses large language models (LLMs) to surface citizens' policy perspectives at scale without sacrificing nuance. In a study with 300 demographically representative UK citizens, participants brainstormed ideas about policy challenges drawn from the United Nations Sustainable Development Goals, such as how to increase vaccination rates in rural areas (Figure 4). An LLM extracted key concepts from their open-ended responses. These concepts were embedded in a semantic vector space and used to construct semantic networks—webs of interconnected ideas that can be analyzed with network metrics like degree centrality. This revealed patterns that simple frequency counts would miss. While rural citizens mentioned "vaccination" and "vaccination education" more often, those concepts were more central to urban respondents' thinking; the reverse held for "education" and "public awareness." Such dissociations could matter for tailoring health messages. The approach complements established methods—large-scale brainstorming can precede a closed-form survey to surface questions pollsters might not think to ask about.

Holford et al. (2023) tackled a related problem, publishing a manifesto for science communication as a collective intelligence endeavor. The typical "lone wolf" model, in which individual scientists broadcast their own findings, is vulnerable to misinformation, false balance, and personal attacks on researchers. The manifesto calls instead for a collaborative, technology-supported approach that aggregates distributed evidence and expert judgments, draws on a more diverse scientific community, incorporates input from stakeholders and citizens, and responds more swiftly to fast-moving evidence. The authors practiced what they preached, using the collective intelligence platform pol.is to develop the manifesto itself.

Both efforts point in the same direction. Closing the gap between scale, nuance, and speed means building better ways to harvest what people—citizens and scientists alike—collectively know.

Key References

Improving the Tracking of Sustainable Development Goals in Research

The United Nations Sustainable Development Goals (SDGs) are a blueprint for addressing global challenges such as poverty and environmental degradation. Yet progress toward achieving the 17 goals by 2030 is off track, hampered in part by limited monitoring due to missing data. One way to address this data gap is by detecting SDGs in text, such as academic publications, policy documents, and news articles. Recognizing both the potential and the pitfalls of existing approaches, researchers developed text2sdg, an open-source R package that detects SDGs in any text source while overcoming key limitations of prior methods (Meier et al., 2025; Wulff et al., 2024).

The authors conducted the most systematic comparison of SDG labeling systems to date, evaluating seven prominent tools across diverse text sources, including research paper titles, academic excerpts, and news articles. The systems varied substantially in their sensitivity–specificity trade-offs, exhibited systematic biases that distorted the appearance of SDGs, and generated alarming rates of false positives as document length increased. For instance, some assigned SDG 3 (Good Health and Well-Being) twice the weight given by human experts, while others consistently underestimated SDGs related to innovation and inequality. Such biases can mislead policymakers about research investment priorities and undermine trust in automated monitoring efforts.

Key References