Can large language models help predict results from a complex behavioural science study?
Abstract
We tested whether large language models (LLMs) can help predict results from a complex behavioural science experiment. In study 1, we investigated the performance of the widely used LLMs GPT-3.5 and GPT-4 in forecasting the empirical findings of a large-scale experimental study of emotions, gender, and social perceptions. We found that GPT-4, but not GPT-3.5, matched the performance of a cohort of 119 human experts, with correlations of 0.89 (GPT-4), 0.07 (GPT-3.5) and 0.87 (human experts) between aggregated forecasts and realized effect sizes. In study 2, providing participants from a university subject pool the opportunity to query a GPT-4 powered chatbot significantly increased the accuracy of their forecasts. Results indicate promise for artificial intelligence (AI) to help anticipate—at scale and minimal cost—which claims about human behaviour will find empirical support and which ones will not. Our discussion focuses on avenues for human–AI collaboration in science.
Add the publication’s full text or supplementary notes here. You can use rich formatting such as including code, math, and images.
Francisco Cruz
Invited Assistant Professor
Francisco Cruz is an invited assistant professor in psychology, statistics, and methods at the Faculdade de Psicologia, Universidade de Lisboa, and Faculdade de Ciências da Saúde, Universidade Europeia. Junior Consulting Editor at the Journal of European Social Psychology, 2025-present. Social Psychology Ph.D. on lay beliefs about science, supervised by Prof. André Mata (Universidade de Lisboa) and Prof. Tania Lombrozo (Princeton University), 2022-2025. Visiting Student Research Collaborator at Princeton University, 2023-2024. Society for General Psychology and Interdisciplinary Inquiry, Fulbright Portugal, and Fundação para a Ciência e Tecnologia awardee. His research interests include lay beliefs about science (i.e., what people believe that science can or cannot explain and why), motivated beliefs in science (i.e., the contexts in which people are more prone to accepting scientific explanations), representation of social groups (i.e., how people integrate information to provide judgments on shared homogeneity vs. heterogeneity across group members), epistemic trespassing (i.e., when people provide judgments on domains beyond those in which they are experts), intuitive mind-body dualism (i.e., a natural tendency to see the world as split in material and immaterial portions), and face perception (i.e., features driving the advantage in recall for own- vs. other-race faces).