Communications AI & Computing Computer science

Quantifying the human visual exposome with vision language models

Christian Rominger, Andreas R. Schwerdtfeger, Malay Gaherwar Singh, Dimitri Khudyakow, Elizabeth A. M. Michels, Fabian Wolf, Jakob Nikolas Kather, Magdalena Katharina Wekenborg
Springer Nature · 2026-11-05 · DOI 10.1038/s44488-026-00023-x

Abstract

Abstract The visual environment is a fundamental yet unquantified determinant of mental health. While the concept of the environmental exposome is well-established, current methods rely on coarse geospatial proxies or biased self-reports, failing to capture the first-person visual context of daily life. Here we show that coupling ecological momentary assessment with vision-language models can quantify the semantic richness of human visual experience. Across 2,674 participant-generated photographs, model-derived estimates of “greenness” robustly predicted momentary affect and chronic stress, consistent with established benchmarks. We then developed a semi-autonomous large language model-based pipeline that mined over seven million scientific publications to extract nearly 1,000 environmental features empirically linked to mental health. When applied to real-world imagery, up to 33% of model-extracted context ratings significantly correlated with affect and stress.

Read at publisher →