意味埋め込みを社会指標へ:プロキシ前提の妥当性検証 The Proxy Presumption: From Semantic Embeddings to Valid Social Measures

Semantic embeddings extracted from large language models are increasingly treated as ready-made measurements of social constructs such as ideology, public opinion, cultural values, or consumer preferences. This paper interrogates that practice, which the authors label the 'proxy presumption', and asks whether the resulting numbers actually measure what researchers claim they measure.

The core argument is methodological rather than technical. Distances and clusters in embedding space superficially resemble the continuous scales produced by survey instruments, but their construct validity is rarely examined. Classical psychometric checks such as content validity, convergent and discriminant validity, and test-retest reliability have well-developed analogues in the social sciences, yet embedding-based measures are often validated only by reporting a correlation with a single external benchmark. The authors argue this is insufficient evidence that an embedding captures the latent construct of interest rather than some correlated artefact of the training corpus.

The paper appears to catalogue typical failure modes for embedding-as-measurement workflows. These likely include sensitivity to prompt wording and preprocessing, instability across model versions, contamination by cultural and demographic biases baked into web-scale training data, and the inferential leap from predictive accuracy to construct validity. A model that predicts survey responses well may still be exploiting stylistic cues rather than the underlying attitude, which has serious implications when the measure is used downstream in causal or descriptive analyses.

The broader research context is worth noting. Political scientists have used word2vec and BERT to scale legislator speech, marketers exploit embeddings to map brand positioning, and a growing body of work in computational social science treats OpenAI or open-source embeddings as off-the-shelf instruments. Parallel critiques, from Bender and colleagues on stochastic parrots to bias diagnostics such as WEAT and SEAT, have flagged that embeddings encode social stereotypes and historical asymmetries. The present paper can be read as situating those concerns within a measurement-theoretic frame and translating them into actionable validation steps for applied researchers.

Practical implications extend beyond academic methodology. If embeddings are to feed into policy evaluation, market research, or content moderation tooling, then version drift becomes a measurement problem: an upgrade from one embedding model to another can silently change the scale, breaking comparability across studies or across time. Reproducibility is similarly fragile when proprietary endpoints are involved, since the underlying model may be retrained without notice. The authors seem to advocate for explicit validation protocols, transparent reporting of model provenance, and triangulation with traditional instruments rather than wholesale replacement.

None of this implies that embeddings are unusable as social measures. The more constructive reading is that they should be treated like any other instrument: calibrated, validated against multiple criteria, and accompanied by an honest account of what they do and do not capture. As LLM-derived indicators proliferate in empirical research, frameworks of the kind proposed here may become a prerequisite for credible inference rather than an optional methodological refinement.