LongDS-Bench：長期的なエージェント型データ分析が失敗する理由を検証 LongDS-Bench: On the Failure of Long-Horizon Agentic Data Analysis

Real-world data analysis is rarely a straight line. Analysts form hypotheses, write code, interpret outputs, and revise their approach — often many times over before reaching a conclusion. Yet most benchmarks used to evaluate AI agents still focus on isolated, short-horizon tasks that fail to capture this iterative reality. LongDS-Bench, introduced in arXiv:2605.30434, is a direct response to that gap.

The paper argues that existing evaluation frameworks systematically overestimate agent capability by testing only contained, well-scoped subtasks. When agents are asked to conduct genuine long-horizon data analysis — spanning exploration, cleaning, modeling, and interpretation across many steps — their performance degrades in ways that short benchmarks simply cannot detect. LongDS-Bench is designed to surface exactly these failure modes.

Among the key findings, the authors identify patterns of compounding errors: mistakes made early in an analysis pipeline propagate and amplify, leading agents to pursue increasingly wrong directions without self-correction. Misinterpretation of intermediate results is another documented failure mode, where an agent might confidently proceed based on a flawed reading of a preliminary output. These are not exotic edge cases — they reflect what happens routinely when AI systems operate in realistic, open-ended analytical workflows.

The timing of this work is notable. Agentic data analysis has become a hotly contested product category. OpenAI's Code Interpreter (now part of ChatGPT's Advanced Data Analysis), Anthropic's tool-use capabilities, and Microsoft's Fabric Copilot all promise to automate complex data workflows. But the evaluations underpinning claims about these systems often rely on benchmarks that don't stress-test multi-step reasoning over extended sessions. LongDS-Bench offers a more rigorous measuring stick.

In the broader benchmark landscape, efforts like DS-1000 and DABench have made meaningful contributions to data science evaluation, but they have been criticized for limited task length and insufficient interactivity. LongDS-Bench positions itself as a next-generation alternative, incorporating longer task horizons and more realistic analytical scenarios.

For the research community, benchmarks shape incentives. If models are optimized for short-task performance, that is what they will excel at — even if it leaves them brittle in real deployments. By reframing evaluation around the kind of iterative, multi-step work that human analysts actually do, LongDS-Bench may help redirect model development toward more practically useful capabilities. Whether it gains traction as a standard evaluation will depend on adoption, but the problem it identifies is hard to dismiss.