IBMとUC Berkeley、IT-BenchとMASTで企業向けAIエージェントの失敗要因を診断 IBM and UC Berkeley Diagnose Why Enterprise Agents Fail Using IT-Bench and MAST

IBM Research and UC Berkeley have released a joint study aimed at systematically diagnosing why enterprise AI agents tend to fall short in production-like environments. As expectations for agentic AI continue to rise across the industry, real-world success rates on substantive IT tasks remain disappointingly low, and the work tries to explain the gap in structural terms rather than anecdotally.

The centerpiece is IT-Bench, a benchmark suite covering three concrete enterprise domains: Site Reliability Engineering (SRE), Chief Information Security Officer (CISO) workflows, and FinOps cloud cost optimization. Tasks include diagnosing failures in live Kubernetes clusters, evaluating compliance posture, and reasoning about cost anomalies. Crucially, IT-Bench evaluates agents end-to-end against realistic infrastructure rather than toy sandboxes. Reported success rates vary by domain but consistently sit well below what human experts would achieve, underscoring how far current agent stacks are from autonomous IT operations.

To decompose those failures, the researchers leverage MAST (Multi-Agent System Failure Taxonomy), developed at UC Berkeley. MAST categorizes breakdowns in multi-agent collaboration into buckets such as specification misunderstanding, inter-agent miscommunication, missing verification, and premature termination. Rather than collapsing every run into a binary pass/fail, applying MAST to IT-Bench traces makes it possible to distinguish whether an agent failed because the underlying model lacked capability, because tool calls were poorly designed, or because the orchestration layer dropped context between steps.

This combination is notable because most existing agent benchmarks, including SWE-bench, τ-bench, and the various Computer Use evaluations from Anthropic and OpenAI, tend to focus on single-task outcomes or relatively narrow domains. Enterprise IT, by contrast, is characterized by long-horizon, multi-step workflows that often span several specialized agents and external systems. By pairing a domain-grounded benchmark with a structured failure taxonomy, IBM and Berkeley provide a complementary lens that may help frameworks such as LangGraph, CrewAI, or AutoGen identify where their abstractions actually break.

There is also a clear strategic dimension. IBM is heavily invested in agentic offerings around watsonx and its consulting practice, and a credible diagnostic methodology gives the company both a research narrative and a tool to differentiate enterprise-grade deployments from generic chatbot wrappers. It is reasonable to assume, though not explicitly stated, that internal product teams will use IT-Bench and MAST to harden their own agents before customer rollout.

For the broader community, the practical value lies in shared vocabulary. Agent development today suffers from inconsistent reporting: every new framework claims improvements, but failure modes are described in idiosyncratic ways. A common taxonomy like MAST, anchored to a reproducible benchmark like IT-Bench, makes it easier to compare interventions, whether those are better planning prompts, stronger tool schemas, or fundamentally different model backbones. If adopted widely, it could push agent research toward the kind of cumulative, error-driven progress that benchmarks like GLUE or MMLU once catalyzed for language models, although the complexity of multi-agent systems means progress is likely to be slower and noisier.