LLMSYS-HPOBench: 実運用LLMシステム向けハイパーパラメータ最適化ベンチマーク LLMSYS-HPOBench: Hyperparameter Optimization Benchmark Suite for Real-World LLM Systems

Production LLM systems are rarely just a model call. They typically combine retrieval-augmented generation, tool use, multi-step agents, and prompt scaffolding, each exposing its own configuration knobs. The paper introduces LLMSYS-HPOBench, a benchmark suite intended to evaluate hyperparameter optimization (HPO) methods specifically for these compound, real-world LLM systems rather than for isolated model training.

Existing HPO benchmarks such as HPOBench, JAHS-Bench, or NAS-Bench have largely focused on classical ML pipelines and neural architecture search. They do not capture the heterogeneous parameter spaces that arise in LLM stacks: chunk size and overlap for document splitters, embedding model choice, retriever top-k, reranker thresholds, prompt templates, decoding temperature, maximum tool-call depth, and agent iteration limits. Each evaluation can also be expensive, both in latency and in API or GPU cost, which makes naive grid search impractical.

LLMSYS-HPOBench appears to package several representative tasks with unified interfaces for configuration spaces, evaluation metrics, and cost accounting. The intent, as far as can be inferred, is to let researchers compare Bayesian optimization, evolutionary search, multi-fidelity methods, and bandit-style approaches on a level playing field, with reproducible budgets and seeds. This kind of standardization has historically accelerated progress in adjacent areas like AutoML.

The broader context is that LLM orchestration frameworks such as LangChain, LlamaIndex, DSPy, and Haystack have rapidly proliferated, each introducing dozens of tunable parameters. Tools like Optuna, Ray Tune, and SMAC have begun to be applied to these pipelines, and DSPy in particular has popularized the idea of optimizing prompts and few-shot demonstrations as part of the search. However, evaluation has remained ad hoc, with each paper rolling its own setup, making cross-study comparison difficult.

A benchmark targeting full LLM systems could help address this fragmentation. It may also surface practical findings, for example whether retrieval-side parameters dominate generation-side ones, or how sensitive end-to-end quality is to prompt wording compared with decoding settings. Such insights would be valuable for the emerging LLMOps discipline, where teams must balance quality, latency, and cost under tight production constraints.

Readers should consult the original paper for concrete task definitions, baseline results, and licensing details, as the description here is based on the stated scope and may not reflect every aspect of the released suite.