マージすべきモデルを間違えていないか？LLMのモデルマージにおける専門家の訓練時間の影響Are we Merging the Right Models? Impact of Expert Training Duration on Model Merging for LLMs

Model merging has quietly become one of the most practical techniques for combining the strengths of several specialized language models into a single set of weights, and a new study posted to arXiv argues that practitioners may be overlooking a decisive variable: how long each expert model was trained before it was folded into the mix. The paper investigates how the training duration of expert models shapes the quality of the merged result, and it concludes that selecting checkpoints from an appropriate training stage is critical for achieving strong post-merge performance.

Model merging refers to the practice of taking two or more models that share the same base architecture and combining their parameters directly, without further gradient training on the combined objective. The appeal is straightforward. Rather than assembling a large multi-task dataset and running an expensive joint fine-tuning job, teams can fine-tune separate experts on separate tasks and then average or otherwise blend their weights. The approach requires no access to the original training data at merge time, costs little more than the arithmetic itself, and can produce a single model that inherits capabilities from each contributor. Popular methods range from simple weight averaging and task arithmetic to more elaborate schemes such as TIES-merging, DARE, and spherical interpolation, many of which are packaged in widely used open-source tooling like mergekit.

The central finding of this work is that the number of training steps an expert has undergone is not a neutral detail but a factor that materially affects how well that expert merges with others. According to the summary, models taken too early or too late in their fine-tuning trajectory can degrade the combined model, while models drawn from a suitable stage combine far more favorably. In other words, two experts that each perform well in isolation may still merge poorly if their training durations are mismatched or if they have drifted too far from the shared starting point.

This connects to a broader body of research on why merging works at all. Weight averaging tends to succeed when the models being combined remain in a compatible region of the loss landscape relative to their common initialization, a property often discussed under the heading of linear mode connectivity. As fine-tuning continues, a model's weights move progressively further from the base, and its internal representations can specialize in ways that conflict with those of a sibling model. The paper's emphasis on training duration appears consistent with this intuition: heavily trained experts may individually reach higher task accuracy yet become harder to reconcile during merging, whereas lightly trained experts may not have acquired enough task-specific capability to contribute. The result is likely a trade-off in which an intermediate checkpoint offers the best balance between specialization and mergeability.

The practical implication is that model selection deserves as much attention as the merging algorithm itself. Much of the existing discussion around merging has focused on which mathematical recipe to use, how to weight each contributor, or how to resolve sign conflicts between parameters. This study suggests that even with a fixed merging method, choosing the right checkpoint for each expert can swing the outcome substantially. That reframes a common assumption that a fully converged, best-performing expert is always the best candidate to merge. For teams that routinely save intermediate checkpoints during fine-tuning, this points toward treating training duration as a tunable hyperparameter of the merge, rather than defaulting to the final checkpoint.

The work sits alongside a growing industry interest in modular approaches to model building, including mixtures of experts, parameter-efficient adapters such as LoRA, and the assembly of model collections from public checkpoint repositories. As organizations increasingly reuse and recombine open-weight models rather than training from scratch, understanding the conditions under which combination succeeds becomes more valuable. Readers should note that findings of this kind can depend on the specific tasks, base models, and merging methods evaluated, and the degree to which the guidance generalizes across architectures and domains will need confirmation from further study. Still, the paper offers a useful and actionable reminder that in model merging, the question is not only how to merge, but which version of each model to merge in the first place.