StressEval: Failure-Driven Dynamic Benchmarking for Knowledge-Intensive Reasoning in Large Language Models

Abstract

Static benchmarks for LLMs are increasingly compromised by contamination and overfitting, especially on knowledge-intensive reasoning tasks. While recent dynamic benchmarks can alleviate staleness, they often increase difficulty at the expense of answerability and controllability. In this paper, we propose StressEval, a failure-driven data synthesis framework that turns observed model failures into dynamic, challenging, and controlable test instances. StressEval consists of three stages: (i) it constructs a semi-structured difficulty card that identifies the failed reasoning step and its root cause; (ii) it applies a dual-perspective instance-synthesis method that targets both knowledge gaps and reasoning breakdowns while preserving the underlying difficulty factors; and (iii) it applies a gating mechanism to retain only grounded, unambiguous instances. Seeding from multiple knowledge-intensive reasoning datasets, we employ StressEval to build Dynamic-OneEval, a focused suite of challenging dynamic benchmark. Across several state-of-the-art LLMs, Dynamic-OneEval yields substantially larger performance drops than the original benchmarks while retaining explicit difficulty factors, enabling more actionable iteration.