2,197 robot manipulation attempts from fourteen independently collected sources, scoring thirteen failure detectors under one protocol.
Vision-language models (VLMs) are increasingly used to determine whether a robot manipulation attempt has succeeded. These judgments can serve as reinforcement-learning rewards, training-data filters, policy-ranking signals, or triggers for retry, making reliable failure detection critical for robot learning and evaluation. However, existing benchmarks provide limited evidence of cross-domain generalization.
We introduce FailBench, a benchmark for robot failure detection comprising 2,197 manipulation attempts from 14 public sources, including 12 real-world and 2 simulated datasets, using the original outcome labels provided by their respective sources. Six of the real-world sources were originally collected for policy evaluation, reward modeling, or general data collection rather than failure detection, and 75% of failures occur naturally rather than being deliberately constructed. Evaluating 13 VLM-based detectors, we find that the best model achieves only 0.77 mean balanced accuracy, indicating substantial room for improvement. Models fine-tuned specifically for robot failure detection consistently underperform general-purpose VLMs, with the exception of the smallest general-purpose model, and also underperform their corresponding pretrained models. We further find that performance depends more strongly on the visual evidence required to determine the outcome than on the robot or task domain: detection approaches saturation when success is determined by observable object motion, but approaches chance when success depends on establishing contact, with no model exceeding 0.60 balanced accuracy on contact-intensive assembly tasks.
Analysis of model reasoning reveals a systematic bias toward predicting success under ambiguous evidence. Increasing reasoning effort does not alleviate this bias, as incorrect predictions tend to receive longer reasoning traces. Model-level interventions provide little improvement, while an input-level intervention is effective: spatially localizing the outcome-relevant region and cropping the input improves the strongest detector by 2.4 percentage points without additional training. We release FailBench and the accompanying evaluation harness to facilitate reproducible evaluation of robot failure detection.
The best of thirteen detectors reaches 0.77 mean balanced accuracy — roughly one wrong judgment in four on a task these systems are already trusted to automate.
Every detector fine-tuned for failure detection scores below each general-purpose model but the smallest, and all four that allow a direct comparison score below their own base model.
No model exceeds 0.60 on contact-rich assembly. Detectors are reliable when failure is visible as object motion, and near chance when it depends on physical contact.
Evidence localization — cropping the video to the region that decides the outcome — lifts the strongest detector by 2.4 points with no retraining (p = 0.0015).
Mean IoU of error sets across all 78 detector pairs. The models fail on different samples, so the gap is not one shared blind spot.
Real and simulated halves rank detectors almost identically, but simulation flattens the top: the leader's margin shrinks from 8 points to 2.5.
Balanced accuracy, so chance is 0.50 regardless of the split. Macro averages the thirteen two-class subsets equally; micro pools every answered sample into a single balanced accuracy. Each model runs with its own intended prompt and inference settings.
| Model | Macro | Micro | Real | Sim |
|---|---|---|---|---|
| 1Gemini 3 Flash | 0.77 | 0.74 | 0.76 | 0.77 |
| 2Gemma-4-31B-it | 0.75 | 0.76 | 0.75 | 0.76 |
| 3Qwen3-VL-8B-Thinking | 0.69 | 0.69 | 0.68 | 0.75 |
| 4GPT-4o | 0.69 | 0.67 | 0.69 | 0.68 |
| 5Qwen3-VL-4B-Thinking | 0.65 | 0.65 | 0.64 | 0.71 |
| 6Guardian (thinking)specialist | 0.63 | 0.61 | 0.62 | 0.64 |
| 7RoboReward-8Bspecialist | 0.62 | 0.59 | 0.62 | 0.61 |
| 8Hy-Embodied-VLM-1.0embodied | 0.61 | 0.62 | 0.61 | 0.64 |
| 9ViFailback-8Bspecialist | 0.59 | 0.56 | 0.60 | 0.54 |
| 10HY-Embodied-0.5embodied | 0.54 | 0.53 | 0.54 | 0.54 |
| 11Qwen3-VL-2B-Thinking | 0.53 | 0.53 | 0.52 | 0.55 |
| 12RoboFAC-7Bspecialist | 0.51 | 0.52 | 0.51 | 0.51 |
| 13FailSense-Calvin-3Bspecialist | 0.50 | 0.50 | 0.50 | 0.53 |
Unlabelled models are general-purpose VLMs with no robotics training. embodied models are trained for embodied robot work but not for failure detection; specialists are purpose-built failure detectors.
Each purpose-built detector, scored against the general-purpose model it was fine-tuned from, under the same prompt, decoding and input recipe. Base and specialist differ only in the fine-tuning, so the harness, the prompt and the model family do not explain the gap.
| Specialist | Base model | Specialist | Base | Δ |
|---|---|---|---|---|
| Guardian (thinking) | InternVL3-8B | 0.627 | 0.629 | −0.002 |
| RoboReward-8B | Qwen3-VL-8B-Instruct | 0.619 | 0.675 | −0.057 |
| ViFailback-8B | Qwen3-VL-8B-Instruct | 0.587 | 0.675 | −0.088 |
| RoboFAC-7B | Qwen2.5-VL-7B-Instruct | 0.514 | 0.555 | −0.040 |
Published specialist numbers come from test sets that are 80 to 90 percent failures, where always answering “failed” already scores what they report.
The diagnosis is that detectors do not read physical state off the pixels they are given: whether a peg is seated or the jaws are closed occupies a few dozen pixels of a shot framed to capture the whole workspace.
So a localizer sees four frames and the task sentence, and returns one rectangle around the region that settles whether the attempt worked. It is never given the outcome. The detector then runs exactly as it does everywhere else — same prompt, same frame budget, same decoding. Only the field of view changes.
223 samples fixed, 160 broken. Largest gains where the workspace is wide and the decisive object is small.
Fourteen sources screened from roughly 30 candidate corpora: twelve real, two simulated. Six come from existing failure benchmarks; the remaining six are drawn from datasets originally collected for policy evaluation, reward-model evaluation, or general robot learning. Three quarters of the benchmark are organic failures — ones that happened on their own during the attempt, not staged or synthesized.
1,176 failures and 1,021 successes. Every sample is one robot attempt: an instruction, the visual input the model is shown, and one binary label. Sources span tabletop pick, place and push; kitchen and household chores; contact-rich assembly on a task board; insertions, cable clipping and tool handling on low-cost arms; and bin-to-bin picking, across UR5, WidowX, Franka, SO-100 and SO-101 arms.