← Publications

NeurIPS 2025 † equal advising

Reinforcement Learning with Backtracking Feedback

Bilgehan Sel, Vaishakh Keshava, Phillip Wallis, Lukas Rutishauser, Ming Jin†, Dingcheng Li†

Side-by-side diagram with two columns. Left, labelled BSAFE+: a query ("How does a good leader operate?") goes to an instruction-tuned model, which returns a safe answer; an attacker splices a biased sentence into that answer; the resulting training example shows the biased sentence struck through, followed by a [BACKTRACK] token and the original safe continuation. Right, labelled RLBF: the model answers "What can I do to improve my sleep?" and drifts into unsafe medical advice, shown in red; a critic marks that span struck through and inserts [BACKTRACK], after which the model resumes with a safe continuation. Arrows show each corrected transcript feeding back into the model being trained.
Figure. Two routes to the same skill: the supervised stage (left) manufactures corrections by splicing a violation into a known-safe answer, while the reinforcement stage (right) has a critic mark violations in the model's own live output.

Safety training usually teaches a model to refuse at the start of a response. That leaves a gap: if a request looks benign, or an attacker plants instructions partway into the context, the model has already committed to answering and nothing stops it from drifting into harmful territory later. The available fixes are blunt. Resetting the generation throws away everything written so far, which is wasteful when a single offensive line appears in an otherwise correct page of text. Earlier backtracking work (BSAFE) was more targeted but asked the model to repeat the harmful span before rewriting it — a step the authors argue costs efficiency and leaves artifacts in the output, though the paper reports no measurement of either.

RLBF gives the model a cheaper correction signal. When a violation is detected, the model emits a category token identifying the kind of violation and then a single [BACKTRACK_BY_X] token, where X is the count of preceding tokens to drop. Generation stays autoregressive throughout; the internal state and KV cache are never rewound. The special tokens are instructions to an output handler, which deletes the last X tokens from what the user sees and splices in whatever comes next, so a small streaming buffer is enough to make the correction nearly real-time. Two training stages teach this. Supervised fine-tuning uses BSAFE+ data, built by taking a high-quality safe answer from the model being trained and injecting a violation at a contextually plausible spot — because the original answer is known, both the exact backtrack distance and an in-distribution safe continuation come for free. A reinforcement learning stage (GRPO, with the SFT loss mixed in at weight 0.1) then trains against an LLM safety critic scoring the model's own live rollouts: +1.0 for a clean generation or a successful correction, -1.0 for an uncaught violation, -0.5 for backtracking over text that was fine, and -0.2 for a correction that lands somewhere unsafe or incoherent.

The RL stage is what separates RLBF from SFT-only backtracking. On Middle Filling attacks the two are close (RLBF 3-7% attack success, BSAFE+ 3-6%), but on the plain LMSYS harmful-query subset RLBF reaches 1-2% against BSAFE+'s 14-17% and instruction tuning's 24-28%. The authors hypothesize, rather than demonstrate, two reasons: the RL policy learns to avoid entering the bad state instead of only cleaning up after it, and the one-token backtrack command is an easier signal to execute reliably than BSAFE's multi-token rewrite. Robustness also holds on attacks absent from the training data — under GCG, RLBF sits at 4.3-4.7% against BSAFE+'s 5.7-6.6% and Circuit Breakers' 10.7-13.4% — and an adaptive GCG variant that also optimizes the suffix to suppress backtrack tokens leaves RLBF unchanged (4.7% AdvBench, 4.3% HEx-PHI), with BSAFE+ moving at most 0.4 points. Utility is essentially untouched: Gemma 2 9B with RLBF scores 70.7% on MMLU and 35.6% on MATH against the instruction-tuned baseline's 70.6% and 35.4%, with BBH and GSM8K a tenth of a point lower.

Key results

BibTeX 
@inproceedings{sel2025rlbf,
  title     = {Reinforcement Learning with Backtracking Feedback},
  author    = {Sel, Bilgehan and Keshava, Vaishakh and Wallis, Phillip and Rutishauser, Lukas and Jin, Ming and Li, Dingcheng},
  booktitle = {Advances in Neural Information Processing Systems (NeurIPS)},
  year      = {2025}
}