Internalizing Safety Understanding in Large Reasoning Models via Verification

Safety Internal (SInternal) — official implementation
International Conference on Machine Learning (ICML) 2026

Answer-centric alignment imitates a safe answer without understanding whether or why it satisfies safety specifications. In contrast, intrinsic safety verification equips the model to reason about the real-world consequences of its own potential answer, producing principled — rather than pattern-matched — safe generations.

---

📖 Overview

Explicit Chain-of-Thought (CoT) makes Large Reasoning Models (LRMs) powerful, but it also lets them generate riskier final answers. Current alignment paradigms mostly rely on externally enforced compliance — training models to detect malicious prompts rather than to evaluate the safety of their own outputs. Our empirical analysis shows this remains largely behavioral: ostensibly aligned models lack intrinsic safety understanding, frequently fail to verify the safety of their own responses, and stay vulnerable to adversarial jailbreaks.

Safety Internal (SInternal) addresses this by internalizing safety specifications: we train LRMs exclusively on safety verification tasks — critiquing their own generated answers using expert reasoning trajectories. Learning to verify induces strong generalization to safe generation, sharply improving robustness against out-of-domain and LRM-specific jailbreaks. When combined with reinforcement learning, SInternal serves as a superior initialization compared to standard SFT, showing that internalizing safety understanding is a more robust foundation for alignment than mimicking safe behaviors.

TL;DR — Teach a reasoning model to verify whether its own answer is safe, and it learns to generate safe answers that generalize to unseen attacks — while preserving its reasoning ability and avoiding over-refusal.

The SInternal framework. (1) SInternal (SFT on verification): the policy model samples candidate responses, an expert model (Claude-4-Sonnet) critiques them against a safety specification 𝒮, and the policy is fine-tuned to reproduce the verification/critique. (2) Follow-up RLVR: the safety-internalized model is further optimized on-policy with a verifiable safety reward.

---

🔑 Key Contributions

• Verification → Generation transfer. Training a model only to verify the safety of its own answers transfers to substantially safer generation, especially on out-of-domain jailbreaks.
• Self-generated verification matters. Verifying a model's own candidate answers (rather than external responses) internalizes safety boundaries aligned with its actual output distribution.
• A better RL initialization. Starting RLVR from SInternal yields consistently safer post-RL ceilings than starting from base or SFT-based methods — and is the only method to defend against the HCoT reasoning-hijack attack.
• Utility preserved. SInternal maintains reasoning ability (MATH / AIME) and avoids over-refusal (XSTest).
• Data efficient. SInternal matches the SFT baseline's safety using only ~50% of the training data.

---

📊 Results

We evaluate on 9 benchmarks across three domains — safety (Harmful: StrongREJECT; Jailbreak: WildJailbreak, Fortress, JailBreak-R1; LRM-Jailbreak: HCoT, Trotter), over-refusal (XSTest), and general reasoning (MATH, AIME2024) — on three LRMs: DeepSeek-R1-Distill Qwen-7B, Llama-8B, and Qwen-14B.

Metrics: ASR = Attack Success Rate (↓ lower is safer, via Llama-Guard-3-8B); CR = Compliance Rate on XSTest (↑ higher is better); MATH pass@1, AIME pass@16 (↑).

Table 1 — SInternal vs. SFT-based baselines

Model	StrongReject ↓	Fortress ↓	WildJailbreak ↓	JailBreak-R1 ↓	HCoT ↓	Trotter ↓	XSTest ↑	Math ↑	AIME ↑
DeepSeek-R1-Distill-Qwen-7B
Base	50.2	39.0	56.0	54.0	100.0	23.7	90.8	89.2	80.0
SafeChain	48.9	46.6	44.0	46.7	100.0	29.8	98.0	82.4	83.3
STAR-1	2.9	33.2	30.0	15.3	98.0	32.3	77.2	88.8	83.3
SInternal	7.0	22.6	21.6	11.8	92.0	19.7	89.6	88.4	83.3
DeepSeek-R1-Distill-Llama-8B
Base	44.7	51.2	48.4	49.5	100.0	37.9	96.0	85.2	83.3
SafeChain	28.8	46.6	42.8	39.0	100.0	29.8	99.6	74.2	66.7
STAR-1	0.0	26.2	18.4	9.3	100.0	34.3	86.8	81.8	76.7
SInternal	1.6	18.6	13.6	6.1	94.0	26.3	92.0	84.4	80.0
DeepSeek-R1-Distill-Qwen-14B
Base	41.2	52.6	44.4	41.5	100.0	54.0	95.6	88.2	86.7
SafeChain	24.9	48.2	45.2	32.9	100.0	44.4	99.6	85.2	83.3
STAR-1	0.6	28.2	18.4	5.1	100.0	49.0	94.0	89.6	83.3
SInternal	0.6	19.2	6.8	1.3	90.0	33.3	98.0	89.6	86.7

Takeaway. By internalizing safety understanding rather than memorizing refusal patterns, SInternal consistently achieves the lowest ASR on OOD Jailbreak and LRM-specific Jailbreak benchmarks — even where it is intentionally sub-optimal on in-domain harmful prompts — while preserving reasoning (Math/AIME) and avoiding over-refusal (XSTest).

Table 2 — RLVR alignment (+ GRPO) on top of each method

Model	StrongReject ↓	Fortress ↓	WildJailbreak ↓	JailBreak-R1 ↓	HCoT ↓	Trotter ↓	XSTest ↑	Math ↑	AIME ↑
DeepSeek-R1-Distill-Qwen-7B
Base (+ GRPO)	0.0	21.4	12.4	8.3	96.0	23.7	93.6	88.8	80.0
SafeChain (+ GRPO)	1.0	25.2	16.8	8.6	96.0	27.3	94.4	86.6	76.7
STAR-1 (+ GRPO)	1.0	17.0	8.8	3.5	98.0	26.3	89.6	89.2	80.0
SInternal (+ GRPO)	0.3	11.0	3.2	2.2	64.0	23.7	94.3	89.8	83.3
DeepSeek-R1-Distill-Llama-8B
Base (+ GRPO)	3.2	14.8	6.0	2.9	94.0	14.4	98.8	81.8	66.7
SafeChain (+ GRPO)	0.3	14.4	8.0	1.6	88.0	18.7	99.6	80.0	56.7
STAR-1 (+ GRPO)	0.0	4.8	0.8	0.0	78.0	21.7	93.6	83.8	60.0
SInternal (+ GRPO)	0.0	2.8	1.2	0.0	38.0	11.1	98.4	84.2	60.0
DeepSeek-R1-Distill-Qwen-14B
Base (+ GRPO)	0.0	12.2	4.4	0.0	96.0	30.8	99.2	89.8	80.0
SafeChain (+ GRPO)	0.0	8.0	2.8	0.0	92.0	23.7	96.4	89.8	80.0
STAR-1 (+ GRPO)	0.0	7.8	3.6	0.6	98.0	28.8	96.0	89.4	80.0
SInternal (+ GRPO)	0.0	5.2	0.4	0.3	62.0	21.7	99.2	89.6	80.0

Takeaway. Starting RLVR from SInternal yields the safest post-RL ceilings. Notably, SInternal (+ GRPO) is the only configuration that meaningfully defends against HCoT (e.g., 96.0 → 64.0 on Qwen-7B, 94.0 → 38.0 on Llama-8B), the state-of-the-art attack that hijacks LRMs with compliant CoT.

---

🔬 Analysis: Why Verification Transfers to Generation

Verification ability (Table 3)

SInternal is the only method to consistently and substantially improve a model's ability to verify the safety of base-model answers (F1 ↑, Claude-4-Sonnet as guardrail). Answer-centric alignment (STAR-1, GRPO) does not reliably help — and can even hurt.

Model	Random	Base	STAR-1	GRPO	SInternal
DS-7B	62.4	48.4	32.2	54.7	85.9
DS-8B	63.8	71.6	59.3	64.7	89.0
DS-14B	60.4	71.1	53.9	73.4	89.3

Generation safety (ASR, bars ↓) plotted against verification ability (F1, line) across training stages. Improving generation safety does not automatically improve verification — motivating a method that trains verification directly.

More proactive verification (Table 4)

Under attack (WildJailbreak), SInternal triggers intrinsic self-verification far more often, and that verification almost always leads to a safe answer.

Method	Verification Trigger Rate	Conditional Safety (Verify → Safe)	Overall Safety Rate
Base	16.0%	100.0%	30.4%
STAR-1	28.4%	95.8%	59.6%
SInternal	50.4%	99.2%	78.8%

Deeper alignment (KL divergence)

Per-token KL divergence from the base LRM on harmful trajectories. SInternal and SInternal (GRPO) maintain consistently higher KL across token positions — evidence of sustained awareness of harmful content rather than shallow, prefix-only alignment.

---

🧪 Ablation Highlights

• Self- vs. external verification (Table 5). Training on the model's own verification trajectories consistently yields lower ASR than swapping in another model's trajectories (e.g., DS-8B on StrongReject: 1.6 self vs. 3.8 other; WildJailbreak: 13.6 vs. 19.6).
• Critique vs. judgment (Table 6). The critique component drives generalization to unseen jailbreaks — removing it spikes Fortress ASR from 19.2 → 46.8 and WildJailbreak from 6.8 → 22.4; removing the judgment mainly hurts in-domain safety.
• Specification robustness (Table 7). Swapping the bullet-point spec for Llama Guard 4's 14-category taxonomy (S14) preserves both defense and low over-refusal on DS-14B (Fortress 19.2 → 21.0, WildJailbreak 6.8 → 6.8, XSTest 97.2 → 97.6).
• Data efficiency. SInternal leads the SFT baseline at every data scale and matches it using only ~50% of the training data.

---

🚀 Getting Started

Installation

git clone https://github.com/zy20031230/SafetyInternal.git
cd SafetyInternal
pip install -r requirements.txt

The repository bundles two training frameworks and an evaluation suite:

Component	Framework	Purpose
LLaMA-Factory/	LLaMA-Factory	SInternal SFT — training LRMs on verification trajectories
verl/recipe/SInternal/	verl	RLVR (GRPO) — reinforcement learning with a verifiable safety reward
benchmark/	—	Safety, over-refusal, and reasoning evaluation
safety_data/, math_data/	—	Seed prompts (WildJailbreak / STAR-1) and DAPO math data

1. SInternal SFT (verification training)

cd LLaMA-Factory/new_scripts
# Edit BASE_MODEL and OUTPUT_BASE inside the script first
bash qwen7B.sh      # or llama8B.sh / qwen14B.sh

2. RLVR alignment (+ GRPO)

cd verl/recipe/SInternal/scripts
bash init_ray.sh              # start the Ray cluster
bash launch_guard_model.sh    # serve the safety reward model (Qwen3-Guard-gen)

# Set MODEL_PATH (SFT checkpoint), MATH_DIR, SAFETY_DIR inside the script first
bash Qwen-7B.sh               # or Llama-8B.sh / Qwen-14B.sh

• Reward-model settings: verl/recipe/SInternal/config/rm_config.yaml
• Reward function: verl/recipe/SInternal/reward_fn.py

3. Evaluation

# Register your model in benchmark/config.py, then:
cd benchmark
bash easy_eval.sh

Individual stages live in benchmark/safe_benchmark/: gen.sh (generate), eval.sh (score ASR), overrefusal.sh (XSTest compliance).

---

🧾 Experimental Setup (summary)

• Models: DeepSeek-R1-Distill-Qwen-7B, -Llama-8B, -Qwen-14B (analysis focuses on DS-14B).
• SInternal data: ~1k harmful + 1k benign prompts from WildJailbreak plus the full STAR-1 set; 8 candidates sampled per prompt, verified by Claude-4-Sonnet, yielding ~6,000 verification examples (contrastive safe/unsafe pairs for harmful prompts).
• RLVR data: the safety prompts above + 3k DAPO math problems to preserve reasoning; Qwen3-Guard-gen provides safety/refusal reward signals.
• Evaluators: Llama-Guard-3-8B (ASR), GPT-4o (XSTest CR), MATH pass@1, AIME2024 pass@16.

---

📚 Citation

If you find this work useful, please cite:

@inproceedings{zhang2026sinternal,
  title     = {Internalizing Safety Understanding in Large Reasoning Models via Verification},
  author    = {Zhang, Yi and Chen, Yuxin and Sheng, Leheng and Zhang, Dongcheng and
               Lu, Chaochao and Wang, Xiang and Zhang, An},
  booktitle = {Proceedings of the 43rd International Conference on Machine Learning (ICML)},
  year      = {2026},
  note      = {arXiv:2605.08930},
  url       = {https://arxiv.org/abs/2605.08930}
}

---

🙏 Acknowledgements

This project builds on LLaMA-Factory for supervised fine-tuning and verl for reinforcement learning. We thank the authors of WildJailbreak, STAR-1, Fortress, HCoT, XSTest, and DAPO for their datasets and benchmarks.