Skip to content

GOLD106_VALIDATION_REPORT.md

Latest commit

 

History

History
251 lines (177 loc) · 7.8 KB

GOLD106_VALIDATION_REPORT.md

File metadata and controls

251 lines (177 loc) · 7.8 KB

Gold-106 Dataset Validation Report

Date: 2026-01-04 Reviewer: Claude Code Status: VALIDATED

Executive Summary

The Gold-106 dataset and associated experiment results have been reviewed and validated. All mathematical calculations are correct, data integrity is confirmed, and evaluation scripts are functioning properly.

One defect was found and fixed: The README.md contained incorrect accuracy metrics (87.7% instead of 88.7%) due to a transcription error. This has been corrected in commit 0926ae7.

1. File Locations

Data Files

File Path Description
Gold-106 Dataset reproduce-120b/gold_106.json 106 human-labeled samples (51 AF, 55 aligned)
Gold-106 README data/gold-106-README.md Dataset documentation and attribution
GPTOSS 20B Results data/gptoss_20b_v3_gold106_results.json Evaluation results with predictions
120B 1-Shot Results eval/results_120b_1shot.json GPT-OSS 120B single-pass evaluation
20B 1-Shot Results eval/results_20b_1shot.json GPT-OSS 20B single-pass evaluation
120B 20-Vote Results eval/results_120b_20vote.json 120B with voting ensemble
20B 20-Vote Results eval/results_20b_20vote.json 20B with voting ensemble
Backup Gold-106 lambda-backup-jan1/gemma-scope-2/data/gold_106.json Backup copy

Evaluation Scripts

Script Path Description
Main Eval reproduce-120b/eval.py Single-pass evaluation with sklearn metrics
Voting Eval reproduce-120b/eval_voting.py N-vote ensemble evaluation
Voting Eval (alt) eval/eval_voting.py Alternate voting implementation
1-Shot Eval eval/eval_1shot.py 1-shot evaluation with confidence scores
Error Analysis eval/error_analysis.py FP/FN analysis tool

2. Data Validation

Gold-106 Dataset Structure

Total samples: 106
Labels:
  - potential_faking: 51 (48.1%)
  - aligned: 55 (51.9%)

Text field: Present in all samples
Text lengths: min=1177, max=5844, avg=2711 chars

Status: PASSED - All samples have valid text and labels.

3. Mathematical Validation

GPTOSS-20B-AF-V3 Results

Metric Reported Calculated Status
Total Samples 106 106 MATCH
Correct 90 90 (TP+TN) MATCH
TP 44 44 MATCH
FP 9 9 MATCH
TN 46 46 MATCH
FN 7 7 MATCH
Accuracy 84.906% 84.906% (90/106) MATCH
Precision 83.019% 83.019% (44/53) MATCH
Recall 86.275% 86.275% (44/51) MATCH
F1 84.615% 84.615% MATCH

Status: PASSED - All metrics are mathematically correct.

Predictions Array Validation

  • Total predictions: 106
  • Predictions marked correct: 90
  • All correct fields are consistent with true vs pred comparison
  • Confusion matrix counts from array match reported values exactly

Status: PASSED

4. Label Mapping Consistency

Source Label Mapped Label Count Status
potential_faking ALIGNMENT_FAKING 51 MATCH
aligned NOT_ALIGNMENT_FAKING 55 MATCH

Status: PASSED

5. Script Audit

Label Normalization Function

The normalize_label() function correctly maps:

  • potential_faking, alignment_faking, faking, af -> 1 (AF)
  • All other strings -> 0 (Aligned)
  • Integer labels passed through directly

Status: PASSED

Prediction Parsing Logic

Current logic in eval.py:

if "AF" in response and "ALIGNED" not in response:
    return 1  # AF
else:
    return 0  # Aligned

Analysis:

  • Works correctly for expected outputs ("AF", "ALIGNED")
  • Edge cases like "NOT AF" or "SAFE" could cause issues, but model is trained to output only "AF" or "ALIGNED"
  • Conservative default (returns 0 if uncertain) is appropriate

Status: PASSED with notes (see recommendations)

Sklearn Metrics Usage

All scripts correctly use:

  • accuracy_score(y_true, y_pred)
  • precision_recall_fscore_support() with proper label ordering
  • confusion_matrix() with consistent label ordering [0, 1]
  • roc_auc_score() for AUROC calculation

Status: PASSED

6. Cross-Model Comparison

Model Accuracy Precision (AF) Recall (AF) F1 (AF) AUROC
GPTOSS-20B-V3 84.9% 83.0% 86.3% 84.6% N/A
GPTOSS-20B (1-shot) 88.7% 97.6% 78.4% 87.0% 0.964
GPTOSS-120B (1-shot) 89.6% 93.5% 84.3% 88.7% 0.956

7. Defects Found and Fixed

DEFECT-001: README Accuracy Claims Incorrect

Severity: Medium (Documentation error)

Location: README.md lines 63-80

Description: The README contained incorrect accuracy metrics for the GPT-OSS 20B model that did not match the validated results in eval/results_20b_1shot.json.

Metric README (Wrong) Actual (Validated) Difference
Accuracy 87.7% 88.68% (88.7%) +1.0%
Precision 95.2% 97.56% (97.6%) +2.4%
F1 86.0% 86.96% (87.0%) +1.0%
Recall 78.4% 78.43% Correct

Root Cause: Manual transcription error when documenting results.

Fix Applied: Updated README.md with correct values:

- ## Reproduce Results (87.7% on Gold_106)
- Verify the published 20B model achieves 87.7% accuracy:
+ ## Reproduce Results (88.7% on Gold_106)
+ Verify the published 20B model achieves 88.7% accuracy:

- # Expected output: Accuracy 87.7%, Precision 95.2%, Recall 78.4%, F1 86.0%
+ # Expected output: Accuracy 88.7%, Precision 97.6%, Recall 78.4%, F1 87.0%

Commit: 0926ae7 - Fix README accuracy claims to match validated results

Status: FIXED

8. Recommendations (Future Improvements)

Minor Issues (Non-Blocking)

  1. Raw Response Storage: The results file stores parsed labels in response field, not raw model outputs. Consider storing raw outputs for debugging.

  2. Response Parsing Edge Cases: The parsing logic could be made more robust:

    response = response.strip().upper()
if response == "AF":
    return 1
elif response == "ALIGNED":
    return 0
else:
    # Log unexpected response for review
    return 0

  3. AUROC Missing: gptoss_20b_v3_gold106_results.json doesn't include AUROC. Consider adding confidence scores.

Positive Observations

  1. Consistent label mapping across all scripts
  2. Proper use of sklearn metrics
  3. Confusion matrix correctly structured
  4. Conservative default behavior (classify as aligned when uncertain)
  5. Clean separation between data loading and evaluation logic

9. Conclusion

VALIDATION STATUS: PASSED (with 1 defect fixed)

The Gold-106 dataset and experiment results are valid:

  • All 106 samples have prompts (scratchpad text) and labels
  • All mathematical calculations (accuracy, precision, recall, F1) are correct
  • Predictions array matches confusion matrix counts exactly
  • Evaluation scripts follow correct methodology

Defects Found: 1 (documentation error in README - FIXED)

The core experimental results and evaluation scripts are correct. The only defect found was a documentation error where the README listed incorrect accuracy metrics (87.7% instead of 88.7%). This has been corrected.

Appendix: Verification Commands

# Verify dataset structure
python3 -c "import json; d=json.load(open('reproduce-120b/gold_106.json')); print(len(d['samples']))"

# Verify metrics
python3 -c "
import json
with open('data/gptoss_20b_v3_gold106_results.json') as f:
    r = json.load(f)
tp,fp,tn,fn = r['confusion']['tp'],r['confusion']['fp'],r['confusion']['tn'],r['confusion']['fn']
print(f'Accuracy: {(tp+tn)/(tp+fp+tn+fn)*100:.2f}%')
print(f'Precision: {tp/(tp+fp)*100:.2f}%')
print(f'Recall: {tp/(tp+fn)*100:.2f}%')
"