AutoConfig.register() ignored when trust_remote_code=True and auto_map is present

[HanFa]

Description

AutoConfig.register() is silently ignored when trust_remote_code=True and the model's config.json contains auto_map.AutoConfig. This makes it impossible for downstream libraries to override a broken remote config class.

Reproduction

from transformers import AutoConfig, PretrainedConfig

# 1. Register a local config class for a model_type
class MyFixedConfig(PretrainedConfig):
    model_type = "hyperclovax_vlm"
    def __init__(self, text_config=None, **kwargs):
        self.text_config = None  # always set
        super().__init__(**kwargs)
    def get_text_config(self, decoder=False):
        return self.text_config if self.text_config is not None else self

AutoConfig.register("hyperclovax_vlm", MyFixedConfig, exist_ok=True)

# 2. Load with trust_remote_code=False — uses registered class ✓
config = AutoConfig.from_pretrained(
    "naver-hyperclovax/HyperCLOVAX-SEED-Vision-Instruct-3B",
    trust_remote_code=False,
)
print(type(config))  # <class 'MyFixedConfig'>

# 3. Load with trust_remote_code=True — ignores registered class ✗
config = AutoConfig.from_pretrained(
    "naver-hyperclovax/HyperCLOVAX-SEED-Vision-Instruct-3B",
    trust_remote_code=True,
)
print(type(config))  # remote HCXVisionConfig, NOT MyFixedConfig
# This crashes because the remote code has a bug in get_text_config()

Root Cause

In configuration_auto.py, from_pretrained checks has_remote_code and trust_remote_code before checking CONFIG_MAPPING:

# Line 1478-1483
if has_remote_code and trust_remote_code:
    config_class = get_class_from_dynamic_module(class_ref, ...)  # always wins
    return config_class.from_pretrained(...)
elif "model_type" in config_dict:
    config_class = CONFIG_MAPPING[config_dict["model_type"]]  # never reached

When a class has been explicitly registered via AutoConfig.register(), it should take precedence over auto_map remote code — the caller has explicitly said "use this class for this model_type."

Impact

This affects any downstream library (e.g., vLLM) that vendors config fixes for models with broken remote code. Even after registering the fixed config, internal calls from AutoTokenizer.from_pretrained() and AutoProcessor.from_pretrained() pass trust_remote_code=True and bypass the registration.

The code trace looks something like:

vLLM code (we control):
    AutoProcessor.from_pretrained(..., trust_remote_code=True)  ← must pass True
                                                                   because the PROCESSOR
                                                                   itself is remote code

    transformers internals (we don't control):
      → AutoConfig.from_pretrained(..., trust_remote_code=True)  ✗ loads broken remote config

Concrete example: vLLM vendors a fixed HCXVisionConfig to handle empty initialization (needed for v5's @strict validation). get_config() correctly uses the vendored class, but AutoProcessor internally calls AutoConfig.from_pretrained(trust_remote_code=True) which loads the broken remote code and crashes.

Related: vllm-project/vllm#38387, #44956

Suggested Fix

Only prefer remote code when no local/registered class exists:

if has_remote_code and trust_remote_code and not has_local_code:
    config_class = get_class_from_dynamic_module(class_ref, ...)
elif "model_type" in config_dict:
    config_class = CONFIG_MAPPING[config_dict["model_type"]]

Environment

• transformers: main branch (commit 9a9997f)
• Python: 3.12

[Rocketknight1]

cc @hmellor since this seems mostly like a vLLM compatibility issue, see also the PR at #45094

[hmellor]

Thanks for the ping, yeah this was made for vLLM compatibility. Spawning from vllm-project/vllm#38387. I'm not sure this is necessary though as it adds implicit behaviour. I'll discuss further in the parent issue

[hmellor]

Ok the desired outcome is slightly different, the expected behaviour is as follows:

• If local code exists in Transformers and remote code exists in the checkpoint - do what trust_remote_code says
• If local code exists that was registered outside of Transformers (i.e. by vLLM) - ignore trust_remote_code and always use the expicitly registered local code

I have updated #45094 to reflect this and it should merge soon.