I'm having trouble running gguf models

[fatihsazan]

Hi,

Is there something wrong with using it this way? I can't run the gguf models I want to try.

from ctransformers import AutoModelForCausalLM

model_name = "SanctumAI/Llama-3.2-3B-Instruct-GGUF"
gguf_file = "llama-3.2-3b-instruct.Q2_K.gguf"

llm = AutoModelForCausalLM.from_pretrained(
    model_name,
    model_file=gguf_file,
    model_type="gguf")

Error output:

, line 8, in <module>
    llm = AutoModelForCausalLM.from_pretrained(
  File "/usr/local/lib/python3.10/dist-packages/ctransformers/hub.py", line 175, in from_pretrained
    llm = LLM(
  File "/usr/local/lib/python3.10/dist-packages/ctransformers/llm.py", line 253, in __init__
    raise RuntimeError(
RuntimeError: Failed to create LLM 'gguf' from '/root/.cache/huggingface/hub/models--SanctumAI--Llama-3.2-3B-Instruct-GGUF/blobs/c77eb142ab869944f388ff093fc7276ea15c4e1f810ceb76554fc5ae77694c19'.

[werruww]

yes

[shama2025]

Hello @fatihsazan I haven't done much with ctransformers but from my understanding, the model type "gguf" isn't a valid type. GGUF is a file format for storing models. You would need to change it to llama since that is the type of the model you are using. Here is an example with The Blokes Mistral models.

MISTRAL_REPO = "TheBloke/Mistral-7B-Instruct-v0.2-GGUF"
model = AutoModelForCausalLM.from_pretrained(
            MISTRAL_REPO, hf=False, model_type="mistral"
        )

If you already download the gguf file locally, that requires the huggingface_hub snapshot_download function which will load the models in a folder of your choice and then you can call using the AutoModelForCasualLm model_path_or_repo_path parameter to point to the folder.

MODEL_DIR = "path/to/mistral/model"
model = AutoModelForCausalLM.from_pretrained(
            MODEL_DIR, hf=False, model_type="mistral"
        )