diffugpt-s (127M) generations aren't fluent and repeats

[nitsanluke]

Hi @summmeer ,

Thanks for all the great contributions on the new dLLMs. I tested the small model (diffugpt-s) with the inference script provided. It yields repeats and incomplete generations. Is this something you've observed or am I missing something in the inference setting.

1. If this is indeed the case can you elaborate on what size of the models do you start to find more fluent and useful generations.
2. Also the small models were CPTd for ~130B tokens and the larger Llama model was CPTd on ~60B, was it a compute limitation or the larger model converts earlier.

Thanks!

Inferences setting:

script: inf_diffugpt.py

# conditional generation with 16 new tokens and 16 steps.


   diffusion_steps = 16
    gen_len = 16

    print("="*20, "Prefix gen...")
    prefix = [tokenizer.bos_token_id] + tokenizer.encode("obama is the president")

    src_mask = [1]*len(prefix)+[0]*(gen_len)
    x0 = prefix + [0]*(gen_len)

    inputs = {
        "input_ids": torch.tensor([x0]), 
        "src_mask": torch.tensor([src_mask])
    }
    print(inputs)
    torch.manual_seed(1234)
    res = generate_samples(model, args, tokenizer, inputs, verbose=args.verbose)
    pred = tokenizer.decode(res.tolist()[0])
    print(pred)
"obama is the president that is being president the president of assistant assistant vice is and assistant vice is the"

[summmeer]

Hi! Thank you for your interest on our projects.
We suggest setting the gen_len according to the pre-trained length, otherwise the model will take this arbitrary length as out-of-domain (OOD). Also, please pass diffusion_steps into args and when diffusion_steps = gen_len-len(prefix) we can get the best decoding quality. I think the following snippet can get more fluent sentences.

DiffuLLaMA/inf_diffugpt.py

Lines 49 to 62 in c17e897

	# conditional generation
	print("="*20, "Prefix gen...")
	prefix = [tokenizer.bos_token_id] + tokenizer.encode("Today is a wonderful day,")
	src_mask = [1]*len(prefix)+[0]*(gen_len-len(prefix))
	x0 = prefix + [0]*(gen_len-len(prefix))
	inputs = {
	"input_ids": torch.tensor([x0]),
	"src_mask": torch.tensor([src_mask])
	}
	res = generate_samples(model, args, tokenizer, inputs, verbose=args.verbose)
	pred = tokenizer.decode(res.tolist()[0])
	print(pred)

DiffuLLaMA was only trained on 60B tokens at that time due to the resource limitation. Please refer to our latest works on dLLMs which get much better performance!!

1. Dream https://hkunlp.github.io/blog/2025/dream/
2. DiffuCoder https://github.com/apple/ml-diffucoder

[nitsanluke]

Hi! Thank you for your interest on our projects. We suggest setting the gen_len according to the pre-trained length, otherwise the model will take this arbitrary length as out-of-domain (OOD). Also, please pass diffusion_steps into args and when diffusion_steps = gen_len-len(prefix) we can get the best decoding quality. I think the following snippet can get more fluent sentences.

DiffuLLaMA/inf_diffugpt.py

Lines 49 to 62 in c17e897

conditional generation

print("="*20, "Prefix gen...")
prefix = [tokenizer.bos_token_id] + tokenizer.encode("Today is a wonderful day,")

src_mask = [1]len(prefix)+[0](gen_len-len(prefix))
x0 = prefix + [0]*(gen_len-len(prefix))

inputs = {
"input_ids": torch.tensor([x0]),
"src_mask": torch.tensor([src_mask])
}
res = generate_samples(model, args, tokenizer, inputs, verbose=args.verbose)
pred = tokenizer.decode(res.tolist()[0])
print(pred)
DiffuLLaMA was only trained on 60B tokens at that time due to the resource limitation. Please refer to our latest works on dLLMs which get much better performance!!

1. Dream https://hkunlp.github.io/blog/2025/dream/
2. DiffuCoder https://github.com/apple/ml-diffucoder

Thanks for the quick response. We found similar results with the sequence length match to the training setting. This seemed more prominent in the small models and the larger models seems to generate well at arbitrary lengths.