Skip to content

contains the question when inference #38

@wac81

Description

@wac81

like this:
user: Maslow's principle <----------------- user input
<reserved_102>Maslow's principle <----------------- model inference repeat question
<reserved_103>The physiological needs of human beings are diverse, including physiological needs and social needs <---Should answer the content after <reserved_103>

my code, use stream
`

while True:
inp = input("user:")
inputs = "<reserved_102>{}\n<reserved_103>".format(inp)
inputs = tokenizer(inputs, return_tensors='pt',padding=True)
inputs = inputs.to('cpu')
generated = model.generate(**inputs, max_new_tokens=128, streamer = streamer,
do_sample=True,
top_k=20,
top_p=0.4,
temperature=0.2,
)
tokenizer.batch_decode(generated)

`

Metadata

Metadata

Assignees

No one assigned

    Labels

    No labels
    No labels

    Projects

    No projects

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions