-
Notifications
You must be signed in to change notification settings - Fork 26
Open
Description
like this:
user: Maslow's principle <----------------- user input
<reserved_102>Maslow's principle <----------------- model inference repeat question
<reserved_103>The physiological needs of human beings are diverse, including physiological needs and social needs <---Should answer the content after <reserved_103>
my code, use stream
`
while True:
inp = input("user:")
inputs = "<reserved_102>{}\n<reserved_103>".format(inp)
inputs = tokenizer(inputs, return_tensors='pt',padding=True)
inputs = inputs.to('cpu')
generated = model.generate(**inputs, max_new_tokens=128, streamer = streamer,
do_sample=True,
top_k=20,
top_p=0.4,
temperature=0.2,
)
tokenizer.batch_decode(generated)
`
Metadata
Metadata
Assignees
Labels
No labels