Thank you for making this open source! Why is the batch size one for the prompt phase ? - https://github.com/pku-lemonade/TokenSim/blob/3a0da974c7f97e2d69d85626cc683c54ed4640e2/TokenSim/llm/llm_engine.py#L337
Thank you for making this open source!
Why is the batch size one for the prompt phase ? -
TokenSim/TokenSim/llm/llm_engine.py
Line 337 in 3a0da97