Skip to content

tiktoken/core.py error #33

@BPejon

Description

@BPejon

Hello,

I am trying to use Auto Survey for Material Science Field on my desktop Ubuntu 24.04.2 LTS.
I ran the code the same way as the README but got this error:

Traceback (most recent call last):
File "/home/breno/Documentos/AutoSurvey/main.py", line 88, in
main(args)
File "/home/breno/Documentos/AutoSurvey/main.py", line 72, in main
outline_with_description, outline_wo_description = write_outline(args.topic, args.model, args.section_num, args.outline_reference_num, db, args.api_key, args.api_url)
File "/home/breno/Documentos/AutoSurvey/main.py", line 33, in write_outline
outline = outline_writer.draft_outline(topic, outline_reference_num, 30000, section_num)
File "/home/breno/Documentos/AutoSurvey/src/agents/outline_writer.py", line 34, in draft_outline
outlines = self.generate_rough_outlines(topic=topic, papers_chunks = abs_chunks, titles_chunks = titles_chunks, section_num=section_num)
File "/home/breno/Documentos/AutoSurvey/src/agents/outline_writer.py", line 113, in generate_rough_outlines
self.output_token_usage += self.token_counter.num_tokens_from_list_string(outlines)
File "/home/breno/Documentos/AutoSurvey/src/utils.py", line 24, in num_tokens_from_list_string
num += len(self.encoding.encode(s))
File "/home/breno/Documentos/AutoSurvey/venv/lib/python3.10/site-packages/tiktoken/core.py", line 116, in encode
if match := _special_token_regex(disallowed_special).search(text):

How this error can be fixed ??
Thanks,
Breno

Metadata

Metadata

Assignees

No one assigned

    Labels

    No labels
    No labels

    Type

    No type

    Projects

    No projects

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions