Skip to content

bug: Inspect AI implementation runs n+1 step before n step #46

@TheFloatingString

Description

@TheFloatingString

I get the following error when running an implementation (mostly the same code) as eval/inspet_ai/scicode.py through Inspect AI:

eval/inspect_ai/scicode.py at line 172:

│   455 │   │   │   │   │   │   )                                                                         │
│   456 │   │   │   │   │   │   self.previous_llm_code[prev_step] = function_code                         │
│   457 │   │   │   │   │   else:                                                                         │
│ ❱ 458 │   │   │   │   │   │   raise Exception(f'Generating problem {prob_id} step {num_steps}           │
│   459 │   │                                                                                             │
│   460 │   │   prompt, previous_code = self.generate_prompt_with_steps(                                  │
│   461 │   │   │   prob_data,                                                                            │
╰─────────────────────────────────────────────────────────────────────────────────────────────────────────╯
Exception: Generating problem 13 step 7 ahead of step 6.  

this is run on the test split, and the error seems to go away on the validation split.

changing the number of --max-connections has no effect on this error. This was run with openai/o3 as a model.

Metadata

Metadata

Assignees

No one assigned

    Labels

    No labels
    No labels

    Type

    No type
    No fields configured for issues without a type.

    Projects

    No projects

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions