Skip to content

zip support, change upload flow#289

Merged
mayinghan merged 5 commits intomainfrom
zip-support
Oct 25, 2025
Merged

zip support, change upload flow#289
mayinghan merged 5 commits intomainfrom
zip-support

Conversation

@shreymodi1
Copy link
Copy Markdown
Contributor

@shreymodi1 shreymodi1 commented Oct 23, 2025

Added support for GCS upload and changed test-discovery

Changes in test-discovery function -

(Issues with older test_discovery)

  1. It was running top level module code for tests, which was causing some background processes to be created and the ep upload command wasn't ending smoothly.
  2. Some tests having relative imports weren't being discovered

Solution - Fixed it by using the pytest internal collector mechanism, (mainly uses our discoverability, just handles modules better)

New upload flow -

3 endpoints

  1. evaluatorsV2 -> getUploadEndpoint -> validateUpload

We call the first entry point with entry function name, then second endpoint after compressing the entire current workingdirectory (we ignore file names listed in .gitignore and .dockerignore, also submit the git hash in pep440 for versioning) then validateUpload after the directory upload has finished.

The developer has to take care of the following things when using ep upload -

You will need to have a requirements.tx in your pwd before running ep upload. (else we don't continue with the upload)
We are ignoring all file patterns stated in .gitignore/.dockerignore while pushing evaluator code for the run
It is recommended to git commit before running ep upload, so that we can version properly and it is easy for the developer to track back the evaluator code version.


Note

Reworks test discovery to use pytest collection and changes evaluator creation to upload a tar.gz of the project to GCS via new V2 endpoints with validation.

  • CLI (ep upload):
    • Use pytest’s collection (--collect-only) to discover @evaluation_test functions; ignore problematic files (e.g., setup.py, hidden, test_discovery*).
    • Handle pytest proxy pytestmark objects; extract parametrization info; generate stable nodeids and entry points.
  • Evaluator create flow:
    • New V2 API usage: .../evaluatorsV2:getUploadEndpoint:validateUpload.
    • Build minimal criteria (no embedded code); include parent, commitHash (pep440), and entryPoint when provided.
    • Require requirements.txt; create tar.gz of CWD honoring .gitignore/.dockerignore; upload to GCS via signed URL with retries; validate upload.
  • Misc/Preview:
    • Remove stray payload print; keep multi-metrics/rollup settings.
  • Tests:
    • Add E2E tests for upload workflow including pytest discovery, GCS upload mocking, and validation.
    • Update integration/unit tests to expect V2 endpoints, minimal criteria, entryPoint, and requirements.txt; improve cleanup with shutil.rmtree.

Written by Cursor Bugbot for commit a766d40. This will update automatically on new commits. Configure here.

cursor[bot]

This comment was marked as outdated.

cursor[bot]

This comment was marked as outdated.

cursor[bot]

This comment was marked as outdated.

f"Upload attempt {attempt + 1}/{max_retries} failed: {e}. "
f"Retrying in {wait_time}s..."
)
time.sleep(wait_time)
Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

consider to remove this manual retry handler, wrap the function that you want to have retry and use a decorator. but this can be done in a followup pr

@mayinghan mayinghan merged commit 8ad7d26 into main Oct 25, 2025
9 checks passed
@mayinghan mayinghan deleted the zip-support branch October 25, 2025 22:18
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

5 participants