Skip to content

Add LC benchmarks#1437

Open
hsiehjackson wants to merge 4 commits into
mainfrom
chsieh/add-lc-benchmarks
Open

Add LC benchmarks#1437
hsiehjackson wants to merge 4 commits into
mainfrom
chsieh/add-lc-benchmarks

Conversation

@hsiehjackson
Copy link
Copy Markdown

@hsiehjackson hsiehjackson commented May 27, 2026 •
edited
Loading

  • Change MRCR / LongBench-V2 / LongCodeBench to support HF tokenizer and customized max_context_tokens.
  • Add Graphwalks

Scores for N3 Ultra:

Method LongBenchV2 ≤ 1M(470) LongCodeBench ≤ 1M(443) MRCR ≤ 128K(1470) GraphWalks < 1M(950)
Skills 61.9 75.4 67.8 64.5
Gym 63.6 74.5 69.1 62.2

@hsiehjackson
add LC benchmarks
3dd31d2 
Signed-off-by: Cheng-Ping Hsieh <chsieh@nvidia.com>
@copy-pr-bot
Copy link
Copy Markdown

copy-pr-bot Bot commented May 27, 2026

This pull request requires additional validation before any workflows can run on NVIDIA's runners.

Pull request vetters can view their responsibilities here.

Contributors can view more details about this message here.

@hsiehjackson hsiehjackson requested a review from gwarmstrong May 27, 2026 22:21
hsiehjackson and others added 3 commits May 27, 2026 15:24
@hsiehjackson
fix
d63d227 
Signed-off-by: Cheng-Ping Hsieh <chsieh@nvidia.com>
@hsiehjackson
Merge branch 'main' into chsieh/add-lc-benchmarks
12509a5
@hsiehjackson
save
b4cf8bb 
Signed-off-by: Cheng-Ping Hsieh <chsieh@nvidia.com>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Reviewers

@gwarmstrong gwarmstrong Awaiting requested review from gwarmstrong

At least 1 approving review is required to merge this pull request.

Assignees

No one assigned

Labels

None yet

Projects

None yet

Milestone

No milestone

Development

Successfully merging this pull request may close these issues.

1 participant

@hsiehjackson