Fix shipped evidence scoring#94
Open
SaharPak wants to merge 1 commit into
Open
Conversation
The previous heuristic scored on a narrow 7-pattern keyword list
(shipped | launched | deployed | built and released | in production |
live users | released to). A strong AI PM CV describing real shipped
work with verbs like 'owned', 'led', 'drove', 'built' would floor at
1/5. A strong backend CV using 'migrated', 'designed', 'led', 'owned'
also scored 1/5 because none of the literal phrases appeared.
Replace the keyword matcher with a multi-signal additive scorer:
strong shipping verbs (shipped/launched/released/...), medium
engineering verbs (built/implemented/migrated/...), leadership
verbs (owned/led/drove/...), improvement verbs (improved/reduced/
cut/scaled/...), concrete evidence objects (feature/api/pipeline/
service/...), quantified impact (percentages, dollar amounts,
'180k MAU' style scaled users), outcome metrics (conversion/latency/
NPS/MAU/DAU/...), and production signals (in production/used by/
open source). Subtract for vague patterns ('worked on', 'helped with',
'responsible for', ...). Final score clamps to [1, 5].
Generic terms (tools, frontend, backend, jobs) and bare noun forms
(software 'engineer') are excluded so weak CVs stay at the floor.
Adds three synthetic fixtures under
packages/core/src/__fixtures__/synthetic/ and a focused
shipped-evidence.test.ts asserting the strong AI PM and strong
backend score 4-5/5, the weak generic CV scores 1-2/5, and the
existing core/sample-cv-strong and core/sample-cv-weak fixtures are
not regressed.
📝 WalkthroughWalkthroughAdds three new synthetic markdown CV fixtures (strong-ai-pm, strong-backend, weak-generic), rewrites the ChangesShipped Evidence Scoring
Estimated code review effort: 3 (Moderate) | ~25 minutes Sequence Diagram(s)sequenceDiagram
participant Test as shipped-evidence.test.ts
participant Evaluate as evaluate()
participant Scorer as scoreShippedEvidence()
participant Counter as countMatches()
Test->>Evaluate: evaluate(cvText)
Evaluate->>Scorer: scoreShippedEvidence(cvText)
Scorer->>Counter: countMatches(text, pattern) for each category
Counter-->>Scorer: match counts
Scorer->>Scorer: apply threshold adjustments
Scorer->>Scorer: clamp and round score to [1,5]
Scorer-->>Evaluate: score
Evaluate-->>Test: dimensions with Shipped Evidence score
Possibly related issues
Possibly related PRs
Suggested reviewers: 🚥 Pre-merge checks | ✅ 5✅ Passed checks (5 passed)
✨ Finishing Touches📝 Generate docstrings
🧪 Generate unit tests (beta)
Thanks for using CodeRabbit! It's free for OSS, and your support helps us grow. If you like it, consider giving us a shout-out. Comment |
![coderabbitai[bot]](https://avatars.githubusercontent.com/in/347564?s=60&v=4){kind=small}
There was a problem hiding this comment.
Actionable comments posted: 1
🧹 Nitpick comments (1)
packages/core/src/evaluator/index.ts (1)
262-284: 🎯 Functional Correctness | 🔵 Trivial | ⚖️ Poor tradeoffAdditive scoring easily clamps to ceiling regardless of shipping-verb signal.
With
strongCount/mediumCount/leadershipCountcontributing at most ~3 combined points, andobjectsCount,quantifiedCount,outcomeCount,productionCounteach able to add up to ~1.5–2 more, a CV with rich quantified metrics and production language but zero shipping/leadership verbs can still reach the 5-point ceiling. This dilutes "Shipped Evidence" as a distinct signal from "Quantified Impact"/"Public Proof" (already separately scored dimensions). Given the case-sensitivity issue above further suppresses the verb-based counts, in practice this dimension is currently driven mostly by objects/metrics/production phrases rather than the verbs called out in the PR description.Consider tightening the ceiling contribution from non-verb signals, or requiring a minimum combined verb signal before applying the outcome/production bonuses, so "Shipped Evidence" measures verb-based shipping evidence more directly.
🤖 Prompt for AI Agents
Verify each finding against current code. Fix only still-valid issues, skip the rest with a brief reason, keep changes minimal, and validate. In `@packages/core/src/evaluator/index.ts` around lines 262 - 284, The Shipped Evidence score in evaluator/index.ts is being driven too much by non-verb signals, so a resume can hit the ceiling without strong shipping/leadership verbs. Update the scoring logic in the function that computes score to make verb-based counts such as strongCount, mediumCount, and leadershipCount the primary gate, and reduce or condition the objectsCount, quantifiedCount, outcomeCount, and productionCount bonuses unless enough verb evidence is present. Keep the final clamp, but adjust the additive weights so this dimension reflects shipping verbs more directly.
🤖 Prompt for all review comments with AI agents
Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.
Inline comments:
In `@packages/core/src/evaluator/index.ts`:
- Around line 142-219: The verb and vague-language matchers in the evaluator are
missing capitalized resume-bullet forms, so the scorer undercounts common
signals. Update the regexes in the strongVerbs, mediumVerbs, leadershipVerbs,
improvementVerbs, and vaguePatterns arrays to be case-insensitive so matches
like Owned, Led, Built, Worked on, and Responsible for are counted consistently.
---
Nitpick comments:
In `@packages/core/src/evaluator/index.ts`:
- Around line 262-284: The Shipped Evidence score in evaluator/index.ts is being
driven too much by non-verb signals, so a resume can hit the ceiling without
strong shipping/leadership verbs. Update the scoring logic in the function that
computes score to make verb-based counts such as strongCount, mediumCount, and
leadershipCount the primary gate, and reduce or condition the objectsCount,
quantifiedCount, outcomeCount, and productionCount bonuses unless enough verb
evidence is present. Keep the final clamp, but adjust the additive weights so
this dimension reflects shipping verbs more directly.
🪄 Autofix (Beta)
Fix all unresolved CodeRabbit comments on this PR:
- Push a commit to this branch (recommended)
- Create a new PR with the fixes
ℹ️ Review info
⚙️ Run configuration
Configuration used: Path: .coderabbit.yaml
Review profile: CHILL
Plan: Pro Plus
Run ID: 47fc6c19-a4c6-4a88-b498-2a72bb61ff58
📒 Files selected for processing (5)
packages/core/src/__fixtures__/synthetic/strong-ai-pm.mdpackages/core/src/__fixtures__/synthetic/strong-backend.mdpackages/core/src/__fixtures__/synthetic/weak-generic.mdpackages/core/src/__tests__/shipped-evidence.test.tspackages/core/src/evaluator/index.ts
Comment on lines
+142
to
+219
| // Strong shipping verbs (each pattern matched counts once). | ||
| const strongVerbs = [ | ||
| /\bshipped\b/, | ||
| /\blaunched\b/, | ||
| /\breleased\b/, | ||
| /\bdelivered\b/, | ||
| /\bdeployed\b/, | ||
| /\brolled\s+out\b/, | ||
| ]; | ||
| const matches = shippedIndicators.filter((r) => r.test(cv)).length; | ||
| return Math.min(5, Math.max(1, matches + 1)); | ||
| // Medium shipping verbs (engineering work verbs). Each pattern requires | ||
| // the verb in past/present-participle form (no bare noun match) so that | ||
| // job titles like "software engineer" don't count as shipped work. | ||
| const mediumVerbs = [ | ||
| /\bbuilt\b/, | ||
| /\bimplement(?:ed|s|ing)\b/, | ||
| /\bmigrat(?:ed|ing|e|ion)\b/, | ||
| /\bintegrat(?:ed|ing|e|ion)\b/, | ||
| /\bautomat(?:ed|ing|e|ion)\b/, | ||
| /\bredesign(?:ed|ing)\b/, | ||
| /\bcreat(?:ed|ing|e|ion)\b/, | ||
| /\bdevelop(?:ed|ing|ment)\b/, | ||
| /\bdesign(?:ed|ing)\b/, | ||
| /\bauthor(?:ed|ing)\b/, | ||
| /\bcod(?:ed|ing)\b/, | ||
| /\bprogram(?:med|ming)\b/, | ||
| /\bengineer(?:ed|ing)\b/, | ||
| ]; | ||
| // Leadership verbs (ownership signals). | ||
| const leadershipVerbs = [ | ||
| /\bowned\b/, | ||
| /\bled\b/, | ||
| /\bdrove\b/, | ||
| /\bspearheaded\b/, | ||
| /\borchestrated\b/, | ||
| /\bchampioned\b/, | ||
| /\bheaded\b/, | ||
| ]; | ||
| // Improvement verbs (measurable change). Each pattern requires the verb | ||
| // in past/present-participle form. | ||
| const improvementVerbs = [ | ||
| /\bimprov(?:ed|ing|ement)\b/, | ||
| /\breduc(?:ed|ing|tion)\b/, | ||
| /\bincreas(?:ed|ing)\b/, | ||
| /\boptimi[sz](?:ed|ing|ation)\b/, | ||
| /\bcut\b/, | ||
| /\bgrew\b/, | ||
| /\bscal(?:ed|ing)\b/, | ||
| /\bdoubled\b/, | ||
| /\btripled\b/, | ||
| /\bboost(?:ed|ing)\b/, | ||
| /\bsav(?:ed|ing)\b/, | ||
| /\baccelerat(?:ed|ing|ion)\b/, | ||
| /\bstreamlin(?:ed|ing|e)\b/, | ||
| /\bharden(?:ed|ing)\b/, | ||
| /\blower(?:ed|ing)\b/, | ||
| /\bshorten(?:ed|ing)\b/, | ||
| /\brais(?:ed|ing)\b/, | ||
| /\bexpand(?:ed|ing)\b/, | ||
| /\bwiden(?:ed|ing)\b/, | ||
| /\blift(?:ed|ing)\b/, | ||
| ]; | ||
| // Vague patterns that look like shipped work but rarely indicate | ||
| // ownership of a shipped artifact. | ||
| const vaguePatterns = [ | ||
| /\bworked on\b/, | ||
| /\bhelped (?:with|the)\b/, | ||
| /\bparticipated in\b/, | ||
| /\bwas involved in\b/, | ||
| /\bassisted (?:with|in)\b/, | ||
| /\btasked with\b/, | ||
| /\bin charge of\b/, | ||
| /\bworked with\b/, | ||
| /\bcollaborated with\b/, | ||
| /\bsome\s+(?:code|things|work|stuff|projects|cloud)\b/, | ||
| /\bdid some\b/, | ||
| /\bdid stuff\b/, | ||
| /\bresponsible for\b/, | ||
| ]; |
There was a problem hiding this comment.
🎯 Functional Correctness | 🟠 Major | ⚡ Quick win
🧩 Analysis chain
🏁 Script executed:
#!/bin/bash
set -euo pipefail
printf '\n== File inventory ==\n'
git ls-files 'packages/core/src/evaluator/index.ts' 'packages/core/**' | sed -n '1,200p'
printf '\n== Outline evaluator file ==\n'
ast-grep outline packages/core/src/evaluator/index.ts --view expanded || true
printf '\n== Search fixtures/tests ==\n'
rg -n --hidden --glob '!**/node_modules/**' --glob '!**/dist/**' --glob '!**/build/**' 'strong-ai-pm|strong-backend|weak-generic|leadershipCount|vagueCount|strongVerbs|mediumVerbs|improvementVerbs|vaguePatterns' packages/core -S || trueRepository: TechImmigrants/cv-builder
Length of output: 3410
🏁 Script executed:
#!/bin/bash
set -euo pipefail
printf '\n== Evaluator excerpt ==\n'
sed -n '135,290p' packages/core/src/evaluator/index.ts
printf '\n== Shipped evidence tests ==\n'
sed -n '1,240p' packages/core/src/__tests__/shipped-evidence.test.ts
printf '\n== Synthetic fixtures ==\n'
for f in \
packages/core/src/__fixtures__/synthetic/strong-ai-pm.md \
packages/core/src/__fixtures__/synthetic/strong-backend.md \
packages/core/src/__fixtures__/synthetic/weak-generic.md
do
printf '\n--- %s ---\n' "$f"
sed -n '1,220p' "$f"
doneRepository: TechImmigrants/cv-builder
Length of output: 14801
🏁 Script executed:
#!/bin/bash
set -euo pipefail
node <<'JS'
const fs = require('fs');
const path = require('path');
const root = process.cwd();
const fixtures = {
'strong-ai-pm.md': fs.readFileSync(path.join(root, 'packages/core/src/__fixtures__/synthetic/strong-ai-pm.md'), 'utf8'),
'strong-backend.md': fs.readFileSync(path.join(root, 'packages/core/src/__fixtures__/synthetic/strong-backend.md'), 'utf8'),
'weak-generic.md': fs.readFileSync(path.join(root, 'packages/core/src/__fixtures__/synthetic/weak-generic.md'), 'utf8'),
};
const sets = {
strongVerbs: [
/\bshipped\b/,
/\blaunched\b/,
/\breleased\b/,
/\bdelivered\b/,
/\bdeployed\b/,
/\brolled\s+out\b/,
],
strongVerbs_i: [
/\bshipped\b/i,
/\blaunched\b/i,
/\breleased\b/i,
/\bdelivered\b/i,
/\bdeployed\b/i,
/\brolled\s+out\b/i,
],
mediumVerbs: [
/\bbuilt\b/,
/\bimplement(?:ed|s|ing)\b/,
/\bmigrat(?:ed|ing|e|ion)\b/,
/\bintegrat(?:ed|ing|e|ion)\b/,
/\bautomat(?:ed|ing|e|ion)\b/,
/\bredesign(?:ed|ing)\b/,
/\bcreat(?:ed|ing|e|ion)\b/,
/\bdevelop(?:ed|ing|ment)\b/,
/\bdesign(?:ed|ing)\b/,
/\bauthor(?:ed|ing)\b/,
/\bcod(?:ed|ing)\b/,
/\bprogram(?:med|ming)\b/,
/\bengineer(?:ed|ing)\b/,
],
mediumVerbs_i: [
/\bbuilt\b/i,
/\bimplement(?:ed|s|ing)\b/i,
/\bmigrat(?:ed|ing|e|ion)\b/i,
/\bintegrat(?:ed|ing|e|ion)\b/i,
/\bautomat(?:ed|ing|e|ion)\b/i,
/\bredesign(?:ed|ing)\b/i,
/\bcreat(?:ed|ing|e|ion)\b/i,
/\bdevelop(?:ed|ing|ment)\b/i,
/\bdesign(?:ed|ing)\b/i,
/\bauthor(?:ed|ing)\b/i,
/\bcod(?:ed|ing)\b/i,
/\bprogram(?:med|ming)\b/i,
/\bengineer(?:ed|ing)\b/i,
],
leadershipVerbs: [
/\bowned\b/,
/\bled\b/,
/\bdrove\b/,
/\bspearheaded\b/,
/\borchestrated\b/,
/\bchampioned\b/,
/\bheaded\b/,
],
leadershipVerbs_i: [
/\bowned\b/i,
/\bled\b/i,
/\bdrove\b/i,
/\bspearheaded\b/i,
/\borchestrated\b/i,
/\bchampioned\b/i,
/\bheaded\b/i,
],
improvementVerbs: [
/\bimprov(?:ed|ing|ement)\b/,
/\breduc(?:ed|ing|tion)\b/,
/\bincreas(?:ed|ing)\b/,
/\boptimi[sz](?:ed|ing|ation)\b/,
/\bcut\b/,
/\bgrew\b/,
/\bscal(?:ed|ing)\b/,
/\bdoubled\b/,
/\btripled\b/,
/\bboost(?:ed|ing)\b/,
/\bsav(?:ed|ing)\b/,
/\baccelerat(?:ed|ing|ion)\b/,
/\bstreamlin(?:ed|ing|e)\b/,
/\bharden(?:ed|ing)\b/,
/\blower(?:ed|ing)\b/,
/\bshorten(?:ed|ing)\b/,
/\brais(?:ed|ing)\b/,
/\bexpand(?:ed|ing)\b/,
/\bwiden(?:ed|ing)\b/,
/\blift(?:ed|ing)\b/,
],
improvementVerbs_i: [
/\bimprov(?:ed|ing|ement)\b/i,
/\breduc(?:ed|ing|tion)\b/i,
/\bincreas(?:ed|ing)\b/i,
/\boptimi[sz](?:ed|ing|ation)\b/i,
/\bcut\b/i,
/\bgrew\b/i,
/\bscal(?:ed|ing)\b/i,
/\bdoubled\b/i,
/\btripled\b/i,
/\bboost(?:ed|ing)\b/i,
/\bsav(?:ed|ing)\b/i,
/\baccelerat(?:ed|ing|ion)\b/i,
/\bstreamlin(?:ed|ing|e)\b/i,
/\bharden(?:ed|ing)\b/i,
/\blower(?:ed|ing)\b/i,
/\bshorten(?:ed|ing)\b/i,
/\brais(?:ed|ing)\b/i,
/\bexpand(?:ed|ing)\b/i,
/\bwiden(?:ed|ing)\b/i,
/\blift(?:ed|ing)\b/i,
],
vaguePatterns: [
/\bworked on\b/,
/\bhelped (?:with|the)\b/,
/\bparticipated in\b/,
/\bwas involved in\b/,
/\bassisted (?:with|in)\b/,
/\btasked with\b/,
/\bin charge of\b/,
/\bworked with\b/,
/\bcollaborated with\b/,
/\bsome\s+(?:code|things|work|stuff|projects|cloud)\b/,
/\bdid some\b/,
/\bdid stuff\b/,
/\bresponsible for\b/,
],
vaguePatterns_i: [
/\bworked on\b/i,
/\bhelped (?:with|the)\b/i,
/\bparticipated in\b/i,
/\bwas involved in\b/i,
/\bassisted (?:with|in)\b/i,
/\btasked with\b/i,
/\bin charge of\b/i,
/\bworked with\b/i,
/\bcollaborated with\b/i,
/\bsome\s+(?:code|things|work|stuff|projects|cloud)\b/i,
/\bdid some\b/i,
/\bdid stuff\b/i,
/\bresponsible for\b/i,
],
};
for (const [name, text] of Object.entries(fixtures)) {
console.log(`\n## ${name}`);
for (const [setName, patterns] of Object.entries(sets)) {
const matches = patterns.filter((p) => p.test(text)).length;
if (/^.*_i$/.test(setName) || ['strongVerbs','mediumVerbs','leadershipVerbs','improvementVerbs','vaguePatterns'].includes(setName)) {
console.log(`${setName}: ${matches}`);
}
}
}
JSRepository: TechImmigrants/cv-builder
Length of output: 771
🏁 Script executed:
#!/bin/bash
set -euo pipefail
printf '\n== Evaluator excerpt ==\n'
sed -n '135,290p' packages/core/src/evaluator/index.ts
printf '\n== Shipped evidence tests ==\n'
sed -n '1,240p' packages/core/src/__tests__/shipped-evidence.test.ts
printf '\n== Synthetic fixtures ==\n'
for f in \
packages/core/src/__fixtures__/synthetic/strong-ai-pm.md \
packages/core/src/__fixtures__/synthetic/strong-backend.md \
packages/core/src/__fixtures__/synthetic/weak-generic.md
do
printf '\n--- %s ---\n' "$f"
sed -n '1,220p' "$f"
done
printf '\n== Regex count probe ==\n'
node <<'JS'
const fs = require('fs');
const path = require('path');
const fixtures = {
'strong-ai-pm.md': fs.readFileSync(path.join(process.cwd(), 'packages/core/src/__fixtures__/synthetic/strong-ai-pm.md'), 'utf8'),
'strong-backend.md': fs.readFileSync(path.join(process.cwd(), 'packages/core/src/__fixtures__/synthetic/strong-backend.md'), 'utf8'),
'weak-generic.md': fs.readFileSync(path.join(process.cwd(), 'packages/core/src/__fixtures__/synthetic/weak-generic.md'), 'utf8'),
};
const groups = {
strongVerbs: [/\bshipped\b/, /\blaunched\b/, /\breleased\b/, /\bdelivered\b/, /\bdeployed\b/, /\brolled\s+out\b/],
strongVerbs_i: [/\bshipped\b/i, /\blaunched\b/i, /\breleased\b/i, /\bdelivered\b/i, /\bdeployed\b/i, /\brolled\s+out\b/i],
mediumVerbs: [/\bbuilt\b/, /\bimplement(?:ed|s|ing)\b/, /\bmigrat(?:ed|ing|e|ion)\b/, /\bintegrat(?:ed|ing|e|ion)\b/, /\bautomat(?:ed|ing|e|ion)\b/, /\bredesign(?:ed|ing)\b/, /\bcreat(?:ed|ing|e|ion)\b/, /\bdevelop(?:ed|ing|ment)\b/, /\bdesign(?:ed|ing)\b/, /\bauthor(?:ed|ing)\b/, /\bcod(?:ed|ing)\b/, /\bprogram(?:med|ming)\b/, /\bengineer(?:ed|ing)\b/],
mediumVerbs_i: [/\bbuilt\b/i, /\bimplement(?:ed|s|ing)\b/i, /\bmigrat(?:ed|ing|e|ion)\b/i, /\bintegrat(?:ed|ing|e|ion)\b/i, /\bautomat(?:ed|ing|e|ion)\b/i, /\bredesign(?:ed|ing)\b/i, /\bcreat(?:ed|ing|e|ion)\b/i, /\bdevelop(?:ed|ing|ment)\b/i, /\bdesign(?:ed|ing)\b/i, /\bauthor(?:ed|ing)\b/i, /\bcod(?:ed|ing)\b/i, /\bprogram(?:med|ming)\b/i, /\bengineer(?:ed|ing)\b/i],
leadershipVerbs: [/\bowned\b/, /\bled\b/, /\bdrove\b/, /\bspearheaded\b/, /\borchestrated\b/, /\bchampioned\b/, /\bheaded\b/],
leadershipVerbs_i: [/\bowned\b/i, /\bled\b/i, /\bdrove\b/i, /\bspearheaded\b/i, /\borchestrated\b/i, /\bchampioned\b/i, /\bheaded\b/i],
improvementVerbs: [/\bimprov(?:ed|ing|ement)\b/, /\breduc(?:ed|ing|tion)\b/, /\bincreas(?:ed|ing)\b/, /\boptimi[sz](?:ed|ing|ation)\b/, /\bcut\b/, /\bgrew\b/, /\bscal(?:ed|ing)\b/, /\bdoubled\b/, /\btripled\b/, /\bboost(?:ed|ing)\b/, /\bsav(?:ed|ing)\b/, /\baccelerat(?:ed|ing|ion)\b/, /\bstreamlin(?:ed|ing|e)\b/, /\bharden(?:ed|ing)\b/, /\blower(?:ed|ing)\b/, /\bshorten(?:ed|ing)\b/, /\brais(?:ed|ing)\b/, /\bexpand(?:ed|ing)\b/, /\bwiden(?:ed|ing)\b/, /\blift(?:ed|ing)\b/],
improvementVerbs_i: [/\bimprov(?:ed|ing|ement)\b/i, /\breduc(?:ed|ing|tion)\b/i, /\bincreas(?:ed|ing)\b/i, /\boptimi[sz](?:ed|ing|ation)\b/i, /\bcut\b/i, /\bgrew\b/i, /\bscal(?:ed|ing)\b/i, /\bdoubled\b/i, /\btripled\b/i, /\bboost(?:ed|ing)\b/i, /\bsav(?:ed|ing)\b/i, /\baccelerat(?:ed|ing|ion)\b/i, /\bstreamlin(?:ed|ing|e)\b/i, /\bharden(?:ed|ing)\b/i, /\blower(?:ed|ing)\b/i, /\bshorten(?:ed|ing)\b/i, /\brais(?:ed|ing)\b/i, /\bexpand(?:ed|ing)\b/i, /\bwiden(?:ed|ing)\b/i, /\blift(?:ed|ing)\b/i],
vaguePatterns: [/\bworked on\b/, /\bhelped (?:with|the)\b/, /\bparticipated in\b/, /\bwas involved in\b/, /\bassisted (?:with|in)\b/, /\btasked with\b/, /\bin charge of\b/, /\bworked with\b/, /\bcollaborated with\b/, /\bsome\s+(?:code|things|work|stuff|projects|cloud)\b/, /\bdid some\b/, /\bdid stuff\b/, /\bresponsible for\b/],
vaguePatterns_i: [/\bworked on\b/i, /\bhelped (?:with|the)\b/i, /\bparticipated in\b/i, /\bwas involved in\b/i, /\bassisted (?:with|in)\b/i, /\btasked with\b/i, /\bin charge of\b/i, /\bworked with\b/i, /\bcollaborated with\b/i, /\bsome\s+(?:code|things|work|stuff|projects|cloud)\b/i, /\bdid some\b/i, /\bdid stuff\b/i, /\bresponsible for\b/i],
};
for (const [name, text] of Object.entries(fixtures)) {
console.log(`\n${name}`);
for (const [group, patterns] of Object.entries(groups)) {
const c = patterns.filter((r) => r.test(text)).length;
console.log(`${group}: ${c}`);
}
}
JSRepository: TechImmigrants/cv-builder
Length of output: 15425
🏁 Script executed:
#!/bin/bash
set -euo pipefail
printf '\n== Evaluator excerpt ==\n'
sed -n '135,290p' packages/core/src/evaluator/index.ts
printf '\n== Shipped evidence tests ==\n'
sed -n '1,240p' packages/core/src/__tests__/shipped-evidence.test.ts
printf '\n== Synthetic fixtures ==\n'
for f in \
packages/core/src/__fixtures__/synthetic/strong-ai-pm.md \
packages/core/src/__fixtures__/synthetic/strong-backend.md \
packages/core/src/__fixtures__/synthetic/weak-generic.md
do
printf '\n--- %s ---\n' "$f"
sed -n '1,220p' "$f"
done
printf '\n== Regex count probe ==\n'
node <<'JS'
const fs = require('fs');
const path = require('path');
const fixtures = {
'strong-ai-pm.md': fs.readFileSync(path.join(process.cwd(), 'packages/core/src/__fixtures__/synthetic/strong-ai-pm.md'), 'utf8'),
'strong-backend.md': fs.readFileSync(path.join(process.cwd(), 'packages/core/src/__fixtures__/synthetic/strong-backend.md'), 'utf8'),
'weak-generic.md': fs.readFileSync(path.join(process.cwd(), 'packages/core/src/__fixtures__/synthetic/weak-generic.md'), 'utf8'),
};
const groups = {
strongVerbs: [/\bshipped\b/, /\blaunched\b/, /\breleased\b/, /\bdelivered\b/, /\bdeployed\b/, /\brolled\s+out\b/],
strongVerbs_i: [/\bshipped\b/i, /\blaunched\b/i, /\breleased\b/i, /\bdelivered\b/i, /\bdeployed\b/i, /\brolled\s+out\b/i],
mediumVerbs: [/\bbuilt\b/, /\bimplement(?:ed|s|ing)\b/, /\bmigrat(?:ed|ing|e|ion)\b/, /\bintegrat(?:ed|ing|e|ion)\b/, /\bautomat(?:ed|ing|e|ion)\b/, /\bredesign(?:ed|ing)\b/, /\bcreat(?:ed|ing|e|ion)\b/, /\bdevelop(?:ed|ing|ment)\b/, /\bdesign(?:ed|ing)\b/, /\bauthor(?:ed|ing)\b/, /\bcod(?:ed|ing)\b/, /\bprogram(?:med|ming)\b/, /\bengineer(?:ed|ing)\b/],
mediumVerbs_i: [/\bbuilt\b/i, /\bimplement(?:ed|s|ing)\b/i, /\bmigrat(?:ed|ing|e|ion)\b/i, /\bintegrat(?:ed|ing|e|ion)\b/i, /\bautomat(?:ed|ing|e|ion)\b/i, /\bredesign(?:ed|ing)\b/i, /\bcreat(?:ed|ing|e|ion)\b/i, /\bdevelop(?:ed|ing|ment)\b/i, /\bdesign(?:ed|ing)\b/i, /\bauthor(?:ed|ing)\b/i, /\bcod(?:ed|ing)\b/i, /\bprogram(?:med|ming)\b/i, /\bengineer(?:ed|ing)\b/i],
leadershipVerbs: [/\bowned\b/, /\bled\b/, /\bdrove\b/, /\bspearheaded\b/, /\borchestrated\b/, /\bchampioned\b/, /\bheaded\b/],
leadershipVerbs_i: [/\bowned\b/i, /\bled\b/i, /\bdrove\b/i, /\bspearheaded\b/i, /\borchestrated\b/i, /\bchampioned\b/i, /\bheaded\b/i],
improvementVerbs: [/\bimprov(?:ed|ing|ement)\b/, /\breduc(?:ed|ing|tion)\b/, /\bincreas(?:ed|ing)\b/, /\boptimi[sz](?:ed|ing|ation)\b/, /\bcut\b/, /\bgrew\b/, /\bscal(?:ed|ing)\b/, /\bdoubled\b/, /\btripled\b/, /\bboost(?:ed|ing)\b/, /\bsav(?:ed|ing)\b/, /\baccelerat(?:ed|ing|ion)\b/, /\bstreamlin(?:ed|ing|e)\b/, /\bharden(?:ed|ing)\b/, /\blower(?:ed|ing)\b/, /\bshorten(?:ed|ing)\b/, /\brais(?:ed|ing)\b/, /\bexpand(?:ed|ing)\b/, /\bwiden(?:ed|ing)\b/, /\blift(?:ed|ing)\b/],
improvementVerbs_i: [/\bimprov(?:ed|ing|ement)\b/i, /\breduc(?:ed|ing|tion)\b/i, /\bincreas(?:ed|ing)\b/i, /\boptimi[sz](?:ed|ing|ation)\b/i, /\bcut\b/i, /\bgrew\b/i, /\bscal(?:ed|ing)\b/i, /\bdoubled\b/i, /\btripled\b/i, /\bboost(?:ed|ing)\b/i, /\bsav(?:ed|ing)\b/i, /\baccelerat(?:ed|ing|ion)\b/i, /\bstreamlin(?:ed|ing|e)\b/i, /\bharden(?:ed|ing)\b/i, /\blower(?:ed|ing)\b/i, /\bshorten(?:ed|ing)\b/i, /\brais(?:ed|ing)\b/i, /\bexpand(?:ed|ing)\b/i, /\bwiden(?:ed|ing)\b/i, /\blift(?:ed|ing)\b/i],
vaguePatterns: [/\bworked on\b/, /\bhelped (?:with|the)\b/, /\bparticipated in\b/, /\bwas involved in\b/, /\bassisted (?:with|in)\b/, /\btasked with\b/, /\bin charge of\b/, /\bworked with\b/, /\bcollaborated with\b/, /\bsome\s+(?:code|things|work|stuff|projects|cloud)\b/, /\bdid some\b/, /\bdid stuff\b/, /\bresponsible for\b/],
vaguePatterns_i: [/\bworked on\b/i, /\bhelped (?:with|the)\b/i, /\bparticipated in\b/i, /\bwas involved in\b/i, /\bassisted (?:with|in)\b/i, /\btasked with\b/i, /\bin charge of\b/i, /\bworked with\b/i, /\bcollaborated with\b/i, /\bsome\s+(?:code|things|work|stuff|projects|cloud)\b/i, /\bdid some\b/i, /\bdid stuff\b/i, /\bresponsible for\b/i],
};
for (const [name, text] of Object.entries(fixtures)) {
console.log(`\n${name}`);
for (const [group, patterns] of Object.entries(groups)) {
console.log(`${group}: ${patterns.filter((r) => r.test(text)).length}`);
}
}
JSRepository: TechImmigrants/cv-builder
Length of output: 15425
🏁 Script executed:
#!/bin/bash
set -euo pipefail
sed -n '135,290p' packages/core/src/evaluator/index.ts
printf '\n====\n'
sed -n '1,240p' packages/core/src/__tests__/shipped-evidence.test.ts
printf '\n==== strong-ai-pm.md ====\n'
sed -n '1,220p' packages/core/src/__fixtures__/synthetic/strong-ai-pm.md
printf '\n==== strong-backend.md ====\n'
sed -n '1,220p' packages/core/src/__fixtures__/synthetic/strong-backend.md
printf '\n==== weak-generic.md ====\n'
sed -n '1,220p' packages/core/src/__fixtures__/synthetic/weak-generic.mdRepository: TechImmigrants/cv-builder
Length of output: 14609
Make the verb/vague regexes case-insensitive
strongVerbs, mediumVerbs, leadershipVerbs, improvementVerbs, and vaguePatterns miss the common capitalized resume-bullet form (Owned, Led, Built, Worked on, Responsible for). Add /i so the scorer counts those signals consistently; otherwise it undercounts leadership and vague-language matches on realistic CVs.
🤖 Prompt for AI Agents
Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.
In `@packages/core/src/evaluator/index.ts` around lines 142 - 219, The verb and
vague-language matchers in the evaluator are missing capitalized resume-bullet
forms, so the scorer undercounts common signals. Update the regexes in the
strongVerbs, mediumVerbs, leadershipVerbs, improvementVerbs, and vaguePatterns
arrays to be case-insensitive so matches like Owned, Led, Built, Worked on, and
Responsible for are counted consistently.
Source: Path instructions
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Problem
scoreShippedEvidenceinpackages/core/src/evaluator/index.tswascapped at a 7-pattern keyword list:
Three of those (
built and released,in production,live users,released to) are exact multi-word phrases that almost no real CVcontains. Strong CVs that describe real shipped work with verbs like
owned, led, drove, built, migrated, designed —
but never happen to use the literal word "shipped" or "launched" — got
floored to 1/5. A strong backend CV using only "Migrated from
Postgres 13 → 16", "Built the routing engine", "Designed the failover"
scored 1/5 because none of the four single-token regexes fired.
Root cause
Math.min(5, max(1, distinctMatches + 1)), so missingall 4 single-token verbs floored at 1 even if the rest of the CV was
filled with shipped work.
concrete shipped objects (API, pipeline, microservice, funnel,
guardrail, …), quantified adoption signals ("180k MAU", "used by
600k patients/month"), outcome metrics (latency, retention, NPS,
error rate, p99, …), or production / open-source signals.
What changed
Replace the keyword matcher in
packages/core/src/evaluator/index.tswith a multi-signal additiveheuristic, clamped to [1, 5]. Signals are grouped into positive
categories and one negative category.
Positive signals (each category adds a small amount at one or two
thresholds so weak CVs aren't accidentally inflated):
deployed, rolled out.
integrated, automated, redesigned, created, developed, designed,
authored, coded, programmed, engineered. Each pattern requires the
past/present-participle form so job titles like "software engineer"
don't match.
championed, headed.
grew, scaled, doubled, tripled, boosted, saved, accelerated,
streamlined, hardened, lowered, shortened, raised, expanded, widened,
lifted.
integration, system, service, platform, module, experiment, release,
rollout, MVP, pipeline, library, app, repo, microservice, funnel,
guardrail, assistant, migration, upgrade, redesign, prototype, squad,
… Generic terms (tools, frontend, backend, jobs) are intentionally
excluded.
customers / teams / MAU / DAU / req/s", "N k/m users" scaled
numbers.
uptime, adoption, churn, engagement, funnel, throughput, MAU, DAU,
WAU, NPS, hallucination, p99/p95/p90, SLO, SLA, error rate, CSAT.
Generic words like "performance", "cost", "usage" are excluded
so "Improved performance" doesn't count as evidence.
publicly available, open source, deployed to, shipped to, launched
to, used by, adopted by, customer use, public release.
Negative signals:
was involved in, assisted (with|in), tasked with, in charge of,
worked with, collaborated with, "some (code|things|work|…)", did
some, did stuff, responsible for. 3+ vague patterns subtract
1; 5+ subtract 0.5 more.
A small
countMatcheshelper is added at the bottom of the file.Test cases added
packages/core/src/__tests__/shipped-evidence.test.tsplus threesynthetic fixtures under
packages/core/src/__fixtures__/synthetic/:strong-ai-pm.md— Senior AI PM, "Shipped an LLM assistant to180k MAU", "Owned the prompt evaluation pipeline", "Built
guardrails", "shipped 14 experiments", etc.
strong-backend.md— Staff Backend Engineer, "Owned the ledgerservice rewrite in Go; p99 latency 1.4s → 220ms", "Led the
migration from Postgres 13 → 16", "Built Kafka-backed event
sourcing layer", etc.
weak-generic.md— vague claims ("Hardworking and passionatesoftware engineer", "Worked on various projects", "Responsible
for backend things", "Wrote some code").
Assertions (8 cases):
sample-cv-strong.mdstays at 4-5/5 (was 3/5; allowedto rise, not fall).
sample-cv-weak.mdstays at 1-2/5.(Quantified Impact 5, ATS 5, Public Proof 3) — no regression outside
Shipped Evidence.
Before / after scores on synthetic CVs
Other dimensions (Quantified Impact, Tooling Visibility, ATS
Compatibility, Keyword Match, Public Proof) are byte-identical
before/after on every fixture above. Only
scoreShippedEvidencewas modified; the other scorers are untouched.
Validation
pnpm lint— passes (one pre-existingbiome.jsondeprecationinfo, unrelated).
pnpm test --force— 57 tests pass across 8 packages (29 in@cv-builder/coreincluding the 8 new ones, 21 in@cv-builder/eval).pnpm build --force— all 6 build targets succeed fresh.pnpm eval— golden fixtures still pass.Privacy confirmation
tmp/sahar-cv.mdwas not read, modified,committed, or used as a fixture.
synth-strong-ai-pm.md,synth-strong-backend.md,synth-weak-generic.md) were used.These live under
packages/core/src/__fixtures__/synthetic/andare public-safe, fictional profiles.
companies, and metrics are invented.
PR #37 (
feat/cli-color-output) is untouched.Summary by CodeRabbit
New Features
Bug Fixes