Skip to content

Improve translation quality: enforce AI_MUST_REPLACE CI failure, extend banned-pattern detection, split translation dictionary#1660

Merged
pethers merged 4 commits intomainfrom
copilot/improve-translation-quality
Apr 10, 2026
Merged

Improve translation quality: enforce AI_MUST_REPLACE CI failure, extend banned-pattern detection, split translation dictionary#1660
pethers merged 4 commits intomainfrom
copilot/improve-translation-quality

Conversation

Copy link
Copy Markdown
Contributor

Copilot AI commented Apr 10, 2026

  • Change 1: validate-news-translations.ts — AI_MUST_REPLACE marker detection + CI failure (exit 1)
  • Change 2: check-banned-patterns.ts — detect AI_MUST_REPLACE markers in HTML comments
  • Change 3: .github/workflows/news-translate.md — AI_MUST_REPLACE handling instructions
  • Change 4: scripts/translation-dictionary.ts — split into 4 domain-specific files
  • Tests: 3 new tests for AI_MUST_REPLACE detection
  • All 4,100 tests pass
  • Review fix 1: checkFileForAIMustReplaceMarkers() now throws on read errors with { cause } instead of silently returning null
  • Review fix 2: Files with AI_MUST_REPLACE markers excluded from totalPassed count — no longer shown with green ✓
  • Review fix 3: Prompt item 6 now consistently says "translated article" (not "source article")
  • Lint fix: Removed unused output assignment in test try blocks (no-useless-assignment)

@github-actions github-actions bot added the size-xs Extra small change (< 10 lines) label Apr 10, 2026
@github-actions
Copy link
Copy Markdown
Contributor

🏷️ Automatic Labeling Summary

This PR has been automatically labeled based on the files changed and PR metadata.

Applied Labels: size-xs

Label Categories

  • 🗳️ Content: news, dashboard, visualization, intelligence
  • 💻 Technology: html-css, javascript, workflow, security
  • 📊 Data: cia-data, riksdag-data, data-pipeline, schema
  • 🌍 I18n: i18n, translation, rtl
  • 🔒 ISMS: isms, iso-27001, nist-csf, cis-controls
  • 🏗️ Infrastructure: ci-cd, deployment, performance, monitoring
  • 🔄 Quality: testing, accessibility, documentation, refactor
  • 🤖 AI: agent, skill, agentic-workflow

For more information, see .github/labeler.yml.

@github-actions
Copy link
Copy Markdown
Contributor

🔍 Lighthouse Performance Audit

Category Score Status
Performance 85/100 🟡
Accessibility 95/100 🟢
Best Practices 90/100 🟢
SEO 95/100 🟢

📥 Download full Lighthouse report

Budget Compliance: Performance budgets enforced via budget.json

…dictionary, workflow update

Agent-Logs-Url: https://github.com/Hack23/riksdagsmonitor/sessions/b182ac4a-002b-47f8-9b70-bf0a7c41f4b8

Co-authored-by: pethers <1726836+pethers@users.noreply.github.com>
@github-actions github-actions bot added documentation Documentation updates workflow GitHub Actions workflows ci-cd CI/CD pipeline changes testing Test coverage refactor Code refactoring news News articles and content generation agentic-workflow Agentic workflow changes size-xl Extra large change (> 1000 lines) labels Apr 10, 2026
@github-actions
Copy link
Copy Markdown
Contributor

🔍 Lighthouse Performance Audit

Category Score Status
Performance 85/100 🟡
Accessibility 95/100 🟢
Best Practices 90/100 🟢
SEO 95/100 🟢

📥 Download full Lighthouse report

Budget Compliance: Performance budgets enforced via budget.json

Copilot AI changed the title [WIP] Fix AI_MUST_REPLACE leakage and enhance validation pipeline Improve translation quality: enforce AI_MUST_REPLACE CI failure, extend banned-pattern detection, split translation dictionary Apr 10, 2026
Copilot AI requested a review from pethers April 10, 2026 09:34
@pethers pethers marked this pull request as ready for review April 10, 2026 09:57
Copilot AI review requested due to automatic review settings April 10, 2026 09:57
Copy link
Copy Markdown
Contributor

Copilot AI left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Copilot encountered an error and was unable to review this pull request. You can try again by re-requesting a review.

Copy link
Copy Markdown
Contributor

Copilot AI left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Pull request overview

Copilot reviewed 8 out of 8 changed files in this pull request and generated 3 comments.

Comment on lines +470 to +471
} catch {
return null;
Copy link

Copilot AI Apr 10, 2026

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

checkFileForAIMustReplaceMarkers() swallows read/parse errors (catch { return null; }), which can silently skip AI_MUST_REPLACE validation if a file is unreadable. This is inconsistent with checkFileForUntranslatedContent() (which surfaces errors) and can let CI pass when it should fail; treat read errors as hard failures and report them in the summary/output.

Suggested change
} catch {
return null;
} catch (error: unknown) {
const message = error instanceof Error ? error.message : String(error);
throw new Error(`Failed to validate AI_MUST_REPLACE markers in ${filepath}: ${message}`);

Copilot uses AI. Check for mistakes.
Comment on lines +553 to +563
// AI_MUST_REPLACE marker check — unresolved placeholders in HTML comments
// are a hard failure: they indicate the AI did not replace template content.
const aiMarkerRecord = checkFileForAIMustReplaceMarkers(filepath);
if (aiMarkerRecord) {
aiMarkerFiles.push(aiMarkerRecord);
totalAIMarkers += aiMarkerRecord.markerCount;
const sampleStr = aiMarkerRecord.samples.length > 0
? ` [${aiMarkerRecord.samples.join(', ')}]`
: '';
console.log(`${colors.red}✗ AI_MUST_REPLACE: ${filename} — ${aiMarkerRecord.markerCount} unresolved marker(s)${sampleStr}${colors.reset}`);
}
Copy link

Copilot AI Apr 10, 2026

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Files containing AI_MUST_REPLACE markers are still counted as “passed” (totalPassed++) and printed with a green ✓ later in the loop, even though they trigger a hard failure at the end. This makes the per-file status and the “Fully translated” summary inaccurate; consider treating AI marker presence as a per-file failure (or at least excluding these files from totalPassed and reporting them separately).

Copilot uses AI. Check for mistakes.
Comment on lines 654 to +670
3. **Raw Swedish API text** — Interpellation excerpts, proposition summaries that come from the Riksdag API are often pasted as-is. You MUST translate these to the target language or summarize them.
4. **English boilerplate phrases** — Remove or translate phrases like "Read the full proposition", "Live intelligence platform for Swedish Parliament monitoring"
5. **Section headings** that were not covered by CONTENT_LABELS (e.g., specific policy domain names used as h3/h4 headings)
6. **🚨 AI_MUST_REPLACE HTML comments** — SCAN every HTML comment in the source article. If any contains `AI_MUST_REPLACE`, you MUST generate replacement content in the target language. See critical section below.

#### 🚨 CRITICAL: AI_MUST_REPLACE Comment Handling

The content generator embeds placeholder HTML comments in the form:
```html
<!-- AI_MUST_REPLACE: marker_name — DATA: hint text. Write specific analysis here. Output MUST be in the article's language. -->
```

**These comments MUST be replaced with real content before publication.** Leaving them in the article is a hard CI failure (exit 1). The translation workflow MUST:

1. **SCAN every HTML comment** in the translated article for `AI_MUST_REPLACE`
2. **For each marker found**, read the `DATA:` hint inside the comment to understand what content to generate
3. **Replace the entire `<!-- AI_MUST_REPLACE ... -->` comment** with genuine, specific analysis written in the **target language** (not English)
Copy link

Copilot AI Apr 10, 2026

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

The new checklist item says to scan every HTML comment in the source article for AI_MUST_REPLACE, but the critical section below says to scan the translated article. Please make this consistent (ideally require scanning the translated output, since that’s what CI validates) to avoid operator confusion.

Copilot uses AI. Check for mistakes.
@pethers
Copy link
Copy Markdown
Member

pethers commented Apr 10, 2026

@copilot apply changes based on the comments in this thread and Quality Checks / typescript-lint (pull_request)
Quality Checks / typescript-lint (pull_request)Failing after 39s

@github-actions
Copy link
Copy Markdown
Contributor

🔍 Lighthouse Performance Audit

Category Score Status
Performance 85/100 🟡
Accessibility 95/100 🟢
Best Practices 90/100 🟢
SEO 95/100 🟢

📥 Download full Lighthouse report

Budget Compliance: Performance budgets enforced via budget.json

…passed count, fix lint, fix prompt consistency

Agent-Logs-Url: https://github.com/Hack23/riksdagsmonitor/sessions/ad9025e1-7491-4238-a5bf-98f4704dd8a6

Co-authored-by: pethers <1726836+pethers@users.noreply.github.com>
Copy link
Copy Markdown
Contributor Author

Copilot AI commented Apr 10, 2026

@copilot apply changes based on the comments in this thread and [Quality Checks / typescript-lint (p...

Applied all review comments and fixed the lint failure in commit 76868ec:

  1. Lint fix (no-useless-assignment): Removed the unused output assignment in the try block of both test cases — the value is only captured from the catch path.
  2. Review: error handling: checkFileForAIMustReplaceMarkers() now throws on read errors with { cause } instead of silently returning null, consistent with checkFileForUntranslatedContent().
  3. Review: passed count: Files with AI_MUST_REPLACE markers are no longer counted in totalPassed or shown with green ✓ — they're reported only as marker failures.
  4. Review: prompt consistency: Item 6 in news-translate.md now says "translated article" (not "source article"), matching the critical section below it.

@github-actions
Copy link
Copy Markdown
Contributor

🔍 Lighthouse Performance Audit

Category Score Status
Performance 85/100 🟡
Accessibility 95/100 🟢
Best Practices 90/100 🟢
SEO 95/100 🟢

📥 Download full Lighthouse report

Budget Compliance: Performance budgets enforced via budget.json

Copy link
Copy Markdown
Contributor

Copilot AI left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Pull request overview

Copilot reviewed 8 out of 8 changed files in this pull request and generated no new comments.

@pethers pethers merged commit 3b4c57f into main Apr 10, 2026
21 checks passed
@pethers pethers deleted the copilot/improve-translation-quality branch April 10, 2026 12:05
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

agentic-workflow Agentic workflow changes ci-cd CI/CD pipeline changes documentation Documentation updates news News articles and content generation refactor Code refactoring size-xl Extra large change (> 1000 lines) size-xs Extra small change (< 10 lines) testing Test coverage workflow GitHub Actions workflows

Projects

None yet

Development

Successfully merging this pull request may close these issues.

Improve Translation Quality — Eliminate AI_MUST_REPLACE Leakage & Strengthen Validation Pipeline

3 participants