fix(scraper): support hash-routed spa crawling (#379)

[arabold]

Summary

• add explicit preserveHashes support across CLI, MCP, web UI, pipeline, refresh, and stored scraper options for hash-routed SPA docs sites
• preserve hash fragments in crawl identity and page storage while keeping normal anchor-link collapsing as the default behavior
• upgrade fetch jobs to playwright when hash preservation is enabled, and cover refresh reuse/override plus redirect/interception edge cases in tests

Validation

• npm run typecheck
• npm run lint
• npx vitest run src/tools/ScrapeTool.test.ts src/cli/commands/scrape.test.ts src/cli/commands/refresh.test.ts src/pipeline/PipelineManager.test.ts src/scraper/strategies/BaseScraperStrategy.test.ts src/scraper/strategies/WebScraperStrategy.test.ts src/scraper/middleware/HtmlPlaywrightMiddleware.test.ts src/mcp/mcpServer.test.ts
• npx vitest run src/store/DocumentStore.test.ts src/store/DocumentManagementService.test.ts src/tools/ListLibrariesTool.test.ts

[Copilot]

Pull request overview

Drafts an OpenSpec change plan to add opt-in support for preserving hash fragments during web crawling so hash-routed SPA documentation sites can be discovered and indexed correctly (issue #379).

Changes:

• Adds OpenSpec proposal/design/spec for a --preserve-hashes / preserveHashes option.
• Defines implementation tasks covering CLI/config wiring, URL normalization changes, Playwright interception adjustments, and tests.
• Introduces OpenSpec metadata for the change package.

Reviewed changes

Copilot reviewed 5 out of 5 changed files in this pull request and generated 3 comments.

Show a summary per file

File	Description
openspec/changes/support-hash-routed-spas/.openspec.yaml	Registers the change package with schema + creation date.
openspec/changes/support-hash-routed-spas/design.md	Documents goals, decisions, and risks for hash-route preservation and Playwright interception behavior.
openspec/changes/support-hash-routed-spas/proposal.md	Explains motivation, high-level approach, and impact of the new opt-in flag.
openspec/changes/support-hash-routed-spas/specs/hash-routed-spa-support/spec.md	Adds requirements/scenarios for preserving hashes, enforcing Playwright, and interception matching.
openspec/changes/support-hash-routed-spas/tasks.md	Breaks the implementation into concrete steps across config/CLI, core scraper, middleware, docs, and tests.

---

💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.

[Copilot]

Pull request overview

Copilot reviewed 41 out of 41 changed files in this pull request and generated 3 comments.

---

💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.