ARCHITECTURE.md

Architecture

Processing stages

1. Input normalization maps common spreadsheet headings to a canonical item schema.
2. URL discovery starts with user-supplied product URLs and can add public search results.
3. Policy checks apply allowlists, denylists, robots.txt, and per-domain rate limits.
4. Fetching uses requests first and optionally starts a headless Chrome session for JavaScript-rendered pages.
5. Candidate extraction reads JSON-LD, product metadata, and price-like DOM elements.
6. Candidate validation uses a local score, an optional Random Forest, or a configured cloud provider.
7. Post-processing marks extreme prices using a median-absolute-deviation rule and selects the highest-confidence non-outlier per item.
8. Reporting writes input snapshots, URL-level results, selected prices, summary metrics, and safe run metadata.

Trust boundaries

• Input workbooks are untrusted data.
• Downloaded HTML is untrusted data and is parsed as text; it is never executed by the application.
• Browser pages can execute their own JavaScript inside the isolated Selenium browser process.
• API credentials and proxy passwords are secrets and are never written to reports or logs.
• Cloud providers receive only the requested item, candidate list, URL, and a bounded page-text excerpt.

Extension points

Implement CandidateProvider to add another candidate validator. Keep provider modules optional so the local pipeline remains usable without cloud services.