OpenPrice AI Scraper

A company-neutral, open-source desktop and command-line application for extracting candidate product prices from public web pages and exporting an auditable Excel report.

The repository contains no customer spreadsheet, supplier list, operational log, private technical manual, browser installer, web-driver binary, API key, proxy credential, or customer branding. It starts from a blank input template and is designed for reuse by any organization that is authorized to collect the target information.

Responsible-use notice: only collect pages that you are authorized to access. Comply with applicable law, website terms, robots.txt, rate limits, privacy requirements, and contractual restrictions. This software does not include features to bypass authentication, CAPTCHAs, paywalls, or technical access controls.

What it does

• imports neutral .xlsx, .xlsm, or .csv product lists;
• accepts direct product URLs or performs optional public URL discovery;
• uses requests first and optional Selenium/Chrome fallback for rendered pages;
• supports an optional authenticated HTTP proxy, including Bright Data credentials;
• extracts candidates from JSON-LD, product metadata, and visible DOM elements;
• validates candidates with a local model, OpenAI, Anthropic, or Firecrawl;
• keeps secrets in the operating-system keychain or environment variables;
• flags extreme candidate prices with a robust MAD rule;
• exports URL-level evidence, selected prices, input snapshots, summary metrics, and safe run metadata;
• includes a local demonstration site, tests, a public-repository audit, and an MIT license.

Repository structure

ProcureLens-AI/
├── assets/                  # Neutral logo and architecture diagram
├── config/                  # Empty allowlist/denylist examples
├── docs/                    # Architecture, mathematics, providers, privacy, and legal use
├── examples/                # Offline-safe end-to-end demonstration
├── scripts/                 # Environment check, demo server, reset, and audit tools
├── src/openprice_scraper/   # Application source code
│   └── providers/           # Local, OpenAI, Anthropic, and Firecrawl integrations
├── templates/               # Blank XLSX and CSV input templates
├── tests/                   # Unit tests and synthetic HTML fixture
├── LICENSE                  # MIT License
├── pyproject.toml           # Package and optional dependency groups
└── README.md

Requirements

Minimum

• Python 3.11 or newer;
• internet access for live scraping;
• permission to access the selected pages.

Optional components

• Chrome or Chromium for Selenium fallback;
• an OS keychain backend for encrypted-at-rest credential storage;
• an OpenAI, Anthropic, or Firecrawl account for the corresponding provider;
• a Bright Data or other HTTP proxy account when proxy routing is required;
• scikit-learn for training the optional local Random Forest;
• PySide6 for the desktop interface.

Selenium 4 includes Selenium Manager, which automatically resolves a compatible driver when no driver is supplied. The repository intentionally does not bundle ChromeDriver, GeckoDriver, browsers, installers, or executables.

Installation

macOS

xcode-select --install 2>/dev/null || true
brew install python@3.12

cd ~/Downloads/ProcureLens-AI
python3.12 -m venv .venv
source .venv/bin/activate
python -m pip install --upgrade pip
python -m pip install -e '.[full]'

Install Chrome only when browser fallback is required:

brew install --cask google-chrome

Windows PowerShell

Install Python 3.11+ from Python.org or winget, then:

cd $HOME\Downloads\ProcureLens-AI
py -3.12 -m venv .venv
.\.venv\Scripts\Activate.ps1
python -m pip install --upgrade pip
python -m pip install -e ".[full]"

Install Chrome only when browser fallback is required:

winget install Google.Chrome

Ubuntu / Debian

sudo apt update
sudo apt install -y python3 python3-venv python3-pip

cd ~/Downloads/ProcureLens-AI
python3 -m venv .venv
source .venv/bin/activate
python -m pip install --upgrade pip
python -m pip install -e '.[full]'

Optional Chrome/Chromium package names vary by distribution. Selenium Manager can manage the driver, but a browser must still be available.

Minimal command-line installation

This omits the desktop UI and cloud/ML providers:

python -m pip install -e '.[secure,browser]'

Feature-specific installation

python -m pip install -e '.[gui]'        # desktop interface
python -m pip install -e '.[openai]'     # OpenAI provider
python -m pip install -e '.[anthropic]'  # Anthropic provider
python -m pip install -e '.[firecrawl]'  # Firecrawl rendering
python -m pip install -e '.[ml]'         # Random Forest training

Check the environment:

python scripts/check_environment.py

First-run configuration

Run the setup wizard once:

openprice setup

It configures:

• local or cloud candidate provider;
• default market and currency;
• number of parallel workers and URLs per item;
• per-domain delay and browser fallback;
• robots.txt behavior;
• optional proxy host, port, username, and password;
• output directory.

API keys and proxy passwords are stored with keyring in the operating-system keychain. They are not written to config.json, workbooks, logs, or source files.

Environment variables can be used instead:

cp .env.example .env
# Load variables with your preferred shell or secret manager.

Recognized secret variables:

OPENAI_API_KEY
ANTHROPIC_API_KEY
FIRECRAWL_API_KEY
BRIGHTDATA_USERNAME
BRIGHTDATA_PASSWORD
OPENPRICE_PROXY_URL

Never commit a populated .env file.

Prepare the product file

Use either:

• templates/parts_template.xlsx; or
• templates/parts_template.csv.

The workbook is intentionally blank. The main sheet is named Items.

Column	Required	Example meaning
item_id	Recommended	Your neutral internal row identifier
supplier_reference	Recommended	Manufacturer or supplier part number
manufacturer	Optional	Brand/manufacturer
description	Recommended	Product description
country	Optional	Two-letter market code such as US
currency	Optional	ISO code such as USD
target_url	Optional	One or more semicolon-separated product URLs
enabled	Optional	true or false
notes	Optional	Non-secret note

At least one of description, supplier_reference, or target_url must be populated. Direct product URLs are more reproducible and create less search traffic.

Validate before scraping:

openprice inspect templates/parts_template.xlsx

More details: Input format.

Run from the command line

openprice scrape /path/to/parts.xlsx

Specify the output and provider:

openprice scrape parts.xlsx \
  --output output/prices.xlsx \
  --provider local \
  --workers 3 \
  --max-urls 5

Cloud provider example after openprice setup:

openprice scrape parts.xlsx --provider openai

Start the desktop application

openprice gui

The Run tab imports and previews the product file. The Settings tab stores provider keys and non-secret settings. The report location is selected before the run begins.

Offline end-to-end demonstration

No external website or API key is needed.

Terminal 1:

source .venv/bin/activate
python scripts/run_demo_server.py

Terminal 2:

source .venv/bin/activate
openprice scrape examples/demo_items.csv \
  --provider local \
  --output output/demo-results.xlsx

The local page contains a synthetic product and price. Open output/demo-results.xlsx to inspect the selected price and extraction evidence.

Output workbook

The generated workbook contains:

• Results — one row per evaluated URL;
• Best Prices — the highest-confidence non-outlier candidate for each item;
• Input Snapshot — canonical copy of input rows;
• Summary — item counts, observations, selected prices, and outlier count;
• Run Metadata — safe settings only, with secret-like keys removed.

The report is evidence for review, not a contractual quotation. Taxes, shipping, minimum order quantities, regional availability, and negotiated pricing may not be visible on the page.

Local AI model

The default provider stays on the user's machine. It creates a feature vector for each amount and computes a logistic confidence:

$$ z_i=b+\mathbf{w}^{\mathsf T}\mathbf{x}_i, \qquad P_i=\frac{1}{1+e^{-z_i}}. $$

Features include structured-data source, currency agreement, price/list-price attributes, product-token overlap, sale terms, logarithmic price magnitude, and context length.

An optional Random Forest replaces the heuristic after training:

$$ \widehat P(y=1\mid\mathbf{x})= \frac{1}{T}\sum_{t=1}^{T}P_t(y=1\mid\mathbf{x}). $$

Train with your own legally collected labels:

openprice train candidate_labels.jsonl

See Mathematical model and Local model training.

Robust price cleaning

For several observations of the same item and currency, the report calculates the median m and median absolute deviation:

$$ \text{MAD}=\text{median}(|p_i-m|). $$

A candidate is flagged when the modified z-score exceeds 3.5:

$$ |z_i^*|= \left|0.6745\frac{p_i-m}{\text{MAD}}\right|>3.5. $$

This is a review aid. It can incorrectly reject valid prices in small or multimodal markets.

Providers

Provider	Page fetching / validation	Page text leaves the computer?	API key
local	Requests/Selenium + local ranking	No	No
openai	Local extraction + OpenAI candidate validation	Yes, bounded excerpt	Yes
anthropic	Local extraction + Claude candidate validation	Yes, bounded excerpt	Yes
firecrawl	Firecrawl rendering + local ranking	Yes, target URL	Optional/current plan dependent

Read Provider details before enabling cloud processing.

Proxy and Bright Data

Proxy routing is optional. Enter the exact host, port, username, and password issued by the provider. Credentials are stored in the OS keychain. The requests fetcher supports authenticated proxy URLs.

Authenticated Chrome proxies require a local gateway or provider-specific browser integration. The application does not expose proxy credentials in browser command-line arguments.

See Proxy and Bright Data configuration.

Testing

python -m unittest discover -s tests -v
python scripts/audit_public_repo.py

With developer tools:

python -m pip install -e '.[dev]'
ruff check .

The tests use synthetic pages and temporary workbooks. They do not access the public internet.

Security and sanitization guarantees

The public repository is designed to reject common accidental disclosures:

• no hard-coded API keys or fallback credentials;
• no absolute personal user paths;
• no input/output workbooks except the blank template;
• no logs, IP histories, cookies, supplier/domain datasets, or scraped HTML archives;
• no customer logo, icon, trademark, or internal documentation;
• no browser installer, driver, proxy-manager installer, .exe, .dll, or .lnk;
• no serialized private model or training dataset;
• secret-like metadata is excluded from output reports;
• scripts/audit_public_repo.py checks common secret and binary patterns in CI.

If any credential was previously stored in another source tree or archive, revoke and rotate it before publishing that history.

Reset local data

python scripts/reset_local_data.py --yes

This deletes local settings, model files, generated outputs, and keychain entries associated with OpenPrice. It does not delete repository files or unrelated browser data.

Known limitations

• Website layouts and structured data change over time.
• Search discovery can miss the correct product or return an accessory.
• Browser fallback is slower and more resource-intensive than direct HTTP.
• Cloud models can make errors and incur costs.
• Currency is detected and matched against the requested currency, but amounts are not converted between currencies (e.g. EUR to USD).
• A public page price may be stale, personalized, excluding taxes, or conditional on quantity.
• The default local model uses fixed, hand-tuned weights and performs well in practice; training the optional Random Forest on your own labeled data gives a formally validated, domain-specific alternative.

Documentation

• Architecture
• Mathematics and machine learning
• Input format
• Providers
• Local model training
• Proxy and Bright Data
• Legal and ethical use
• Privacy
• Security

License

Released under the MIT License.

Author

Dev Kumar.