feat: add structured archive export

[chboishabba]

Summary

This PR adds a structured archive layer for Perplexity exports before Markdown/vector indexing, and improves long-thread extraction beyond the initial captured API page.

The exporter now writes itir.perplexity.thread.v1 JSON with normalized messages, stable source thread/message IDs, thread metadata, and captured API provenance. Markdown export remains available as an optional sidecar, and the existing vector/RAG flow can still be enabled when Markdown sidecars are present.

Why

The current Markdown-first flow is useful for reading and local vector search, but it makes dedupe, reimport, pagination validation, and downstream archive integrations harder than they need to be. A canonical thread/message export gives SQLite/MyChatArchive/ITIR-style tools a stable source of truth, while Markdown and vector indexes can be regenerated from that canonical layer.

A major practical issue is long Perplexity threads. The first browser-captured /rest/thread/<id> response can contain only the first page of entries. Without structured IDs and pagination checks, it is hard to know whether a run captured the whole thread, only page one, or duplicated page one repeatedly.

Pagination / long-thread behavior

This PR changes thread extraction so it does not stop at the first captured thread API response when Perplexity reports more pages:

• captures the exact /rest/thread/<thread-id> response for the conversation being exported
• follows next_cursor from page context with authenticated browser cookies
• appends additional entries when Perplexity returns genuinely new pages
• tracks entry identities across pages
• stops pagination if Perplexity replays only duplicate entries, instead of inflating the archive with repeated page-one content
• keeps the raw API response/entries in the structured export so downstream tools can audit what was captured

That last point is important: this does not pretend to bypass Perplexity/Cloudflare or guarantee all private webapp pagination works forever. It makes successful pagination useful, and failed/replayed pagination detectable and safe.

For cases where Perplexity's private API still only yields the first page but the UI download contains more content, this PR also adds a downloaded Markdown recovery path. npm run bundle:perplexity-downloads converts downloaded Markdown chunks into the same structured JSON shape so the recovered turns can attach to the same canonical thread downstream.

Changes

• Adds structured JSON export by default via EXPORT_STRUCTURED_JSON=true.
• Makes Markdown export optional via EXPORT_MARKDOWN=true.
• Adds normalized user/assistant message extraction from Perplexity API entries.
• Adds authenticated cursor pagination for thread detail responses.
• Adds duplicate-page detection so replayed first pages do not create fake extra messages.
• Adds bundle:perplexity-downloads for converting downloaded Markdown chunks into the same structured JSON shape.
• Defaults headful browser mode because Cloudflare/Turnstile makes headless unreliable for this flow.
• Adds unit coverage for config, structured file writes, pagination, duplicate-page replay, filename truncation, and downloaded Markdown bundling.

Validation

• npm run type-check
• npm run test:unit (25 tests passed)

[simwai]

Hello, thanks for the PR. 🙌

I like some of the ideas, the ITIR and structured JSON export thing is something I don't really understand why it is needed or what is the purpose of it. Maybe you wanna give more insights there.

[chboishabba]

Hi mate, thanks for making the repo.

Firstly I had a few issues around sign-in detection (I forget exactly which now - I think around session detection/redirects), and then I also have some stupidly long threads which the scraper reported as having pulled completely, when this was not the case.

Regarding the JSON export, I figured better to just provide a generic interface, however, the canonical use case within my repos is to store within SQLite (see https://github.com/1ch1n/mychatarchive).

I don't think I PR'd any ITIR integration per-se, more that, again, canonically ITIR operates over that db (arbitrary text also fine, dedupe is priority re SQL)...

Please let me know if this helps :)

[simwai]

@chboishabba Heyo, I added a new folder where you can add custom file exporters. Maybe that could work out to solve some of your problems.

[chboishabba]

I'll have a look. I started on adding artifact export for links, pictures, and code...

The main thing is being able to reliably export massive threads. Perplexity's pagination will happily omit loading turns if it can avoid it... I modified the loader script to handle this better...

I think I also added pdf/md export

[simwai]

@chboishabba Is the unreliable fetching of long threads still unreliable on the latest commit?