diff --git a/.gitignore b/.gitignore index 957d46c..20badc7 100644 --- a/.gitignore +++ b/.gitignore @@ -14,7 +14,6 @@ venv/ .pytest_cache/ .mypy_cache/ uv.lock -.firebase/ .planning/ .claude/ npm/.mcpregistry_* diff --git a/CHANGELOG.md b/CHANGELOG.md index b1c2fb8..686fa7a 100644 --- a/CHANGELOG.md +++ b/CHANGELOG.md @@ -1,5 +1,19 @@ # Changelog +## v0.15.0 — Remove NeuroStack Cloud (2026-06-03) + +### Breaking changes + +- **Removed NeuroStack Cloud.** The hosted service has been discontinued. All cloud client commands (`neurostack cloud *`), the `--cloud` flag, cloud sync, and the `neurostack.cloud` package have been removed. NeuroStack is now a purely local tool: it indexes a local Markdown vault into local SQLite, uses local Ollama for embeddings and summaries, and runs the MCP server and OpenAI-compatible API on your own machine. No data leaves your machine unless you configure a third-party LLM provider. + + **Migration**: run `neurostack init` (or `--mode lite|full`) for local indexing. Connect AI clients to the local MCP server with `neurostack serve`. There is no hosted endpoint to connect to. + +### Removed + +- `neurostack cloud login/push/pull/sync/consent/install-hooks/auto-sync` CLI commands and the `--cloud` init flag +- `cloud` mode from `manifest.json` and the hosted remote endpoint from `server.json` +- `DPA.md` (Data Processing Agreement) and `docs/api-contract-v2.md` (hosted sync API contract) + ## v0.13.0 — Remove vault_capture (2026-05-05) ### Breaking changes diff --git a/CLAUDE.md b/CLAUDE.md index 57fb082..de794cf 100644 --- a/CLAUDE.md +++ b/CLAUDE.md @@ -8,7 +8,7 @@ NeuroStack is a neuroscience-grounded knowledge management system. CLI + MCP ser ```bash npm install -g neurostack # bootstraps CLI, Python, uv, deps -neurostack init # cloud/local → lite/full → vault setup → index +neurostack init # lite/full → vault setup → index ``` ### MCP Server (recommended for Claude Code) @@ -92,7 +92,7 @@ Models: `neurostack-ask` (RAG), `neurostack-search` (hybrid), `neurostack-tiered ### Setup & Diagnostics | Command | Description | |---------|-------------| -| `neurostack init [path]` | One-command setup: cloud/local, lite/full, deps, vault, index. `--mode lite\|full`, `--cloud`, `--profession`, `--pull-models` | +| `neurostack init [path]` | One-command setup: lite/full, deps, vault, index. `--mode lite\|full`, `--profession`, `--pull-models` | | `neurostack scaffold [profession]` | Apply profession pack to existing vault. `--list` to see options | | `neurostack onboard /path/to/notes` | Onboard existing Markdown notes. `--dry-run`, `--no-index` | | `neurostack install` | **(Deprecated)** Use `neurostack init` instead | diff --git a/DPA.md b/DPA.md deleted file mode 100644 index 58dc951..0000000 --- a/DPA.md +++ /dev/null @@ -1,277 +0,0 @@ -# Data Processing Agreement - -**Effective date:** 25 March 2026 -**Last updated:** 25 March 2026 - -This Data Processing Agreement ("DPA") forms part of the [NeuroStack Terms of Service](TERMS.md) between Raphael Southall, operating NeuroStack ("Processor", "we", "us"), and the individual or entity using NeuroStack Cloud or Remote MCP features ("Controller", "you"). - -This DPA applies **only** to Cloud Mode and Remote MCP Mode. Local mode involves no data processing by NeuroStack and is outside the scope of this agreement. - -**Contact:** [hello@neurostack.sh](mailto:hello@neurostack.sh) - ---- - -## 1. Definitions - -| Term | Meaning | -|------|---------| -| **Personal Data** | Any information relating to an identified or identifiable natural person, as defined in GDPR Article 4(1). In the context of NeuroStack, this includes account information and any personal data contained within vault files you upload. | -| **Controller** | You, the NeuroStack user. You determine the purposes and means of processing your vault data. | -| **Processor** | NeuroStack (Raphael Southall). We process your data on your behalf to provide indexing, search, and related services. | -| **Sub-processor** | A third party engaged by the Processor to process Personal Data on behalf of the Controller. | -| **Processing** | Any operation performed on Personal Data, including collection, storage, indexing, retrieval, and deletion. | -| **Data Subject** | An identifiable natural person whose Personal Data is processed. | -| **Supervisory Authority** | An independent public authority responsible for monitoring GDPR compliance. | - ---- - -## 2. Scope and Purpose - -### 2.1 Scope - -This DPA covers the processing of Personal Data that occurs when you: - -- Upload vault files via `neurostack cloud push` -- Use server-side search via `neurostack cloud query` -- Connect AI assistants via Remote MCP at `mcp.neurostack.sh` -- Create and maintain a NeuroStack account - -### 2.2 Processing purposes - -We process your data solely for the following purposes: - -- **Indexing:** Generating embeddings, summaries, and knowledge graph triples from your uploaded vault files. -- **Search:** Executing queries against your indexed database and returning results. -- **Authentication:** Verifying your identity and managing your account. -- **Billing:** Tracking usage against tier limits and processing subscription payments via Stripe. - -### 2.3 Categories of data - -| Category | Examples | Retention | -|----------|---------|-----------| -| Account data | Email, display name, user ID | Until account deletion | -| Vault content | Markdown files uploaded via cloud push | Deleted after indexing completes | -| Indexed data | SQLite database (embeddings, summaries, triples) | Until account deletion | -| Usage data | Query counts, note counts | Until account deletion | -| Authentication data | API key hashes (SHA-256), OAuth tokens | Until account deletion or key revocation | -| Payment data | Handled entirely by Stripe -- NeuroStack stores tier level only | Per Stripe's retention policy | - ---- - -## 3. Controller Obligations - -As Controller, you are responsible for: - -- Ensuring you have a lawful basis to process any Personal Data contained in your vault files. -- Not uploading vault content that you do not have the right to process. -- Informing any Data Subjects whose Personal Data may be included in your vault files. - ---- - -## 4. Processor Obligations - -We will: - -- Process Personal Data **only** on your documented instructions (i.e., the operations you initiate via NeuroStack commands and the MCP interface). -- Not process your data for any purpose other than providing the NeuroStack service. -- Not sell, share, or use your data for advertising, profiling, or training AI models. -- Ensure that persons authorised to process Personal Data are bound by confidentiality obligations. -- Implement appropriate technical and organisational security measures (see Section 6). -- Assist you in fulfilling Data Subject rights requests. -- Delete or return all Personal Data upon termination of the service (see Section 8). -- Make available information necessary to demonstrate compliance with this DPA. - ---- - -## 5. Sub-Processors - -### 5.1 Authorised sub-processors - -You authorise the use of the following sub-processors: - -| Sub-processor | Processing activity | Location | Their DPA | -|--------------|---------------------|----------|-----------| -| **Google Cloud Run** | Compute: receives vault files for indexing, executes search queries | us-central1 (Iowa, US) | [Google Cloud DPA](https://cloud.google.com/terms/data-processing-addendum) | -| **Google Cloud Storage** | Stores indexed SQLite databases | southamerica-east1 (Sao Paulo, BR) | [Google Cloud DPA](https://cloud.google.com/terms/data-processing-addendum) | -| **Google Gemini API** | Generates embeddings and summaries from vault text during indexing | Google API infrastructure | [Google API Terms](https://ai.google.dev/gemini-api/terms) | -| **Firebase Auth** | Authentication and session management | Google infrastructure | [Firebase DPA](https://firebase.google.com/terms/data-processing-terms) | -| **Cloud Firestore** | Stores user records, API key hashes, usage data | Google infrastructure | [Firebase DPA](https://firebase.google.com/terms/data-processing-terms) | -| **Stripe** | Payment processing (under SolidPlus LTD) | Stripe infrastructure | [Stripe DPA](https://stripe.com/legal/dpa) | - -### 5.2 Changes to sub-processors - -We will notify you of any new sub-processors by updating this DPA and announcing the change via the project's GitHub repository. You may object to a new sub-processor by contacting [hello@neurostack.sh](mailto:hello@neurostack.sh) within 30 days of notification. - -### 5.3 Sub-processor liability - -We remain fully liable for the acts and omissions of our sub-processors with respect to the processing of your Personal Data. - ---- - -## 6. Security Measures - -We implement the following technical and organisational measures: - -### 6.1 Data in transit - -- All data transmitted between your device and NeuroStack servers is encrypted via **HTTPS/TLS**. -- All communication with sub-processors uses encrypted channels. - -### 6.2 Data at rest - -- Indexed databases in Google Cloud Storage are encrypted at rest using Google-managed encryption keys. -- Database downloads use **signed URLs** with time-limited validity. - -### 6.3 Access control - -- **Tenant isolation:** Each user's data is stored under a unique prefix (`vaults/{user_id}/`), preventing cross-user access. -- API key authentication uses **constant-time comparison** to prevent timing attacks. -- API keys are stored as **SHA-256 hashes** -- plaintext keys are never retained. -- OAuth 2.1 with Firebase Auth for account access. - -### 6.4 Data minimisation - -- Uploaded vault files are **not retained** after indexing completes. -- Only the derived indexed database is stored. -- NeuroStack never stores payment card details -- Stripe handles all payment data. - -### 6.5 Operational security - -- Infrastructure runs on Google Cloud Platform with its [security practices](https://cloud.google.com/security). -- No persistent servers -- Cloud Run scales to zero, reducing attack surface. -- No shared databases between users. - ---- - -## 7. Data Breach Notification - -In the event of a Personal Data breach: - -- We will notify you **without undue delay** and in any event within **72 hours** of becoming aware of the breach. -- Notification will include: the nature of the breach, categories and approximate number of Data Subjects affected, likely consequences, and measures taken or proposed to mitigate the breach. -- Notification will be sent to the email address associated with your NeuroStack account. -- We will cooperate with you and any supervisory authority in investigating and resolving the breach. - ---- - -## 8. Data Deletion and Return - -### 8.1 During the service - -- You can download your indexed database at any time using `neurostack cloud pull`. -- Your vault files remain on your local machine -- they are never the sole copy held by NeuroStack. - -### 8.2 Account deletion - -When you run `neurostack cloud delete-account`: - -- Your indexed database is permanently deleted from Google Cloud Storage. -- Your user record is deleted from Firestore. -- Your authentication record is deleted from Firebase Auth. -- All API key hashes associated with your account are deleted. -- Usage records are deleted. - -### 8.3 Post-termination - -After account deletion, we will have no copies of your Personal Data, with the exception of: - -- Data retained in sub-processor backups that are automatically purged according to their retention schedules. -- Data we are required to retain by applicable law. - ---- - -## 9. Data Subject Rights - -We will assist you in responding to Data Subject requests, including: - -- **Access** -- providing copies of Personal Data we hold. -- **Rectification** -- correcting inaccurate data. -- **Erasure** -- deleting Personal Data (achievable via `neurostack cloud delete-account`). -- **Portability** -- providing data in a structured, machine-readable format (SQLite database via `neurostack cloud pull`). -- **Restriction and objection** -- restricting or ceasing processing upon request. - -If we receive a Data Subject request directly, we will redirect the Data Subject to you unless legally required to respond directly. - ---- - -## 10. Audit Rights - -You may request reasonable information about our data processing practices to verify compliance with this DPA. Requests should be directed to [hello@neurostack.sh](mailto:hello@neurostack.sh). - -We will provide: - -- Written responses to audit questionnaires. -- Summaries of relevant third-party audit reports or certifications held by our sub-processors (e.g., Google Cloud SOC 2 reports). - -On-site audits are not available given the serverless nature of the infrastructure. All compute runs on Google Cloud Platform, which maintains [comprehensive compliance certifications](https://cloud.google.com/security/compliance). - ---- - -## 11. International Data Transfers - -### 11.1 Transfer locations - -| Data | Location | Transfer mechanism | -|------|----------|-------------------| -| Vault files (during indexing) | us-central1 (US) | Google Cloud DPA with EU SCCs | -| Indexed database | southamerica-east1 (Brazil) | Google Cloud DPA with EU SCCs | -| Account data | Google infrastructure | Firebase DPA with EU SCCs | -| Payment data | Stripe infrastructure | Stripe DPA with EU SCCs | - -### 11.2 Safeguards for EU/EEA transfers - -For transfers of Personal Data from the EU/EEA to countries without an adequacy decision, we rely on: - -- **EU Standard Contractual Clauses (SCCs)** incorporated into our sub-processors' data processing agreements (Google Cloud DPA, Firebase DPA, Stripe DPA). -- Google's commitment to challenge disproportionate government access requests, as documented in their [transparency reports](https://transparencyreport.google.com/). - -### 11.3 UK transfers - -For transfers from the UK, we rely on the UK International Data Transfer Addendum to the EU SCCs, as incorporated by our sub-processors. - ---- - -## 12. GDPR Compliance - -### 12.1 Legal basis for processing - -As Processor, we process Personal Data on the basis of your instructions (GDPR Article 28). As Controller, you are responsible for establishing a lawful basis (e.g., legitimate interest, consent) for any Personal Data in your vault files. - -For account data, we process on the basis of **contract performance** (providing the NeuroStack service) and **legitimate interest** (preventing abuse, maintaining security). - -### 12.2 Data Protection Impact Assessment - -Given that NeuroStack processes vault content that may contain sensitive personal data, you may need to conduct a Data Protection Impact Assessment (DPIA) under GDPR Article 35. We will provide reasonable assistance if requested. - -### 12.3 Records of processing - -We maintain records of processing activities as required by GDPR Article 30(2). - ---- - -## 13. Term and Termination - -- This DPA is effective for as long as you maintain a NeuroStack account with cloud features. -- Upon account deletion, the data deletion provisions in Section 8 apply. -- Sections that by their nature should survive termination (data deletion obligations, liability, audit rights) will survive. - ---- - -## 14. Liability - -Our liability under this DPA is subject to the limitations set out in the [NeuroStack Terms of Service](TERMS.md). Nothing in this DPA limits liability for breaches of data protection law that cannot be limited under applicable law. - ---- - -## 15. Amendments - -We may update this DPA to reflect changes in our processing activities, sub-processors, or applicable law. Material changes will be announced via the project's GitHub repository. Continued use of cloud features after notification constitutes acceptance. - ---- - -## 16. Contact - -For questions about this DPA, data processing practices, or to exercise your rights: - -- **Email:** [hello@neurostack.sh](mailto:hello@neurostack.sh) -- **GitHub:** [https://github.com/neurostackai/neurostack](https://github.com/neurostackai/neurostack) diff --git a/PRIVACY.md b/PRIVACY.md index 7d00c8f..2182a35 100644 --- a/PRIVACY.md +++ b/PRIVACY.md @@ -1,197 +1,35 @@ # Privacy Policy -**Effective date:** 25 March 2026 -**Last updated:** 25 March 2026 +**Last updated:** 3 June 2026 -NeuroStack is an open-source (Apache-2.0) knowledge management tool created by Raphael Southall. This policy explains what data NeuroStack collects, how it is processed, and your rights regarding that data. - -**Contact:** [hello@neurostack.sh](mailto:hello@neurostack.sh) -**Website:** [https://neurostack.sh](https://neurostack.sh) - ---- - -## 1. Overview - -NeuroStack operates in three modes, each with different data implications: - -| Mode | Data leaves your machine? | Account required? | -|------|--------------------------|-------------------| -| **Local (Lite/Full/Community)** | No | No | -| **Cloud** | Yes, when you explicitly push | Yes | -| **Remote MCP** | Search queries only | Yes | - -**The core principle:** in local mode, NeuroStack collects zero data. No telemetry, no analytics, no crash reporting, no phone-home behaviour of any kind. - ---- - -## 2. Local Mode (Lite / Full / Community) - -### What happens - -- Your Markdown vault files are read and indexed into a local SQLite database. -- Vault files are **read-only** -- NeuroStack never modifies your source files. -- The database is stored at `~/.local/share/neurostack/`. -- In Full mode, vault text is sent to a **local** Ollama instance running on `localhost` for embeddings and summaries. This traffic never leaves your machine. - -### What is collected - -**Nothing.** There is: - -- No telemetry -- No analytics -- No usage tracking -- No crash reports sent externally -- No network calls to any external service - -### How to delete your data - -Delete the database directory at `~/.local/share/neurostack/`. Uninstalling NeuroStack removes all traces. - ---- - -## 3. Cloud Mode - -Cloud mode is entirely opt-in. You must explicitly run `neurostack cloud push` to upload any data. - -### 3.1 What data is uploaded - -When you run `neurostack cloud push`, changed `.md` files from your vault are uploaded via HTTPS to a processing server. - -### 3.2 How data is processed - -- **Compute:** Google Cloud Run (us-central1) receives and processes your files. -- **AI processing:** Google Gemini API processes your vault text to generate embeddings (gemini-embedding-001) and summaries/triples (gemini-2.5-flash). -- **Storage:** The resulting indexed SQLite database is stored in Google Cloud Storage (gs://neurostack-prod, southamerica-east1), isolated under your user prefix (`vaults/{user_id}/`). - -### 3.3 Data retention - -- **Uploaded vault files are not retained after indexing completes.** Only the processed SQLite database is stored. -- Your indexed database remains in cloud storage until you delete your account. -- You can download your database at any time with `neurostack cloud pull`. After download, all searches run locally. - -### 3.4 Server-side queries - -When you use `neurostack cloud query`, your search query is sent to Cloud Run, which searches against your cached indexed database and returns results. No vault files are transmitted during queries. - ---- - -## 4. Remote MCP Mode - -When you connect an AI assistant (Claude, ChatGPT, etc.) to `https://mcp.neurostack.sh/mcp`: - -- **Authentication** is handled via OAuth 2.1 wrapping Firebase Auth (Google sign-in). -- **Search queries** are sent to Cloud Run and executed against your indexed database. -- **No vault files are transmitted** during MCP queries -- only search queries and search results. - ---- - -## 5. Authentication and Accounts - -When you create a NeuroStack account (required for Cloud and Remote MCP modes only): - -| Data collected | Purpose | Where stored | -|---------------|---------|-------------| -| Email address | Account identification, communication | Firebase Auth, Firestore | -| Display name | Account display | Firebase Auth, Firestore | -| User ID | Internal identifier | Firebase Auth, Firestore | -| Tier level | Billing and feature access | Firestore | -| Account creation date | Record keeping | Firestore | -| API key hashes (SHA-256) | API authentication | Firestore | -| Usage counts | Enforcing tier limits | Firestore | - -**We never store plaintext API keys.** Only SHA-256 hashes are retained. Authentication uses the device code flow for CLI login. - ---- - -## 6. Billing - -Billing is handled entirely by **Stripe** (operating under SolidPlus LTD). - -- NeuroStack **never sees, stores, or processes** your payment card numbers or bank details. -- Stripe processes payments in accordance with [Stripe's privacy policy](https://stripe.com/privacy). -- NeuroStack stores only your subscription tier level and usage counts. +NeuroStack is an open-source (Apache-2.0) knowledge management tool created by Raphael Southall. It runs entirely on your own machine. --- -## 7. Third-Party Sub-Processors - -The following third parties process data only in Cloud and Remote MCP modes: +## What NeuroStack collects -| Sub-processor | Purpose | Data processed | Location | -|--------------|---------|---------------|----------| -| Google Cloud Run | Compute / indexing / queries | Vault files (during indexing), search queries | us-central1 | -| Google Cloud Storage | Database storage | Indexed SQLite database | southamerica-east1 | -| Google Gemini API | Embedding and summary generation | Vault text (during indexing) | Google API infrastructure | -| Firebase Auth | Authentication | Email, display name, OAuth tokens | Google infrastructure | -| Cloud Firestore | User records | User profile, API key hashes, usage data | Google infrastructure | -| Stripe | Payment processing | Payment information (handled by Stripe directly) | Stripe infrastructure | +Nothing. NeuroStack has no telemetry, no analytics, no crash reporting, and no phone-home behaviour. The maintainer receives no personal data from your use of the software. -Google's data processing terms apply to all Google Cloud services listed above. See [Google Cloud Data Processing Terms](https://cloud.google.com/terms/data-processing-addendum). +## How your data is handled ---- +- NeuroStack reads your Markdown vault and indexes it into a local SQLite database at `~/.local/share/neurostack/`. +- Your vault files are read-only during indexing. NeuroStack never modifies your source files unless you explicitly use the opt-in MCP write tools, which act on your own local git repository. +- The index stays on your machine. It is never sent to any server operated by NeuroStack or its maintainer. -## 8. Data Security +## Third-party LLM and embedding providers -- All data in transit is encrypted via **HTTPS/TLS**. -- Cloud-stored databases are **tenant-isolated** by user ID prefix. -- Database downloads use **signed URLs** with limited validity. -- API key authentication uses **constant-time comparison** to prevent timing attacks. -- Vault files are **not retained** after indexing -- only the derived database is stored. - ---- +In Full mode, NeuroStack sends vault text to a language and embedding backend to generate summaries, embeddings, and knowledge-graph triples. By default this is a local [Ollama](https://ollama.ai) instance on `localhost`, so that traffic never leaves your machine. -## 9. Your Rights +If you configure NeuroStack to use a third-party provider instead (for example an OpenAI-compatible endpoint, Together AI, or Groq), the text and queries you send for processing go to that provider and are handled under that provider's privacy policy. This is the only situation in which any data leaves your machine, and it happens only because you configured it. -You have the right to: +## Deleting your data -- **Access** your data -- download your indexed database with `neurostack cloud pull`. -- **Delete** your data -- run `neurostack cloud delete-account` to permanently delete your account, all stored databases, and all associated records. -- **Export** your data -- your vault files remain on your local machine at all times. The indexed database can be downloaded at any time. -- **Withdraw consent** -- stop using cloud features at any time. Your local installation continues to work independently. +Delete the database directory at `~/.local/share/neurostack/`, or run `neurostack uninstall`. Your vault files are untouched. -For users in the EU/EEA, you additionally have the right to: - -- **Rectification** -- request correction of inaccurate personal data. -- **Restriction** -- request restricted processing of your data. -- **Portability** -- receive your personal data in a structured, machine-readable format. -- **Object** -- object to processing of your personal data. -- **Lodge a complaint** with your local data protection authority. - -To exercise any of these rights, contact [hello@neurostack.sh](mailto:hello@neurostack.sh). - ---- - -## 10. International Data Transfers - -Cloud mode processes data on Google Cloud infrastructure in the United States (us-central1) and stores indexed databases in South America (southamerica-east1). For users in the EU/EEA, transfers to the US are covered by Google's Data Processing Addendum, which includes EU Standard Contractual Clauses. See [Google's compliance documentation](https://cloud.google.com/privacy/gdpr). - ---- - -## 11. Cookies - -The NeuroStack dashboard at `app.neurostack.sh` uses **only essential cookies** for Firebase Auth session management. There are: - -- No tracking cookies -- No third-party advertising cookies -- No analytics cookies - ---- - -## 12. Children's Privacy - -NeuroStack is not intended for use by anyone under the age of 13. We do not knowingly collect personal information from children under 13. If you believe a child under 13 has provided personal information, contact [hello@neurostack.sh](mailto:hello@neurostack.sh) and we will delete it. - ---- - -## 13. Changes to This Policy - -We will update this policy as NeuroStack evolves. Material changes will be announced via the project's GitHub repository and changelog. Continued use of cloud features after changes constitutes acceptance of the updated policy. - ---- +## Changes to this policy -## 14. Contact +Material changes will be announced through the project's GitHub repository and changelog. -For privacy questions, data requests, or concerns: +## Contact -- **Email:** [hello@neurostack.sh](mailto:hello@neurostack.sh) - **GitHub:** [https://github.com/raphasouthall/neurostack](https://github.com/raphasouthall/neurostack) diff --git a/README.md b/README.md index 4ea71e2..e3ccc5b 100644 --- a/README.md +++ b/README.md @@ -1,4 +1,4 @@ -NeuroStack +NeuroStack [![PyPI](https://img.shields.io/pypi/v/neurostack)](https://pypi.org/project/neurostack/) [![npm](https://img.shields.io/npm/v/neurostack)](https://www.npmjs.com/package/neurostack) @@ -29,8 +29,7 @@ By default, NeuroStack is a read-only indexing layer: - Indexing, search, summaries, and graph analysis **never modify your Markdown files** - All index data lives in NeuroStack's own separate database - To remove it completely: `neurostack uninstall` — your notes are untouched -- In local mode: nothing ever leaves your machine -- In cloud mode: you review and approve exactly what gets sent, and can exclude any folder with a `.neurostackignore` file +- Nothing ever leaves your machine, unless you configure a third-party LLM provider for summaries and embeddings If your vault is a git repo, four **opt-in MCP write tools** let an AI client author and edit notes for you: `vault_write_file`, `vault_delete_file`, plus `vault_read_file` / `vault_list_files`. Every write commits and pushes to your git remote with a descriptive message — so every change is visible in `git log`, revertable with `git revert`, and serialised under a per-vault lock. Writes hard-reject invalid frontmatter, paths outside the vault, and hidden directories (`.git`, `.obsidian`, …). Because the tools are exposed to any client talking to `neurostack serve`, gate them at the transport (auth, tunnel, LAN only) if you put the MCP endpoint on the public internet. @@ -52,7 +51,7 @@ You do not need to be a developer. If you take notes in Markdown — or can expo ## Get started in three steps -You will need [Node.js](https://nodejs.org) installed (most computers already have it). That is the only prerequisite for cloud mode — no GPU, no Python knowledge, nothing else. +You will need [Node.js](https://nodejs.org) installed (most computers already have it). The npm package handles the Python setup for you. **Step 1 — Install** @@ -66,7 +65,7 @@ npm install -g neurostack neurostack init ``` -The setup wizard asks: cloud or local, which vault folder, which profession pack. It does everything else automatically. +The setup wizard asks which vault folder to index, which mode to run (Lite or Full), and which profession pack to apply. It does everything else automatically. **Step 3 — Connect to your AI** @@ -85,36 +84,12 @@ For Cursor, Windsurf, Gemini CLI, or VS Code: neurostack setup-client cursor # or: windsurf, gemini, vscode ``` -**Zero-install option** — connect Claude to your vault via NeuroStack Cloud with nothing installed locally: -``` -claude mcp add neurostack --transport http https://mcp.neurostack.sh/mcp -``` - Done. Open a new conversation and ask your AI about something from your notes. -**Free tier:** 500 queries/month, 200 notes. No credit card required. [Start at app.neurostack.sh](https://app.neurostack.sh) - -
-Cloud vs Local — what's the difference? - -| | Cloud (recommended) | Local | -|--|--------------------|-| -| **What you need** | Just Node.js | Node.js + Ollama (a local AI engine) | -| **GPU required** | No | Recommended, but not required | -| **Setup time** | About 2 minutes | 10-20 minutes | -| **Works offline** | No | Yes | -| **Syncs across devices** | Yes, automatically | Manual | -| **Cost** | Free tier: 500 queries/month, 200 notes. [Pro plans](https://neurostack.sh) for more. | Free. Your hardware, your cost. | -| **Your files** | Sent for indexing via encrypted connection, not stored after processing | Never leave your machine | - -**Privacy notice:** Cloud mode requires explicit consent before uploading. Your vault files are sent to Google's Gemini API for indexing (embeddings, summaries, connections between notes). Files are processed via HTTPS and not retained after indexing completes. Run `neurostack cloud consent` to review and grant consent. Exclude sensitive files with a `.neurostackignore` file (gitignore syntax). - -
-
-Local mode (Lite and Full) +Lite and Full modes -Run everything on your machine with Ollama. Choose a tier during `neurostack init`: +Everything runs on your machine. Choose a tier during `neurostack init`: - **Lite** (~130 MB) — keyword search, link-based connections between notes, stale detection, MCP server. No GPU or Ollama required. - **Full** (~560 MB) — adds semantic search (finds notes by meaning, not just keywords), AI-generated summaries, connections between notes, and topic clustering via local [Ollama](https://ollama.ai). GPU or 6+ core CPU recommended. @@ -122,8 +97,8 @@ Run everything on your machine with Ollama. Choose a tier during `neurostack ini Non-interactive setup: ```bash -neurostack init --mode full ~/my-notes # local full mode -neurostack init --cloud ~/my-notes # cloud mode +neurostack init --mode lite ~/my-notes # lite mode +neurostack init --mode full ~/my-notes # full mode ```
@@ -265,12 +240,6 @@ Your vault changes. NeuroStack watches it. neurostack watch # auto-index on vault changes ``` -Or sync on every git commit: - -```bash -neurostack cloud install-hooks -``` - The index updates as you write. Stale detection runs continuously. You don't maintain it — it maintains itself. --- @@ -284,7 +253,6 @@ The index updates as you write. Stale detection runs continuously. You don't mai | No memory of yesterday's session | `session_brief` reconstructs working context | | Reading 10 notes to find one fact | Tiered retrieval: ~15 tokens for a structured fact | | Decisions lost after `/clear` | Typed memories persist indefinitely | -| Cross-machine notes out of sync | Cloud sync: push once, pull anywhere | --- @@ -363,7 +331,6 @@ NeuroStack reads your vault. By default, it writes nothing back — all index da # Setup neurostack init # one-command setup: deps, vault, index neurostack init --mode full ~/brain # non-interactive full mode -neurostack init --cloud ~/brain # non-interactive cloud mode neurostack onboard ~/my-notes # import existing Markdown notes neurostack scaffold researcher # apply a profession pack neurostack scaffold --list # see all packs @@ -405,15 +372,6 @@ neurostack harvest --sessions 5 # extract session insights neurostack sessions search "query" # search transcripts neurostack hooks install # hourly harvest timer -# Cloud -neurostack cloud login # browser OAuth login -neurostack cloud push # upload + index vault -neurostack cloud pull # download indexed DB -neurostack cloud sync # push changes + fetch memories -neurostack cloud install-hooks # auto-sync on git commit/merge -neurostack cloud auto-sync enable # periodic sync via systemd timer -neurostack cloud consent # review and grant privacy consent - # Client setup neurostack setup-client cursor # or: windsurf, gemini, vscode, claude-code neurostack setup-client --list @@ -427,33 +385,6 @@ neurostack demo # interactive demo with sample vault -
-Cloud sync details - -Keep your vault indexed across machines without manual steps. - -**Automatic sync triggers:** - -- **Git hooks** — sync on every commit or merge: `neurostack cloud install-hooks` -- **systemd timer** — periodic background sync: `neurostack cloud auto-sync enable --interval 15min` -- **Manual** — push changes and fetch memories in one command: `neurostack cloud sync` - -**Upload format:** Vault files are packed into a compressed tar.gz archive. Typical compression is 60-80%. - -**Concurrent push safety:** A server-side push lock prevents two devices from pushing simultaneously. - -**`.neurostackignore`:** Place in your vault root to exclude sensitive paths (gitignore syntax): - -``` -private/ -journal/*.md -*-draft.md -``` - -**Upgrading from v0.10.x:** Cloud mode now requires explicit consent before uploading. Run `neurostack cloud consent` on first push after upgrading. - -
-
Neuroscience basis @@ -479,19 +410,15 @@ Full citations: [docs/neuroscience-appendix.md](docs/neuroscience-appendix.md) **Does it modify my vault files?** Not by default. Indexing, search, summaries, and every read tool leave your files untouched — all index data lives in NeuroStack's own SQLite databases. Four opt-in MCP write tools (`vault_write_file`, `vault_delete_file`, plus `vault_read_file` / `vault_list_files`) let an AI client author and edit notes; every write commits and pushes to your git remote, so changes are tracked and revertable. If your vault is not a git repo, the file is still written to disk but the commit step is skipped. -**Do I need a GPU?** No. Cloud mode requires only Node.js. Local Lite mode has zero ML dependencies. Local Full mode runs on CPU but summarization is slow without a GPU. +**Do I need a GPU?** No. Lite mode has zero ML dependencies. Full mode runs on CPU but summarization is slow without a GPU. **Do I need to know Python?** No. The npm package handles everything. You never touch a virtualenv. -**What's the catch with the free tier?** 500 queries/month, 200 notes. No credit card required. Pro plans at [neurostack.sh](https://neurostack.sh) remove those limits. - -**How large a vault can it handle?** Tested with ~5,000 notes. FTS5 search stays fast at any size. Cloud indexing handles 500+ notes in minutes. +**How large a vault can it handle?** Tested with ~5,000 notes. FTS5 search stays fast at any size. **Can I use it without an AI client?** Yes. The CLI works standalone and pipes into any LLM. -**Is my vault private in local mode?** Yes. Nothing leaves your machine. - -**What if I want to exclude sensitive notes from cloud?** Add a `.neurostackignore` file to your vault root (gitignore syntax). Those files are never uploaded. +**Is my vault private?** Yes. Nothing leaves your machine, unless you point Full mode at a third-party LLM provider instead of local Ollama. In that case the text you index goes to that provider under its own policy. **What AI clients does it work with?** Claude Code, Claude Desktop, Cursor, Windsurf, Gemini CLI, VS Code, and Codex — anything that supports MCP. @@ -500,9 +427,8 @@ Full citations: [docs/neuroscience-appendix.md](docs/neuroscience-appendix.md) ## Requirements - Linux or macOS -- **Cloud mode:** Node.js only. No GPU, no Ollama, no Python setup. -- **Local Lite mode:** Node.js + Python 3.11+. No GPU or Ollama required. -- **Local Full mode:** [Ollama](https://ollama.ai) with `nomic-embed-text` and a summary model. GPU or 6+ core CPU recommended. +- **Lite mode:** Node.js + Python 3.11+. No GPU or Ollama required. +- **Full mode:** [Ollama](https://ollama.ai) with `nomic-embed-text` and a summary model. GPU or 6+ core CPU recommended. --- @@ -515,12 +441,10 @@ neurostack init Two minutes. One wizard. Your AI stops forgetting. -- **Website:** [neurostack.sh](https://neurostack.sh) -- **Dashboard:** [app.neurostack.sh](https://app.neurostack.sh) - **Contributing:** [CONTRIBUTING.md](CONTRIBUTING.md) -- **Contact:** [hello@neurostack.sh](mailto:hello@neurostack.sh) +- **GitHub:** [github.com/raphasouthall/neurostack](https://github.com/raphasouthall/neurostack) - **Sponsor:** [GitHub Sponsors](https://github.com/sponsors/raphasouthall) | [Buy me a coffee](https://buymeacoffee.com/raphasouthall) --- -Apache-2.0 — see [LICENSE](LICENSE). No GPL dependencies. Built by [SolidPlus LTD](https://neurostack.sh). +Apache-2.0 — see [LICENSE](LICENSE). No GPL dependencies. diff --git a/SECURITY.md b/SECURITY.md index 1c8cbbd..4f24cd2 100644 --- a/SECURITY.md +++ b/SECURITY.md @@ -10,7 +10,7 @@ **Do not open a public issue for security vulnerabilities.** -Email: hello@neurostack.sh +Report privately through GitHub Security Advisories: [github.com/raphasouthall/neurostack/security/advisories/new](https://github.com/raphasouthall/neurostack/security/advisories/new) Include: - Description of the vulnerability diff --git a/TERMS.md b/TERMS.md index 5950ede..f75b303 100644 --- a/TERMS.md +++ b/TERMS.md @@ -1,314 +1,29 @@ -# NeuroStack Terms of Service +# NeuroStack Terms -**Effective date:** 27 March 2026 -**Last updated:** 27 March 2026 +**Last updated:** 3 June 2026 ---- - -## 1. Acceptance of Terms - -By creating a NeuroStack account, accessing the NeuroStack Cloud service, or using any cloud-hosted features (including Remote MCP), you agree to be bound by these Terms of Service ("Terms"). If you do not agree to these Terms, do not use the NeuroStack Cloud service. - -These Terms apply to the **NeuroStack Cloud service** operated by SolidPlus LTD. The open-source NeuroStack software, licensed under Apache-2.0, may be used independently without accepting these Terms. However, any use of cloud-hosted features — including indexing, search, Remote MCP, and account management — is governed by these Terms. - ---- - -## 2. Service Description - -### 2.1 Open-source software (not covered by these Terms) - -NeuroStack's core software is open-source under the Apache-2.0 licence. You may install it locally, run it against your own infrastructure, and modify it freely under that licence. Local usage does not involve any data processing by SolidPlus LTD and is outside the scope of these Terms. - -### 2.2 Cloud service (covered by these Terms) - -NeuroStack Cloud is a managed service that provides: - -- **Cloud indexing** — upload vault files for server-side embedding, summarisation, and knowledge graph extraction. -- **Cloud search** — query your indexed database via `neurostack cloud query` or the Remote MCP endpoint at `mcp.neurostack.sh`. -- **Account management** — authentication via Firebase Auth (Google OAuth), API key management, and usage tracking. -- **Billing** — subscription management and payment processing via Stripe. - -The Cloud service runs on Google Cloud Platform (Cloud Run, Cloud Storage, Firestore) and uses the Gemini API for AI processing. - ---- - -## 3. Account Registration and Security - -### 3.1 Account creation - -To use NeuroStack Cloud, you must create an account by authenticating via Google OAuth through Firebase Auth. You must provide accurate information and keep your account details up to date. - -### 3.2 Account security - -You are responsible for: - -- Maintaining the confidentiality of your API keys and authentication credentials. -- All activity that occurs under your account. -- Notifying us promptly at [hello@neurostack.sh](mailto:hello@neurostack.sh) if you become aware of any unauthorised access to your account. - -We store API keys as SHA-256 hashes and never retain plaintext keys. However, you are solely responsible for safeguarding your keys after generation. - -### 3.3 One account per person - -Each account is for a single individual. You may not share account credentials or API keys with others. Organisation and team features, if offered in the future, will have separate terms. - ---- - -## 4. Acceptable Use Policy - -You agree to use NeuroStack Cloud in a manner that is lawful, respectful, and consistent with the intended purpose of the service. You may use the service to: - -- Index and search your personal or professional knowledge base. -- Connect AI assistants via Remote MCP for knowledge retrieval. -- Manage and organise your notes, documents, and research. - -You must not use the service in any way that violates applicable law or regulation, infringes the rights of others, or compromises the integrity and availability of the service for other users. - ---- - -## 5. Prohibited Content and Abuse - -### 5.1 Prohibited content - -You must not upload, index, or process through NeuroStack Cloud any content that: - -- Violates any applicable law or regulation. -- Infringes the intellectual property rights of any third party. -- Contains malware, viruses, or any code designed to disrupt, damage, or gain unauthorised access to systems. -- Is designed to exploit or harm minors in any way. -- Constitutes illegal pornography, incitement to violence, or terrorism-related material. - -### 5.2 Abuse of the service - -You must not: - -- Attempt to access another user's data or account. -- Reverse-engineer, decompile, or attempt to extract the source code of proprietary cloud components (the open-source components are freely available under Apache-2.0). -- Use the service to build a competing product by systematically extracting proprietary features or methodologies. -- Circumvent usage limits, rate limits, or other technical restrictions. -- Use automated means to create accounts or generate excessive load on the service. -- Resell or redistribute access to the Cloud service without prior written consent. - -### 5.3 Enforcement - -We reserve the right to suspend or terminate accounts that violate this section, with or without notice, depending on the severity of the violation. Where practicable, we will provide notice and an opportunity to remedy the violation before taking action. - ---- - -## 6. Service Availability - -### 6.1 No uptime guarantee for free tier - -NeuroStack Cloud runs on Google Cloud Run, which scales to zero when not in use. The free tier is provided **as-is** with **no uptime guarantee, no SLA, and no guarantee of availability**. The service may be unavailable due to maintenance, scaling delays, infrastructure issues, or capacity constraints. - -### 6.2 Paid tier availability - -Paid tiers may include availability commitments as specified in the applicable plan description. Any such commitments will be documented separately and form part of these Terms by reference. - -### 6.3 Maintenance and changes - -We may modify, suspend, or discontinue any part of the Cloud service at any time. We will make reasonable efforts to provide advance notice of material changes. We are not liable for any modification, suspension, or discontinuation of the service. - -### 6.4 Data durability - -While we use industry-standard infrastructure (Google Cloud Storage) to store your indexed data, we do not guarantee against data loss. You are responsible for maintaining local copies of your vault files. NeuroStack is designed so that your original vault files always remain on your local machine — the cloud stores only derived indexed data that can be regenerated. - ---- - -## 7. Free Tier and Paid Tiers - -### 7.1 Free tier - -The free tier provides access to NeuroStack Cloud with usage limits as described on [neurostack.sh](https://neurostack.sh). The free tier: - -- Has no uptime or availability guarantee. -- May be subject to rate limiting and capacity constraints. -- May be modified or discontinued at any time. - -### 7.2 Paid tiers - -Paid subscriptions are available with increased limits, priority processing, and additional features as described on [neurostack.sh](https://neurostack.sh). Paid tiers: - -- Are billed via Stripe on a recurring basis (monthly or annual, depending on the plan selected). -- Renew automatically unless cancelled before the end of the billing period. -- May be cancelled at any time. Cancellation takes effect at the end of the current billing period; no partial refunds are issued for the remaining period. - -### 7.3 Payment terms - -- All payments are processed by Stripe. SolidPlus LTD does not store payment card details. -- Prices are listed on [neurostack.sh](https://neurostack.sh) and may be changed with 30 days' notice. -- If payment fails, we may suspend access to paid features until payment is resolved. Your data will be retained for a reasonable period to allow you to resolve payment issues or downgrade to the free tier. - -### 7.4 Refunds - -Refunds are handled on a case-by-case basis. Contact [hello@neurostack.sh](mailto:hello@neurostack.sh) for refund requests. - ---- - -## 8. Intellectual Property - -### 8.1 Your content - -You retain all rights, title, and interest in your vault data, including any files you upload, the content of your notes, and any intellectual property contained within them. Uploading content to NeuroStack Cloud does not transfer ownership of that content to us. - -We claim no intellectual property rights over your vault data. The indexed representations (embeddings, summaries, knowledge graph triples) derived from your content are considered your data and are subject to the same ownership. - -### 8.2 Licence to operate - -By uploading content to NeuroStack Cloud, you grant us a limited, non-exclusive, non-transferable licence to process your content solely for the purpose of providing the service (indexing, search, and retrieval). This licence terminates when you delete your account or remove your data. - -### 8.3 NeuroStack software and branding - -The NeuroStack name, logo, and branding are the property of SolidPlus LTD. The open-source NeuroStack software is licensed under Apache-2.0. Proprietary cloud components, infrastructure, and service-specific features remain the intellectual property of SolidPlus LTD. - -### 8.4 Feedback - -If you provide feedback, suggestions, or feature requests, you grant us a non-exclusive, royalty-free, perpetual licence to use that feedback to improve the service without obligation to you. +NeuroStack is open-source software distributed under the Apache-2.0 licence. These terms summarise how you may use the software and the limits of the maintainer's liability. The Apache-2.0 licence governs in full; see [LICENSE](LICENSE). --- -## 9. Account Termination and Data Deletion - -### 9.1 Termination by you +## 1. Licence -You may delete your account at any time by running `neurostack cloud delete-account`. Upon deletion: +NeuroStack is licensed under the Apache License, Version 2.0. You may install, run, modify, and redistribute it under the terms of that licence. The full text is in [LICENSE](LICENSE). -- Your indexed database is permanently deleted from Google Cloud Storage. -- Your user record is deleted from Firestore. -- Your authentication record is deleted from Firebase Auth. -- All API key hashes associated with your account are deleted. -- Usage records are deleted. +## 2. Acceptable use -### 9.2 Termination by us - -We may suspend or terminate your account if: - -- You breach these Terms, including the Acceptable Use Policy or Prohibited Content provisions. -- Your account has been inactive for an extended period (we will provide at least 90 days' notice before deletion due to inactivity). -- We are required to do so by law. -- The service is discontinued. - -Where practicable, we will provide notice and an opportunity to export your data before termination. - -### 9.3 Effect of termination - -Upon termination, your right to access the Cloud service ceases immediately. Data deletion follows the process described in the [Data Processing Agreement](DPA.md), Section 8. - -### 9.4 Survival - -Sections that by their nature should survive termination — including Intellectual Property, Liability Limitations, Indemnification, and Governing Law — will survive. - ---- - -## 10. Liability Limitations - -### 10.1 Disclaimer of warranties - -To the maximum extent permitted by applicable law, NeuroStack Cloud is provided **"as is"** and **"as available"**, without warranties of any kind, whether express, implied, or statutory, including but not limited to implied warranties of merchantability, fitness for a particular purpose, and non-infringement. - -We do not warrant that the service will be uninterrupted, error-free, or secure, or that any defects will be corrected. - -### 10.2 Limitation of liability - -To the maximum extent permitted by applicable law, SolidPlus LTD shall not be liable for any indirect, incidental, special, consequential, or punitive damages, including but not limited to loss of profits, data, or goodwill, arising out of or in connection with your use of the service. - -Our total aggregate liability for any claims arising out of or relating to these Terms or the service shall not exceed the greater of: (a) the total amount you paid to us in the 12 months preceding the claim, or (b) GBP 100. - -### 10.3 Exceptions - -Nothing in these Terms excludes or limits liability for: - -- Death or personal injury caused by negligence. -- Fraud or fraudulent misrepresentation. -- Any other liability that cannot be excluded or limited under applicable law. - ---- - -## 11. Indemnification - -You agree to indemnify, defend, and hold harmless SolidPlus LTD and its directors, officers, and employees from and against any claims, damages, losses, liabilities, and expenses (including reasonable legal fees) arising out of or in connection with: - -- Your use of the Cloud service. -- Your violation of these Terms. -- Your violation of any applicable law or regulation. -- Any content you upload or process through the service that infringes the rights of a third party. - ---- +Use NeuroStack lawfully. Do not use it to violate applicable law, infringe the rights of others, or process content you have no right to process. NeuroStack runs on your own machine against your own vault; you are responsible for the data you index and for how you connect it to AI clients. -## 12. Modification of Terms +## 3. No warranty -We may update these Terms from time to time to reflect changes in the service, legal requirements, or business practices. +To the maximum extent permitted by applicable law, NeuroStack is provided **"as is"**, without warranty of any kind, whether express, implied, or statutory, including but not limited to implied warranties of merchantability, fitness for a particular purpose, and non-infringement. There is no guarantee that the software will be uninterrupted, error-free, or secure. -### 12.1 Notice of changes +## 4. Limitation of liability -We will notify you of material changes to these Terms by: +To the maximum extent permitted by applicable law, the maintainer shall not be liable for any indirect, incidental, special, consequential, or punitive damages, including loss of profits, data, or goodwill, arising out of or in connection with your use of the software. -- Sending a notification to the email address associated with your account, and/or -- Displaying a prominent notice within the service or on [neurostack.sh](https://neurostack.sh). +Nothing in these terms excludes or limits liability that cannot be excluded or limited under applicable law, including liability for death or personal injury caused by negligence, or for fraud. -We will provide at least 30 days' notice before material changes take effect. +## 5. Contact -### 12.2 Explicit consent required - -Material changes to these Terms require your **explicit consent**. Continued use of the service after notification does **not** constitute acceptance. If you do not affirmatively accept updated Terms within 60 days of notification, your access to the Cloud service may be suspended until you accept the updated Terms or delete your account. - -### 12.3 Non-material changes - -Minor corrections, clarifications, or formatting changes that do not alter the substance of these Terms may be made without prior notice. - ---- - -## 13. Governing Law and Dispute Resolution - -### 13.1 Governing law - -These Terms shall be governed by and construed in accordance with the laws of **England and Wales**, without regard to conflict of law principles. - -### 13.2 Jurisdiction - -Any disputes arising out of or in connection with these Terms shall be subject to the exclusive jurisdiction of the courts of **England and Wales**. - -### 13.3 Informal resolution - -Before initiating formal proceedings, both parties agree to attempt to resolve disputes informally by contacting [hello@neurostack.sh](mailto:hello@neurostack.sh). We will make reasonable efforts to resolve complaints within 30 days. - ---- - -## 14. Age Restriction - -You must be at least **16 years of age** to create a NeuroStack account or use the Cloud service. This requirement is set in accordance with GDPR Article 8 and the UK Age Appropriate Design Code. If we become aware that a user is under 16, we will terminate their account and delete their data. - -If you are between 16 and 18 years of age, you confirm that you have obtained consent from a parent or guardian to use the service. - ---- - -## 15. General Provisions - -### 15.1 Entire agreement - -These Terms, together with the [Data Processing Agreement](DPA.md) and any applicable plan terms, constitute the entire agreement between you and SolidPlus LTD regarding the NeuroStack Cloud service. - -### 15.2 Severability - -If any provision of these Terms is found to be unenforceable, the remaining provisions shall remain in full force and effect. - -### 15.3 Waiver - -Failure to enforce any provision of these Terms does not constitute a waiver of that provision or any other provision. - -### 15.4 Assignment - -You may not assign your rights or obligations under these Terms without our prior written consent. We may assign our rights and obligations without restriction. - ---- - -## 16. Contact Information - -For questions about these Terms, the NeuroStack Cloud service, or to report abuse: - -- **Email:** [hello@neurostack.sh](mailto:hello@neurostack.sh) -- **Website:** [https://neurostack.sh](https://neurostack.sh) - **GitHub:** [https://github.com/raphasouthall/neurostack](https://github.com/raphasouthall/neurostack) - -**SolidPlus LTD** -A company registered in England and Wales. diff --git a/docs/api-contract-v2.md b/docs/api-contract-v2.md deleted file mode 100644 index 334b314..0000000 --- a/docs/api-contract-v2.md +++ /dev/null @@ -1,1125 +0,0 @@ -# NeuroStack Cloud API Contract v2 -- Sync & Data Transfer - -Covers all REST API changes needed to implement findings from: -- `neurostack-vault-sync-architecture-research.md` (sync triggers, memory merge, removed files bug) -- `neurostack-data-transfer-architecture-research.md` (tar.gz uploads, ETag caching, compression, push lock, db-version) - -## Table of Contents - -1. [Modified: POST /v1/vault/upload](#1-modified-post-v1vaultupload) -2. [New: POST /v1/vault/sync](#2-new-post-v1vaultsync) -3. [Modified: POST /v1/vault/query](#3-modified-post-v1vaultquery) -4. [New: GET /v1/vault/db-version](#4-new-get-v1vaultdb-version) -5. [New: POST /v1/vault/push-lock](#5-new-post-v1vaultpush-lock) -6. [Modified: GET /v1/vault/download](#6-modified-get-v1vaultdownload) -7. [New: GET /v1/vault/memories/since](#7-new-get-v1vaultmemoriessince) -8. [New: POST /v1/vault/upload/artifacts](#8-new-post-v1vaultuploadartifacts) -9. [New: DELETE /v1/vault/files](#9-new-delete-v1vaultfiles) -10. [CloudClient Method Signatures](#10-cloudclient-method-signatures) -11. [Breaking Changes Summary](#11-breaking-changes-summary) -12. [Migration Guide](#12-migration-guide) - ---- - -## Common Conventions - -**Authentication:** All endpoints require `Authorization: Bearer ` unless noted. - -**Error envelope:** -```json -{ - "detail": "Human-readable error message" -} -``` - -**Rate limiting headers** (returned on all authenticated responses): -``` -X-RateLimit-Limit: 1000 -X-RateLimit-Remaining: 994 -X-RateLimit-Reset: 1711497600 -``` - -**Tenant isolation:** All data is scoped to the authenticated `user_id`. No endpoint can access another user's data. - ---- - -## 1. Modified: POST /v1/vault/upload - -**Research finding:** Switch from multipart to tar.gz (breaks 32MB multipart limit), transmit removed files list (fixes ghost entry bug), support pre-computed artifacts to eliminate Gemini dependency. - -### Current Behavior -- Accepts `multipart/form-data` with individual `.md` files -- No removed files support -- No compression -- 32MB practical limit from Cloud Run body default - -### New Behavior -- Accepts `application/gzip` body (tar.gz archive) OR legacy `multipart/form-data` -- Includes removed files list and optional pre-computed artifacts via JSON metadata inside the archive -- Supports vaults up to 500MB compressed - -### Request - -**Option A: tar.gz upload (new, preferred)** - -``` -POST /v1/vault/upload -Content-Type: application/gzip -Content-Length: -Authorization: Bearer -X-Upload-Format: tar.gz - - -``` - -The tar.gz archive MUST contain a `_manifest.json` at the root with this schema: - -```json -{ - "format_version": 1, - "removed": ["old-note.md", "archive/deleted.md"], - "file_hashes": { - "notes/new-note.md": "sha256:abc123...", - "notes/changed-note.md": "sha256:def456..." - } -} -``` - -All other entries in the archive are the actual `.md` file contents, at paths matching `file_hashes` keys. - -**Option B: multipart upload (legacy, still supported)** - -``` -POST /v1/vault/upload -Content-Type: multipart/form-data -Authorization: Bearer - -files[]: (binary .md files) -removed: ["old-note.md"] (form field, JSON-encoded string list) -``` - -The `removed` field is a new optional form field. Omitting it preserves backward compatibility. - -### Response (unchanged) - -``` -HTTP 202 Accepted -``` - -```json -{ - "job_id": "550e8400-e29b-41d4-a716-446655440000", - "status": "queued", - "message": "Received 12 files for indexing, 3 files marked for removal" -} -``` - -### Error Responses - -| Status | Detail | Condition | -|--------|--------|-----------| -| 400 | `"Invalid tar.gz archive"` | Archive is corrupt or missing `_manifest.json` | -| 400 | `"Too many files: N exceeds limit of 5000"` | File count exceeded | -| 400 | `"Total upload size exceeds 500MB"` | Size limit exceeded | -| 400 | `"Only .md files accepted, got: X"` | Non-markdown file in archive | -| 400 | `"Absolute path not allowed: X"` | Path traversal attempt | -| 401 | `"Invalid or expired API key"` | Auth failure | -| 409 | `"Push already in progress. Lock held until "` | Concurrent push (see push-lock) | -| 429 | `"Note limit exceeded for tier: free"` | Tier quota hit | - -### Rate Limiting -- Free: 2 uploads/hour, 200 notes max -- Pro: 20 uploads/hour, 5000 notes max - -### Breaking Changes -- **None.** Multipart path is preserved. `X-Upload-Format: tar.gz` triggers new path. Clients without the header get legacy behavior. - ---- - -## 2. New: POST /v1/vault/sync - -**Research finding:** Orchestrate push + memory pull in one call. `neurostack cloud sync` CLI command needs a single round-trip that uploads changes AND fetches new memories created since last sync. - -### Request - -``` -POST /v1/vault/sync -Content-Type: application/gzip -Authorization: Bearer -X-Upload-Format: tar.gz - - -``` - -Query parameters: -- `memories_since` (optional, ISO 8601 timestamp): fetch memories created after this time. If omitted, no memories are returned. -- `wait` (optional, boolean, default `false`): if `true`, block until indexing completes (up to 300s). If `false`, return immediately with `job_id`. - -``` -POST /v1/vault/sync?memories_since=2026-03-25T10:00:00Z&wait=true -``` - -If there are no file changes to upload but the client wants memories only, send an empty tar.gz (just `_manifest.json` with empty `file_hashes` and empty `removed`). - -### Response - -**When `wait=false` (default):** - -``` -HTTP 202 Accepted -``` - -```json -{ - "job_id": "550e8400-e29b-41d4-a716-446655440000", - "status": "queued", - "message": "Received 5 files for indexing, 1 removed", - "memories": [ - { - "uuid": "mem-abc-123", - "content": "Rapha prefers kebab-case filenames", - "entity_type": "convention", - "tags": ["formatting"], - "created_at": "2026-03-25T14:30:00Z", - "updated_at": "2026-03-25T14:30:00Z", - "source_agent": "claude-code" - } - ], - "memories_count": 1, - "db_version": "gen-1711497600-abc123" -} -``` - -**When `wait=true` and indexing completes:** - -``` -HTTP 200 OK -``` - -```json -{ - "job_id": "550e8400-e29b-41d4-a716-446655440000", - "status": "complete", - "message": "Indexed 5 files, removed 1", - "db_version": "gen-1711497600-def456", - "db_size": 15728640, - "note_count": 458, - "memories": [], - "memories_count": 0 -} -``` - -**When `wait=true` and indexing times out:** - -``` -HTTP 202 Accepted -``` - -```json -{ - "job_id": "550e8400-e29b-41d4-a716-446655440000", - "status": "indexing", - "message": "Indexing still in progress after 300s", - "memories": [], - "memories_count": 0, - "db_version": null -} -``` - -### Error Responses - -| Status | Detail | Condition | -|--------|--------|-----------| -| 400 | `"Invalid tar.gz archive"` | Bad archive | -| 400 | `"Invalid memories_since timestamp"` | Unparseable ISO 8601 | -| 409 | `"Push already in progress"` | Concurrent push conflict | -| 429 | `"Rate limit exceeded"` | Tier limit hit | - -### Rate Limiting -- Same as `/v1/vault/upload` (counts as one upload) - ---- - -## 3. Modified: POST /v1/vault/query - -**Research finding:** Query-time memory union. Cloud search currently only queries SQLite. Memories live in Firestore but are invisible to search. Fix: merge Firestore memories into search results at query time. - -### Request (unchanged schema, new optional field) - -```json -{ - "query": "neurostack architecture decisions", - "top_k": 10, - "mode": "hybrid", - "depth": "auto", - "workspace": null, - "include_memories": true -} -``` - -New field: -- `include_memories` (boolean, default `true`): When true, the server queries Firestore memories in parallel with the SQLite search and merges results. When false, behaves identically to current behavior (SQLite only). - -### Response (extended) - -```json -{ - "triples": [...], - "summaries": [...], - "chunks": [...], - "depth_used": "summaries", - "memories": [ - { - "uuid": "mem-abc-123", - "content": "NeuroStack uses neuroscience-grounded retrieval", - "entity_type": "observation", - "tags": ["neurostack", "architecture"], - "relevance_score": 0.85, - "created_at": "2026-03-20T10:00:00Z" - } - ], - "db_version": "gen-1711497600-abc123" -} -``` - -New fields: -- `memories` (list): Firestore memories matching the query, ranked by relevance. Empty list when `include_memories=false`. -- `db_version` (string): Current database version for cache coherence. - -### Error Responses - -No new error codes. Firestore memory query failure is non-fatal -- the response returns with `memories: []` and logs the error server-side. - -### Breaking Changes -- **Additive only.** New fields (`memories`, `db_version`) are added to the response. Existing clients that don't read these fields are unaffected. -- `include_memories` defaults to `true`, which means existing clients will see slightly higher latency (~100ms) from the parallel Firestore query. Set to `false` to opt out. - ---- - -## 4. New: GET /v1/vault/db-version - -**Research finding:** Firestore version check for cross-instance cache coherence. When a push completes on one Cloud Run instance, other instances (and other devices) need to know the DB changed so they don't serve stale cached copies. - -### Request - -``` -GET /v1/vault/db-version -Authorization: Bearer -``` - -No request body. - -### Response - -``` -HTTP 200 OK -``` - -```json -{ - "db_version": "gen-1711497600-abc123", - "updated_at": "2026-03-26T14:30:00Z", - "db_size": 15728640, - "note_count": 458 -} -``` - -Fields: -- `db_version` (string): Opaque version identifier. Format: `gen--`. Changes on every successful push/index. -- `updated_at` (string, ISO 8601): When the DB was last updated. -- `db_size` (int): DB file size in bytes. -- `note_count` (int): Number of indexed notes. - -**When no DB exists yet:** - -``` -HTTP 404 Not Found -``` - -```json -{ - "detail": "No database found for this user" -} -``` - -### Error Responses - -| Status | Detail | Condition | -|--------|--------|-----------| -| 401 | `"Invalid or expired API key"` | Auth failure | -| 404 | `"No database found for this user"` | User has never pushed | - -### Rate Limiting -- No rate limit (Firestore read: ~$0.06/100K requests). Clients may poll this every 30-60s for staleness detection. - -### Implementation Notes -- Server writes `db_version` to Firestore doc `users/{user_id}/vault_meta/db_version` after every successful index job. -- Cloud Run instances check this before serving a cached SQLite DB. If the cached version mismatches, re-download from GCS. - ---- - -## 5. New: POST /v1/vault/push-lock - -**Research finding:** Server-side push lock prevents concurrent push conflicts from multiple devices. Uses Firestore for distributed locking. - -### Acquire Lock - -``` -POST /v1/vault/push-lock -Authorization: Bearer -Content-Type: application/json -``` - -```json -{ - "action": "acquire", - "ttl_seconds": 300, - "device_id": "laptop-home" -} -``` - -Fields: -- `action` (string, required): `"acquire"` or `"release"` -- `ttl_seconds` (int, default 300): Lock auto-expires after this duration (max 600) -- `device_id` (string, optional): Identifier for the device acquiring the lock. For diagnostics and stale lock identification. - -### Acquire Response - -**Success:** - -``` -HTTP 200 OK -``` - -```json -{ - "locked": true, - "lock_id": "lock-abc-123", - "expires_at": "2026-03-26T14:35:00Z", - "device_id": "laptop-home" -} -``` - -**Already locked by another device:** - -``` -HTTP 409 Conflict -``` - -```json -{ - "locked": false, - "held_by": "desktop-office", - "expires_at": "2026-03-26T14:32:00Z", - "detail": "Push lock held by another device until 2026-03-26T14:32:00Z" -} -``` - -### Release Lock - -``` -POST /v1/vault/push-lock -Authorization: Bearer -Content-Type: application/json -``` - -```json -{ - "action": "release", - "lock_id": "lock-abc-123" -} -``` - -### Release Response - -``` -HTTP 200 OK -``` - -```json -{ - "released": true -} -``` - -**Lock not found or not owned:** - -``` -HTTP 404 Not Found -``` - -```json -{ - "detail": "Lock not found or not owned by this user" -} -``` - -### Error Responses - -| Status | Detail | Condition | -|--------|--------|-----------| -| 400 | `"Invalid action, must be acquire or release"` | Bad action value | -| 400 | `"ttl_seconds must be between 1 and 600"` | TTL out of range | -| 401 | `"Invalid or expired API key"` | Auth failure | -| 404 | `"Lock not found or not owned by this user"` | Release of non-existent lock | -| 409 | `"Push lock held by another device"` | Lock contention | - -### Rate Limiting -- 10 requests/minute per user - -### Implementation Notes -- Firestore document: `users/{user_id}/locks/push` -- Uses Firestore transactions for atomic acquire (check-and-set) -- Expired locks are treated as unlocked (server checks `expires_at` on acquire) -- `/v1/vault/upload` and `/v1/vault/sync` should auto-acquire the lock if not already held, and release on completion - ---- - -## 6. Modified: GET /v1/vault/download - -**Research finding:** ETag caching to skip redundant downloads (90%+ requests skip download). Gzip compression for 46% bandwidth reduction. Expose `db_version` for cache coherence. - -### Request - -``` -GET /v1/vault/download -Authorization: Bearer -If-None-Match: "gen-1711497600-abc123" -Accept-Encoding: gzip -``` - -New request headers: -- `If-None-Match` (optional): The `db_version` value from a previous download or from `GET /v1/vault/db-version`. If the current DB matches this version, the server returns 304. -- `Accept-Encoding: gzip` (optional): If present, the presigned URL will point to a gzip-compressed copy of the DB. - -### Response -- DB has changed (or no ETag provided) - -``` -HTTP 200 OK -``` - -```json -{ - "download_url": "https://storage.googleapis.com/...", - "expires_in": 3600, - "db_version": "gen-1711497600-def456", - "db_size": 15728640, - "db_size_compressed": 8503296, - "compressed": true, - "note_count": 458 -} -``` - -New fields: -- `db_version` (string): Current version, to be stored by the client for future `If-None-Match`. -- `db_size` (int): Uncompressed DB size in bytes. -- `db_size_compressed` (int | null): Compressed size if `compressed=true`, null otherwise. -- `compressed` (bool): Whether the download URL points to a gzip file. -- `note_count` (int): Number of indexed notes. - -### Response -- DB unchanged (ETag match) - -``` -HTTP 304 Not Modified -``` - -```json -{ - "db_version": "gen-1711497600-abc123", - "message": "Database unchanged since last download" -} -``` - -No `download_url` is generated (saves GCS signed URL computation). - -### Error Responses - -| Status | Detail | Condition | -|--------|--------|-----------| -| 401 | `"Invalid or expired API key"` | Auth failure | -| 404 | `"No database found for this user"` | User has never pushed | - -### Rate Limiting -- No rate limit on the metadata check (304 path). The actual download goes directly to GCS. - -### Breaking Changes -- **Additive.** New fields in the 200 response body. The `download_url` and `expires_in` fields are unchanged. -- Clients not sending `If-None-Match` or `Accept-Encoding: gzip` get identical behavior to today. - ---- - -## 7. New: GET /v1/vault/memories/since - -**Research finding:** Fetch memories created after a timestamp, enabling local clients to pull new memories from Firestore and merge them into the local SQLite DB without a full re-index. - -### Request - -``` -GET /v1/vault/memories/since?after=2026-03-25T10:00:00Z&limit=100 -Authorization: Bearer -``` - -Query parameters: -- `after` (required, ISO 8601): Return memories created or updated after this timestamp. -- `limit` (optional, int, default 100, max 500): Maximum number of memories to return. -- `include_deleted` (optional, bool, default false): If true, include memories deleted after `after` (for local cache invalidation). Deleted memories have `"deleted": true`. - -### Response - -``` -HTTP 200 OK -``` - -```json -{ - "memories": [ - { - "uuid": "mem-abc-123", - "content": "NeuroStack uses debounced file watching", - "entity_type": "observation", - "tags": ["neurostack", "sync"], - "created_at": "2026-03-25T14:30:00Z", - "updated_at": "2026-03-25T14:30:00Z", - "source_agent": "claude-code", - "session_id": 42, - "workspace": null, - "ttl_hours": null, - "deleted": false - } - ], - "count": 1, - "has_more": false, - "server_time": "2026-03-26T15:00:00Z" -} -``` - -Fields: -- `memories` (list): Memories matching the time filter, ordered by `updated_at` ascending. -- `count` (int): Number of memories in this response. -- `has_more` (bool): If true, there are more memories after the last one in this response. Client should paginate using the last memory's `updated_at` as the new `after`. -- `server_time` (string, ISO 8601): Server's current time. Client should store this as the `after` value for the next sync. - -### Error Responses - -| Status | Detail | Condition | -|--------|--------|-----------| -| 400 | `"Missing required parameter: after"` | No timestamp provided | -| 400 | `"Invalid timestamp format"` | Unparseable ISO 8601 | -| 401 | `"Invalid or expired API key"` | Auth failure | - -### Rate Limiting -- 30 requests/minute per user - ---- - -## 8. New: POST /v1/vault/upload/artifacts - -**Research finding:** Client-side indexing with Ollama eliminates Gemini API costs (97-99% of total cost). Clients push pre-computed embeddings, summaries, and triples alongside vault files. - -### Request - -``` -POST /v1/vault/upload/artifacts -Content-Type: application/gzip -Authorization: Bearer -X-Upload-Format: tar.gz -``` - -The tar.gz archive contains a `_artifacts.json` manifest and the artifact data files: - -```json -{ - "format_version": 1, - "embed_model": "nomic-embed-text", - "embed_dimensions": 768, - "llm_model": "phi3.5", - "artifacts": { - "notes/my-note.md": { - "sha256": "abc123...", - "summary": "This note discusses the architecture of...", - "triples": [ - {"subject": "NeuroStack", "predicate": "uses", "object": "SQLite"} - ], - "chunks": [ - { - "text": "NeuroStack is a neuroscience-grounded...", - "embedding": [0.123, -0.456, ...], - "start_line": 1, - "end_line": 15 - } - ] - } - } -} -``` - -Fields in `_artifacts.json`: -- `format_version` (int): Schema version. Currently `1`. -- `embed_model` (string): Name of the embedding model used (for compatibility validation). -- `embed_dimensions` (int): Embedding vector dimensions. -- `llm_model` (string): Name of the LLM used for summaries/triples. -- `artifacts` (dict): Keyed by note relative path. Each entry contains: - - `sha256` (string): Content hash of the source `.md` file. Server uses this to verify the artifact matches the uploaded file. - - `summary` (string): AI-generated note summary. - - `triples` (list): SPO triples extracted from the note. - - `chunks` (list): Text chunks with embeddings. Each chunk has `text`, `embedding` (float array), `start_line`, `end_line`. - -### Response - -``` -HTTP 202 Accepted -``` - -```json -{ - "job_id": "550e8400-e29b-41d4-a716-446655440000", - "status": "queued", - "message": "Received artifacts for 12 notes", - "skipped_reindex": 12, - "gemini_calls_saved": 12 -} -``` - -Fields: -- `skipped_reindex` (int): Number of notes that will skip Gemini re-indexing because valid artifacts were provided. -- `gemini_calls_saved` (int): Number of Gemini API calls saved. - -### Error Responses - -| Status | Detail | Condition | -|--------|--------|-----------| -| 400 | `"Invalid artifact archive"` | Missing `_artifacts.json` or corrupt archive | -| 400 | `"Embedding dimension mismatch: expected 768, got 384"` | Incompatible embedding model | -| 400 | `"SHA256 mismatch for notes/my-note.md"` | Artifact does not match uploaded file content | -| 401 | `"Invalid or expired API key"` | Auth failure | -| 413 | `"Artifact archive too large (max 200MB)"` | Size limit exceeded | -| 429 | `"Rate limit exceeded"` | Tier quota hit | - -### Rate Limiting -- Same as `/v1/vault/upload` (counts as one upload) - -### Usage Notes -- This endpoint is typically called AFTER `/v1/vault/upload` (or combined via `/v1/vault/sync`). The server matches artifacts to already-uploaded files by `sha256` hash. -- Alternatively, the tar.gz sent to `/v1/vault/upload` can include an `_artifacts.json` alongside `_manifest.json`. The server will detect and process both in a single upload. This is the preferred approach for the combined flow. -- Notes without matching artifacts fall back to server-side Gemini indexing. - ---- - -## 9. New: DELETE /v1/vault/files - -**Research finding:** While `/v1/vault/upload` now accepts a `removed` list, a standalone delete endpoint is useful for git hook integrations where a commit only deletes files (no additions/changes). - -### Request - -``` -DELETE /v1/vault/files -Content-Type: application/json -Authorization: Bearer -``` - -```json -{ - "files": ["old-note.md", "archive/deleted.md"] -} -``` - -Fields: -- `files` (list[string], required): Relative paths to remove from the cloud index. Max 500. - -### Response - -``` -HTTP 200 OK -``` - -```json -{ - "removed": 2, - "db_version": "gen-1711497600-abc123" -} -``` - -### Error Responses - -| Status | Detail | Condition | -|--------|--------|-----------| -| 400 | `"files list is required and must be non-empty"` | Missing or empty list | -| 400 | `"Too many files: N exceeds limit of 500"` | List too long | -| 400 | `"Path traversal not allowed: .."` | Path traversal attempt | -| 401 | `"Invalid or expired API key"` | Auth failure | -| 404 | `"No database found for this user"` | No index exists | - -### Rate Limiting -- 10 requests/minute per user - ---- - -## 10. CloudClient Method Signatures - -All methods below are additions or modifications to `neurostack/cloud/client.py::CloudClient`. - -### New/Modified Methods - -```python -class CloudClient: - - # --- Modified --- - - def upload_vault( - self, - files: dict[str, bytes], - *, - removed: list[str] | None = None, - artifacts: dict[str, dict] | None = None, - use_tar: bool = True, - timeout: float = 300.0, - ) -> dict: - """Upload vault files to the cloud. - - Args: - files: Mapping of relative_path -> file content bytes. - removed: List of relative paths to remove from cloud index. - artifacts: Pre-computed artifacts keyed by note path. - Each value has keys: sha256, summary, triples, chunks. - use_tar: If True (default), pack into tar.gz. If False, use legacy - multipart upload for backward compatibility. - timeout: Request timeout in seconds. - - Returns: - {"job_id": "...", "status": "queued", "message": "..."} - """ - - # --- New --- - - def sync( - self, - files: dict[str, bytes], - *, - removed: list[str] | None = None, - artifacts: dict[str, dict] | None = None, - memories_since: str | None = None, - wait: bool = False, - timeout: float = 300.0, - ) -> dict: - """Upload changes and fetch new memories in one call. - - Args: - files: Mapping of relative_path -> file content bytes. - removed: List of relative paths removed from vault. - artifacts: Pre-computed indexing artifacts (summaries, triples, - embeddings) keyed by note path. - memories_since: ISO 8601 timestamp. Fetch memories created after - this time. None = don't fetch memories. - wait: If True, block until indexing completes (up to 300s). - timeout: Request timeout in seconds. - - Returns: - { - "job_id": "...", - "status": "queued" | "complete" | "indexing", - "memories": [...], - "memories_count": N, - "db_version": "..." - } - """ - - def get_db_version(self) -> dict: - """Check the current database version for cache coherence. - - Returns: - { - "db_version": "gen-...", - "updated_at": "2026-...", - "db_size": 15728640, - "note_count": 458 - } - - Raises: - FileNotFoundError: User has never pushed a vault. - """ - - def acquire_push_lock( - self, - *, - ttl_seconds: int = 300, - device_id: str | None = None, - ) -> dict: - """Acquire a per-user push lock. - - Args: - ttl_seconds: Lock auto-expires after this many seconds (max 600). - device_id: Identifier for the pushing device. - - Returns: - {"locked": True, "lock_id": "...", "expires_at": "..."} - - Raises: - httpx.HTTPStatusError: 409 if lock is held by another device. - """ - - def release_push_lock(self, lock_id: str) -> dict: - """Release a previously acquired push lock. - - Args: - lock_id: The lock_id returned by acquire_push_lock. - - Returns: - {"released": True} - """ - - def download_db( - self, - *, - db_version: str | None = None, - accept_gzip: bool = True, - ) -> dict: - """Get download URL with ETag caching and gzip support. - - Args: - db_version: If provided, sent as If-None-Match. Returns - {"not_modified": True} if DB hasn't changed. - accept_gzip: If True, request gzip-compressed DB. - - Returns: - { - "download_url": "https://...", - "db_version": "gen-...", - "compressed": True, - "db_size": 15728640 - } - OR {"not_modified": True, "db_version": "gen-..."} on 304. - """ - - def get_memories_since( - self, - after: str, - *, - limit: int = 100, - include_deleted: bool = False, - ) -> dict: - """Fetch memories created after a timestamp. - - Args: - after: ISO 8601 timestamp. - limit: Max memories to return (1-500). - include_deleted: Include deleted memories for cache invalidation. - - Returns: - { - "memories": [...], - "count": N, - "has_more": False, - "server_time": "2026-..." - } - """ - - def upload_artifacts( - self, - artifacts: dict[str, dict], - *, - embed_model: str = "nomic-embed-text", - embed_dimensions: int = 768, - llm_model: str = "phi3.5", - timeout: float = 300.0, - ) -> dict: - """Upload pre-computed indexing artifacts. - - Args: - artifacts: Mapping of note_path -> {sha256, summary, triples, chunks}. - embed_model: Name of embedding model used. - embed_dimensions: Embedding vector dimensionality. - llm_model: Name of LLM used for summaries/triples. - timeout: Request timeout in seconds. - - Returns: - {"job_id": "...", "skipped_reindex": N, "gemini_calls_saved": N} - """ - - def delete_files(self, files: list[str]) -> dict: - """Remove files from the cloud index. - - Args: - files: List of relative paths to remove. - - Returns: - {"removed": N, "db_version": "gen-..."} - """ -``` - -### Modified VaultSyncEngine Methods - -```python -class VaultSyncEngine: - - def push( - self, - *, - progress_callback: Callable[[str], None] | None = None, - include_artifacts: bool = False, - ) -> dict: - """Upload changed vault files and wait for indexing. - - Changes from current: - - Transmits diff.removed to the server via _manifest.json - - Uses tar.gz format instead of multipart - - Optionally includes pre-computed artifacts - - Args: - progress_callback: Called with status messages. - include_artifacts: If True, include local pre-computed - embeddings/summaries/triples in the upload. - - Returns: - {"status": "complete", "job_id": "...", ...} - """ - - def pull( - self, - *, - db_path: Path | None = None, - cached_version: str | None = None, - ) -> Path | None: - """Download indexed DB from cloud with ETag caching. - - Changes from current: - - Sends If-None-Match with cached_version - - Accepts gzip-compressed downloads - - Returns None if DB is unchanged (304) - - Args: - db_path: Target path for the DB file. - cached_version: db_version from previous download. - If DB hasn't changed, returns None. - - Returns: - Path to downloaded DB, or None if unchanged. - """ - - def sync( - self, - *, - progress_callback: Callable[[str], None] | None = None, - memories_since: str | None = None, - include_artifacts: bool = False, - wait: bool = True, - ) -> dict: - """Push changes and pull memories in one operation. - - New method that orchestrates the full sync lifecycle: - 1. Scan vault and compute diff - 2. Acquire push lock - 3. Upload changes via POST /v1/vault/sync - 4. Optionally wait for indexing - 5. Release push lock - 6. Save manifest on success - 7. Return job result + memories - - Args: - progress_callback: Called with status messages. - memories_since: Fetch memories created after this timestamp. - include_artifacts: Include pre-computed artifacts. - wait: Block until indexing completes. - - Returns: - { - "status": "complete" | "queued", - "job_id": "...", - "memories": [...], - "db_version": "..." - } - """ -``` - ---- - -## 11. Breaking Changes Summary - -| Endpoint | Change | Breaking? | Migration | -|----------|--------|-----------|-----------| -| `POST /v1/vault/upload` | New tar.gz format, `removed` field | **No** | Multipart still works. New format activated by `X-Upload-Format: tar.gz` header. | -| `POST /v1/vault/query` | New `memories` and `db_version` fields in response | **No** | Additive. Existing clients ignore new fields. | -| `POST /v1/vault/query` | New `include_memories` request field | **No** | Defaults to `true`. Set to `false` to preserve old behavior and latency. | -| `GET /v1/vault/download` | New response fields, 304 support | **No** | Additive. Clients not sending `If-None-Match` get full behavior. | -| All new endpoints | New endpoints | **No** | Existing clients never call them. | - -**Zero breaking changes.** All modifications are additive or opt-in via new headers/fields. - ---- - -## 12. Migration Guide - -### Client Upgrade Path - -**Phase 1 -- Immediate (no server changes needed for existing clients):** -1. Server deploys new endpoints. Old clients continue working. -2. New client versions start using `removed` field in uploads. - -**Phase 2 -- Opt-in improvements:** -1. Clients switch to tar.gz uploads for large vaults (>32MB uncompressed). -2. Clients start sending `If-None-Match` on downloads. -3. Clients call `GET /v1/vault/db-version` for staleness checks. - -**Phase 3 -- Full sync:** -1. Clients switch from `push()` + `pull()` to `sync()`. -2. Clients implement local memory merge from `memories_since`. -3. Clients generate artifacts locally and skip Gemini costs. - -### Version Negotiation - -The client should check the server version before using new features: - -```python -health = client.health() -server_version = health.get("version", "0.8.0") - -# tar.gz upload requires server >= 0.9.0 -if parse_version(server_version) >= parse_version("0.9.0"): - client.upload_vault(files, removed=removed, use_tar=True) -else: - client.upload_vault(files, use_tar=False) # legacy multipart -``` - -### Server Version Header - -All responses include: -``` -X-NeuroStack-Version: 0.9.0 -``` - -Clients can use this to detect feature availability without a separate health check. - ---- - -## Appendix A: Endpoint Summary Table - -| Method | Path | Status | Research Finding | -|--------|------|--------|-----------------| -| POST | `/v1/vault/upload` | Modified | tar.gz upload, removed files bug fix, artifact support | -| POST | `/v1/vault/sync` | **New** | Orchestrated push + memory pull | -| POST | `/v1/vault/query` | Modified | Query-time Firestore memory merge | -| GET | `/v1/vault/db-version` | **New** | Firestore version for cache coherence | -| POST | `/v1/vault/push-lock` | **New** | Concurrent push safety | -| GET | `/v1/vault/download` | Modified | ETag caching, gzip, db_version | -| GET | `/v1/vault/memories/since` | **New** | Incremental memory pull | -| POST | `/v1/vault/upload/artifacts` | **New** | Client-side indexing, Gemini cost elimination | -| DELETE | `/v1/vault/files` | **New** | Standalone file deletion for git hooks | - -## Appendix B: Firestore Schema Additions - -``` -users/{user_id}/ - vault_meta/ - db_version # {version: "gen-...", updated_at: timestamp, db_size: int, note_count: int} - locks/ - push # {lock_id: str, device_id: str, expires_at: timestamp, user_id: str} -``` - -## Appendix C: GCS Layout Changes - -``` -vaults/{user_id}/ - neurostack.db # existing: uncompressed SQLite - neurostack.db.gz # new: gzip-compressed SQLite (written alongside .db after index) -``` - -Both files are written on every successful index. The download endpoint serves `.db.gz` when client sends `Accept-Encoding: gzip`, `.db` otherwise. diff --git a/manifest.json b/manifest.json index 113195d..f496bac 100644 --- a/manifest.json +++ b/manifest.json @@ -2,7 +2,7 @@ "manifest_version": "0.4", "name": "neurostack", "display_name": "NeuroStack", - "version": "0.13.0", + "version": "0.15.0", "description": "Neuroscience-grounded knowledge management for your Markdown vault. Semantic search, knowledge graphs, AI memory, stale note detection, and 21 MCP tools.", "author": { "name": "Raphael Southall" @@ -32,14 +32,14 @@ "mode": { "type": "string", "title": "Mode", - "description": "lite = FTS5 search only (~130MB). full = adds embeddings + summaries via Ollama (~560MB). cloud = use NeuroStack Cloud for indexing.", + "description": "lite = FTS5 search only (~130MB). full = adds embeddings + summaries via Ollama (~560MB).", "default": "lite", - "enum": ["lite", "full", "cloud"] + "enum": ["lite", "full"] }, "api_key": { "type": "string", - "title": "Cloud API Key", - "description": "NeuroStack Cloud API key (only needed for cloud mode). Get one at https://app.neurostack.sh", + "title": "API Key", + "description": "Optional API key required to authenticate to the local OpenAI-compatible HTTP API (NEUROSTACK_API_KEY).", "required": false, "sensitive": true }, @@ -185,8 +185,8 @@ "type": "git", "url": "https://github.com/raphasouthall/neurostack" }, - "homepage": "https://neurostack.sh", - "documentation": "https://neurostack.sh", + "homepage": "https://github.com/raphasouthall/neurostack", + "documentation": "https://github.com/raphasouthall/neurostack#readme", "license": "Apache-2.0", "keywords": [ "knowledge-management", @@ -203,7 +203,7 @@ "privacy_policies": [ { "name": "Privacy Policy", - "url": "https://neurostack.sh/privacy" + "url": "https://github.com/raphasouthall/neurostack/blob/main/PRIVACY.md" } ] } diff --git a/npm/README.md b/npm/README.md index b899a21..445feae 100644 --- a/npm/README.md +++ b/npm/README.md @@ -42,7 +42,6 @@ NEUROSTACK_MODE=lite npm install -g neurostack ## Links - [GitHub](https://github.com/raphasouthall/neurostack) -- [Website](https://neurostack.sh) - [PyPI](https://pypi.org/project/neurostack/) ## License diff --git a/npm/package.json b/npm/package.json index d06e411..d979d2d 100644 --- a/npm/package.json +++ b/npm/package.json @@ -1,6 +1,6 @@ { "name": "neurostack", - "version": "0.14.0", + "version": "0.15.0", "description": "Build, maintain, and search your knowledge vault with AI", "bin": { "neurostack": "bin/neurostack.js", @@ -23,14 +23,14 @@ "neuroscience", "cli" ], - "author": "Raphael Southall ", + "author": "Raphael Southall", "license": "Apache-2.0", "repository": { "type": "git", "url": "https://github.com/raphasouthall/neurostack.git" }, "mcpName": "io.github.raphasouthall/neurostack", - "homepage": "https://neurostack.sh", + "homepage": "https://github.com/raphasouthall/neurostack", "bugs": { "url": "https://github.com/raphasouthall/neurostack/issues" }, diff --git a/pyproject.toml b/pyproject.toml index 13a4d3e..27d0b65 100644 --- a/pyproject.toml +++ b/pyproject.toml @@ -1,6 +1,6 @@ [project] name = "neurostack" -version = "0.14.0" +version = "0.15.0" description = "Build, maintain, and search your knowledge vault. CLI + MCP server with stale note detection, semantic search, and neuroscience-grounded memory." readme = "README.md" license = "Apache-2.0" @@ -59,8 +59,8 @@ api = [ neurostack = "neurostack.cli:main" [project.urls] -Homepage = "https://neurostack.sh" -Documentation = "https://neurostack.sh" +Homepage = "https://github.com/raphasouthall/neurostack" +Documentation = "https://github.com/raphasouthall/neurostack#readme" Repository = "https://github.com/raphasouthall/neurostack" Issues = "https://github.com/raphasouthall/neurostack/issues" diff --git a/server.json b/server.json index ddedaf6..910644c 100644 --- a/server.json +++ b/server.json @@ -7,19 +7,13 @@ "url": "https://github.com/raphasouthall/neurostack", "source": "github" }, - "version": "0.13.0", - "websiteUrl": "https://neurostack.sh", - "remotes": [ - { - "type": "streamable-http", - "url": "https://mcp.neurostack.sh/mcp" - } - ], + "version": "0.15.0", + "websiteUrl": "https://github.com/raphasouthall/neurostack", "packages": [ { "registryType": "npm", "identifier": "neurostack", - "version": "0.13.0", + "version": "0.15.0", "transport": { "type": "stdio" }, diff --git a/src/neurostack/cli/__init__.py b/src/neurostack/cli/__init__.py index 46cf9bf..6a94ed8 100644 --- a/src/neurostack/cli/__init__.py +++ b/src/neurostack/cli/__init__.py @@ -9,7 +9,6 @@ from .. import __version__ from ..config import get_config from .api import cmd_api, cmd_bundle, cmd_serve -from .cloud import cmd_cloud from .index import cmd_backfill, cmd_index, cmd_reembed_chunks, cmd_watch from .memories import cmd_memories from .search import ( @@ -73,10 +72,6 @@ def main(): "--mode", "-m", choices=["lite", "full"], help="Installation mode (lite=FTS5 only, full=+ML+communities)", ) - p.add_argument( - "--cloud", action="store_true", default=False, - help="Use cloud mode (Gemini indexing)", - ) p.add_argument( "--index", action="store_true", default=True, help="Index vault after init (default: true)", @@ -135,71 +130,6 @@ def main(): p = sub.add_parser("status", help="Show NeuroStack status") p.set_defaults(func=cmd_status) - # cloud - p = sub.add_parser("cloud", help="Manage NeuroStack Cloud authentication") - cloud_sub = p.add_subparsers(dest="cloud_command") - - cp = cloud_sub.add_parser("login", help="Authenticate with an API key") - cp.add_argument("--key", "-k", help="API key (or prompted interactively)") - - cloud_sub.add_parser("logout", help="Clear stored cloud credentials") - - cp = cloud_sub.add_parser("status", help="Show cloud auth state and usage") - cp.add_argument("--json", action="store_true", default=False, help="Output as JSON") - - cloud_sub.add_parser("setup", help="Interactive cloud endpoint and key configuration") - - cloud_sub.add_parser("consent", help="Grant privacy consent for cloud features") - - cp = cloud_sub.add_parser("push", help="Upload vault files for cloud indexing") - cp.add_argument("--json", action="store_true", default=False, help="Output as JSON") - - cp = cloud_sub.add_parser("pull", help="Download indexed database from cloud") - cp.add_argument("--json", action="store_true", default=False, help="Output as JSON") - - cp = cloud_sub.add_parser("query", help="Search vault via cloud API") - cp.add_argument("query", help="Search text") - cp.add_argument("--top-k", type=int, default=10, help="Number of results") - cp.add_argument( - "--depth", default="auto", - choices=["triples", "summaries", "full", "auto"], - help="Result depth (default: auto)", - ) - cp.add_argument( - "--mode", default="hybrid", - choices=["hybrid", "semantic", "keyword"], - help="Search mode (default: hybrid)", - ) - cp.add_argument("--workspace", "-w", help="Scope to vault subdirectory") - cp.add_argument("--json", action="store_true", default=False, help="Output as JSON") - - cp = cloud_sub.add_parser("triples", help="Search knowledge graph triples via cloud") - cp.add_argument("query", help="Search text") - cp.add_argument("--top-k", type=int, default=10, help="Number of results") - cp.add_argument("--workspace", "-w", help="Scope to vault subdirectory") - cp.add_argument("--json", action="store_true", default=False, help="Output as JSON") - - cp = cloud_sub.add_parser("summary", help="Get note summary from cloud") - cp.add_argument("note_path", help="Note path (e.g. research/my-note.md)") - cp.add_argument("--json", action="store_true", default=False, help="Output as JSON") - - cp = cloud_sub.add_parser("sync", help="Push vault changes and fetch new memories") - cp.add_argument("--json", action="store_true", default=False, help="Output as JSON") - cp.add_argument("--quiet", "-q", action="store_true", default=False, help="Suppress output") - - cloud_sub.add_parser("install-hooks", help="Install git hooks for automatic cloud sync") - cloud_sub.add_parser("uninstall-hooks", help="Remove git hooks for cloud sync") - cloud_sub.add_parser("hooks-status", help="Check git hook installation status") - - cp = cloud_sub.add_parser("auto-sync", help="Manage automatic periodic sync") - auto_sub = cp.add_subparsers(dest="auto_sync_command") - ap = auto_sub.add_parser("enable", help="Enable periodic sync via systemd timer") - ap.add_argument("--interval", default="15min", help="Sync interval (default: 15min)") - auto_sub.add_parser("disable", help="Disable periodic sync") - auto_sub.add_parser("status", help="Show auto-sync status") - - p.set_defaults(func=cmd_cloud) - # memories p = sub.add_parser("memories", help="Manage agent-written memories") mem_sub = p.add_subparsers(dest="memories_command") @@ -695,10 +625,6 @@ def main(): # watch p = sub.add_parser("watch", help="Watch vault for changes") - p.add_argument( - "--cloud", action="store_true", default=False, - help="Enable automatic cloud push after idle period (60s)", - ) p.set_defaults(func=cmd_watch) args = parser.parse_args() @@ -708,7 +634,7 @@ def main(): # Preflight: nudge user to run init if vault doesn't exist yet _skip_preflight = { - "init", "install", "uninstall", "doctor", "status", "demo", "update", "cloud", + "init", "install", "uninstall", "doctor", "status", "demo", "update", "setup-desktop", "setup-client", "bundle", } vault_path = Path(args.vault) diff --git a/src/neurostack/cli/cloud.py b/src/neurostack/cli/cloud.py deleted file mode 100644 index 94f26e9..0000000 --- a/src/neurostack/cli/cloud.py +++ /dev/null @@ -1,607 +0,0 @@ -# SPDX-License-Identifier: Apache-2.0 -# Copyright (c) 2024-2026 Raphael Southall -"""Cloud CLI commands.""" - -import json -import sys -import time -import webbrowser - -import httpx - -from ..cloud.client import CloudClient -from ..cloud.config import ( - CloudConfig, - clear_cloud_credentials, - load_cloud_config, - save_cloud_config, - save_consent, -) -from ..config import get_config - - -def _cmd_cloud_device_login() -> None: - """Authenticate via OAuth device code flow (browser handoff).""" - cfg = load_cloud_config() - cloud_url = cfg.cloud_api_url or "https://neurostack-api-911077737485.us-central1.run.app" - base = cloud_url.rstrip("/") - - # Step 1: Request device code - try: - resp = httpx.post(f"{base}/api/v1/auth/device-code", timeout=15.0) - resp.raise_for_status() - except httpx.ConnectError: - print(f" Error: Cannot reach cloud API at {base}.") - sys.exit(1) - except httpx.HTTPStatusError as e: - print(f" Error: Device code request failed ({e.response.status_code}).") - sys.exit(1) - - data = resp.json() - device_code = data["device_code"] - user_code = data["user_code"] - verification_uri = data.get("verification_uri", f"{base}/device") - expires_in = data.get("expires_in", 600) - interval = data.get("interval", 5) - - # Step 2: Show code and open browser (code passed via URL for auto-fill) - verification_url = f"{verification_uri}?code={user_code}" - print("\n Opening browser to sign in...") - print(" If the browser doesn't open, visit:") - print(f" URL: {verification_url}") - print(f" Code: \033[1m{user_code}\033[0m\n") - - try: - webbrowser.open(verification_url) - except Exception: - pass # Non-fatal if browser fails to open - - # Step 3: Poll for token - deadline = time.monotonic() + expires_in - dots = 0 - while time.monotonic() < deadline: - time.sleep(interval) - dots = (dots + 1) % 4 - spinner = "." * (dots + 1) - print(f"\r Waiting for browser authorization{spinner} ", end="", flush=True) - - try: - token_resp = httpx.post( - f"{base}/api/v1/auth/device-token", - json={"device_code": device_code}, - timeout=15.0, - ) - except (httpx.ConnectError, httpx.TimeoutException): - continue # Retry on transient errors - - if token_resp.status_code == 428: - # Authorization pending -- keep polling - continue - elif token_resp.status_code == 200: - token_data = token_resp.json() - api_key = token_data["api_key"] - save_cloud_config(cloud_api_url=cloud_url, cloud_api_key=api_key) - print("\r Login successful! API key stored in config. ") - return - elif token_resp.status_code == 400: - print("\r Code expired. Please try again. ") - sys.exit(1) - else: - status = token_resp.status_code - print(f"\r Unexpected error ({status}). Please try again.") - sys.exit(1) - - print("\r Authorization timed out. Please try again. ") - sys.exit(1) - - -def _ensure_cloud_auth(): - """Check cloud auth, trigger login if missing. Returns config.""" - cloud_cfg = load_cloud_config() - if not cloud_cfg.cloud_api_url or not cloud_cfg.cloud_api_key: - if sys.stdin.isatty(): - print(" Not logged in. Starting login...") - print() - _cmd_cloud_device_login() - cloud_cfg = load_cloud_config() - if not cloud_cfg.cloud_api_key: - print("Error: Login failed.", - file=sys.stderr) - sys.exit(1) - print() - else: - print( - "Error: Not authenticated." - " Run `neurostack cloud login` first.", - file=sys.stderr, - ) - sys.exit(1) - return cloud_cfg - - -def cmd_cloud(args): - """Manage cloud authentication and configuration.""" - subcmd = getattr(args, "cloud_command", None) - - if subcmd == "login": - api_key = getattr(args, "key", None) - if api_key: - # Direct API key login (--key flag) - cfg = load_cloud_config() - cloud_url = cfg.cloud_api_url or "https://neurostack-api-911077737485.us-central1.run.app" - test_cfg = CloudConfig(cloud_api_url=cloud_url, cloud_api_key=api_key) - client = CloudClient(test_cfg) - try: - if client.validate_key(): - save_cloud_config(cloud_api_url=cloud_url, cloud_api_key=api_key) - print(f" Logged in to {cloud_url}") - else: - print(" Error: Invalid API key.") - sys.exit(1) - except ConnectionError as e: - print(f" Error: {e}") - sys.exit(1) - else: - # Device code flow (browser-based login) - _cmd_cloud_device_login() - - elif subcmd == "logout": - clear_cloud_credentials() - print(" Logged out. Cloud credentials cleared.") - - elif subcmd == "status": - cfg = load_cloud_config() - client = CloudClient(cfg) - - result = { - "authenticated": client.is_configured, - "cloud_url": cfg.cloud_api_url or "(not configured)", - } - - if client.is_configured: - try: - status = client.status() - result.update(status) - except (ConnectionError, Exception): - result["connection"] = "unreachable" - - if args.json: - print(json.dumps(result, indent=2)) - else: - if result["authenticated"]: - print(" Status: Authenticated") - print(f" Cloud: {result['cloud_url']}") - tier = result.get("tier", "unknown") - print(f" Tier: {tier}") - if result.get("connection") == "unreachable": - print(" Note: Cloud API unreachable (credentials stored locally)") - # Staleness indicator - try: - from ..cloud.sync import VaultSyncEngine - ns_cfg = get_config() - sync_engine = VaultSyncEngine( - cloud_api_url=cfg.cloud_api_url, - cloud_api_key=cfg.cloud_api_key, - vault_root=ns_cfg.vault_root, - db_dir=ns_cfg.db_dir, - ) - staleness = sync_engine.get_staleness() - if staleness["is_stale"]: - parts = [] - if staleness["stale_files_count"]: - parts.append(f"{staleness['stale_files_count']} files changed") - if staleness["behind_hours"] is not None: - parts.append(f"{staleness['behind_hours']} hours behind") - elif staleness["last_sync"] is None: - parts.append("never synced") - detail = ", ".join(parts) if parts else "changes detected" - print(f" Sync: Stale \u2014 {detail}") - else: - print(f" Sync: Up to date (last sync: {staleness['last_sync']})") - except Exception: - pass # Non-critical - else: - print(" Status: Not authenticated") - url = result["cloud_url"] - print(f" Cloud: {url}") - print(" Run: neurostack cloud login") - - elif subcmd == "setup": - cfg = load_cloud_config() - default_url = cfg.cloud_api_url or "https://neurostack-api-911077737485.us-central1.run.app" - - url = input(f" Cloud API URL [{default_url}]: ").strip() or default_url - api_key = input(" API key: ").strip() - if not api_key: - print(" Error: API key is required.") - sys.exit(1) - - # Validate - test_cfg = CloudConfig(cloud_api_url=url, cloud_api_key=api_key) - client = CloudClient(test_cfg) - try: - if client.validate_key(): - save_cloud_config(cloud_api_url=url, cloud_api_key=api_key) - print(f" Cloud configured: {url}") - print(" Authenticated successfully.") - else: - print(" Error: Invalid API key.") - sys.exit(1) - except ConnectionError as e: - print(f" Error: {e}") - sys.exit(1) - - elif subcmd == "push": - cmd_cloud_push(args) - - elif subcmd == "pull": - cmd_cloud_pull(args) - - elif subcmd == "sync": - cmd_cloud_sync(args) - - elif subcmd == "query": - cmd_cloud_query(args) - - elif subcmd == "triples": - cmd_cloud_triples(args) - - elif subcmd == "summary": - cmd_cloud_summary(args) - - elif subcmd == "consent": - prompt = ( - "Your vault content will be sent to Google's Gemini API for indexing. " - "This includes note text, which is processed to generate embeddings, " - "summaries, and knowledge graph triples. Continue? [y/N] " - ) - answer = input(prompt).strip().lower() - if answer in ("y", "yes"): - save_consent() - print("Consent granted.") - else: - print("Consent not granted. Cloud features require consent.") - sys.exit(1) - - elif subcmd == "install-hooks": - from ..cloud.hooks import install_hooks - ns_cfg = get_config() - try: - result = install_hooks(ns_cfg.vault_root) - if result["installed"]: - print(f" Installed: {', '.join(result['installed'])}") - if result["skipped"]: - print(f" Already installed: {', '.join(result['skipped'])}") - print(f" Git dir: {result['git_dir']}") - except ValueError as e: - print(f"Error: {e}", file=sys.stderr) - sys.exit(1) - - elif subcmd == "uninstall-hooks": - from ..cloud.hooks import uninstall_hooks - ns_cfg = get_config() - try: - result = uninstall_hooks(ns_cfg.vault_root) - if result["removed"]: - print(f" Removed: {', '.join(result['removed'])}") - if result["not_found"]: - print(f" Not found: {', '.join(result['not_found'])}") - except ValueError as e: - print(f"Error: {e}", file=sys.stderr) - sys.exit(1) - - elif subcmd == "hooks-status": - from ..cloud.hooks import hooks_status - ns_cfg = get_config() - status = hooks_status(ns_cfg.vault_root) - if not status["git_repo"]: - print(" Vault is not a git repository") - else: - for hook in ("post_commit", "post_merge"): - name = hook.replace("_", "-") - state = "installed" if status[hook] else "not installed" - print(f" {name}: {state}") - - elif subcmd == "auto-sync": - auto_cmd = getattr(args, "auto_sync_command", None) - if auto_cmd == "enable": - from ..cloud.timer import install_timer - result = install_timer(interval=args.interval) - print(f" Timer installed ({result['interval']} interval)") - print(f" Service: {result['service_path']}") - print(f" Timer: {result['timer_path']}") - if result['enabled']: - print(" Status: Active") - else: - print(" Status: Installed but not started (systemctl not available)") - elif auto_cmd == "disable": - from ..cloud.timer import uninstall_timer - result = uninstall_timer() - if result['removed']: - print(" Auto-sync disabled") - for p in result['paths']: - print(f" Removed: {p}") - else: - print(" No timer found") - elif auto_cmd == "status": - from ..cloud.timer import timer_status - status = timer_status() - if not status['installed']: - print(" Auto-sync: Not installed") - print(" Run: neurostack cloud auto-sync enable") - else: - state = "active" if status['active'] else "inactive" - print(f" Auto-sync: {state}") - if status['interval']: - print(f" Interval: {status['interval']}") - if status['next_run']: - print(f" Next run: {status['next_run']}") - else: - print("Usage: neurostack cloud auto-sync {enable|disable|status}") - - else: - print( - "Usage: neurostack cloud " - "{login|logout|status|setup|push|pull|sync|query|triples|" - "summary|consent|install-hooks|uninstall-hooks|hooks-status|auto-sync}" - ) - print("\nCommands:") - print(" login Authenticate via browser (device code) or --key") - print(" logout Clear stored credentials") - print(" status Show authentication state and usage") - print(" setup Interactive cloud configuration") - print(" consent Grant privacy consent for cloud features") - print(" push Upload vault files for cloud indexing") - print(" pull Download indexed database from cloud") - print(" sync Push vault changes and fetch new memories") - print(" query Search vault via cloud API") - print(" triples Search knowledge graph triples") - print(" summary Get a note summary") - print(" install-hooks Install git hooks for automatic cloud sync") - print(" uninstall-hooks Remove git hooks for cloud sync") - print(" hooks-status Check git hook installation status") - print(" auto-sync Manage automatic periodic sync (systemd timer)") - - -def cmd_cloud_push(args): - """Upload vault files to cloud for indexing.""" - from ..cloud.sync import ConsentError, SyncError, VaultSyncEngine - - cfg = get_config() - cloud_cfg = _ensure_cloud_auth() - - engine = VaultSyncEngine( - cloud_api_url=cloud_cfg.cloud_api_url, - cloud_api_key=cloud_cfg.cloud_api_key, - vault_root=cfg.vault_root, - db_dir=cfg.db_dir, - consent_given=cloud_cfg.consent_given, - ) - - def on_progress(msg: str): - print(f" {msg}") - - try: - result = engine.push(progress_callback=on_progress) - except ConsentError as e: - print(f"Error: {e}", file=sys.stderr) - sys.exit(1) - except SyncError as e: - print(f"Error: {e}", file=sys.stderr) - sys.exit(1) - - if getattr(args, "json", False): - print(json.dumps(result)) - else: - print(f"Push complete: {result.get('message', 'done')}") - - -def cmd_cloud_sync(args): - """Push vault changes and fetch new memories from cloud.""" - from ..cloud.sync import ConsentError, SyncError, VaultSyncEngine - - quiet = getattr(args, "quiet", False) - - cfg = get_config() - cloud_cfg = _ensure_cloud_auth() - - engine = VaultSyncEngine( - cloud_api_url=cloud_cfg.cloud_api_url, - cloud_api_key=cloud_cfg.cloud_api_key, - vault_root=cfg.vault_root, - db_dir=cfg.db_dir, - consent_given=cloud_cfg.consent_given, - ) - - def on_progress(msg: str): - if not quiet: - print(f" {msg}") - - try: - result = engine.sync(progress_callback=on_progress) - except ConsentError as e: - if not quiet: - print(f"Error: {e}", file=sys.stderr) - sys.exit(1) - except SyncError as e: - if not quiet: - print(f"Error: {e}", file=sys.stderr) - sys.exit(1) - - if quiet: - return - - if getattr(args, "json", False): - print(json.dumps(result)) - else: - push_msg = result.get("message", "done") - mem_count = result.get("memories_fetched", 0) - print(f"Sync complete: {push_msg}") - print(f" Memories fetched: {mem_count}") - - -def cmd_cloud_pull(args): - """Download indexed database from cloud.""" - from ..cloud.sync import SyncError, VaultSyncEngine - - cfg = get_config() - cloud_cfg = _ensure_cloud_auth() - - engine = VaultSyncEngine( - cloud_api_url=cloud_cfg.cloud_api_url, - cloud_api_key=cloud_cfg.cloud_api_key, - vault_root=cfg.vault_root, - db_dir=cfg.db_dir, - ) - - try: - db_path = engine.pull() - except SyncError as e: - print(f"Error: {e}", file=sys.stderr) - sys.exit(1) - - if getattr(args, "json", False): - print(json.dumps({ - "db_path": str(db_path), - "size": db_path.stat().st_size, - })) - else: - size_mb = db_path.stat().st_size / (1024 * 1024) - print(f" \033[32m\u2713\033[0m Downloaded" - f" ({size_mb:.1f} MB)") - print() - print(" \033[1m\u2501\u2501\u2501 Setup complete \u2501\u2501\u2501\033[0m") - print() - print(" All search modes now available:") - print(" neurostack cloud query '...'" - " # Search via cloud API") - print(" neurostack search 'query'" - " # Hybrid search (local)") - print(" neurostack serve" - " # Start MCP server") - print() - print(" Dashboard:" - " https://app.neurostack.sh") - print() - - -def cmd_cloud_query(args): - """Query vault via cloud API with tiered search.""" - cloud_cfg = _ensure_cloud_auth() - client = CloudClient(cloud_cfg) - - try: - result = client.query( - args.query, - top_k=args.top_k, - depth=args.depth, - mode=args.mode, - workspace=getattr(args, "workspace", None), - ) - except FileNotFoundError as e: - print(f"Error: {e}", file=sys.stderr) - sys.exit(1) - except Exception as e: - print(f"Error: {e}", file=sys.stderr) - sys.exit(1) - - if getattr(args, "json", False): - print(json.dumps(result)) - return - - depth = result.get("depth_used", "unknown") - print(f" Depth: {depth}\n") - - triples = result.get("triples", []) - if triples: - print(f" Triples ({len(triples)}):") - for t in triples: - print(f" {t['subject']} -> {t['predicate']} -> {t['object']}") - print(f" [{t.get('score', 0):.2f}] {t['note']}") - print() - - summaries = result.get("summaries", []) - if summaries: - print(f" Summaries ({len(summaries)}):") - for s in summaries: - title = s.get("title", s.get("note", "untitled")) - summary = s.get("summary", "")[:200] - print(f" {title}") - print(f" {summary}") - print() - - chunks = result.get("chunks", []) - if chunks: - print(f" Results ({len(chunks)}):") - for i, c in enumerate(chunks, 1): - title = c.get("title", c.get("note", "untitled")) - section = c.get("section", "") - snippet = c.get("snippet", "")[:150] - score = c.get("score", 0) - print(f" {i}. [{score:.2f}] {title}") - if section: - print(f" Section: {section}") - if snippet: - print(f" {snippet}") - - if not triples and not summaries and not chunks: - print(" No results found.") - - -def cmd_cloud_triples(args): - """Search knowledge graph triples via cloud API.""" - cloud_cfg = load_cloud_config() - if not cloud_cfg.cloud_api_url or not cloud_cfg.cloud_api_key: - print("Error: Not authenticated. Run `neurostack cloud login` first.", file=sys.stderr) - sys.exit(1) - - client = CloudClient(cloud_cfg) - - try: - results = client.triples( - args.query, - top_k=args.top_k, - workspace=getattr(args, "workspace", None), - ) - except FileNotFoundError as e: - print(f"Error: {e}", file=sys.stderr) - sys.exit(1) - - if getattr(args, "json", False): - print(json.dumps(results)) - return - - if not results: - print("No triples found.") - return - - for t in results: - print(f" {t['subject']} -> {t['predicate']} -> {t['object']}") - print(f" [{t.get('score', 0):.2f}] {t.get('note', '')}") - - -def cmd_cloud_summary(args): - """Get a note summary via cloud API.""" - cloud_cfg = load_cloud_config() - if not cloud_cfg.cloud_api_url or not cloud_cfg.cloud_api_key: - print("Error: Not authenticated. Run `neurostack cloud login` first.", file=sys.stderr) - sys.exit(1) - - client = CloudClient(cloud_cfg) - - try: - result = client.summary(args.note_path) - except Exception as e: - print(f"Error: {e}", file=sys.stderr) - sys.exit(1) - - if result is None: - print(f"No summary found for: {args.note_path}", file=sys.stderr) - sys.exit(1) - - if getattr(args, "json", False): - print(json.dumps(result)) - return - - print(f" {result.get('title', args.note_path)}") - print(f" {result.get('summary', 'No summary')}") diff --git a/src/neurostack/cli/index.py b/src/neurostack/cli/index.py index 6df3508..38c4e52 100644 --- a/src/neurostack/cli/index.py +++ b/src/neurostack/cli/index.py @@ -64,5 +64,4 @@ def cmd_watch(args): vault_root=Path(args.vault), embed_url=args.embed_url, summarize_url=args.summarize_url, - cloud=getattr(args, "cloud", False), ) diff --git a/src/neurostack/cli/setup.py b/src/neurostack/cli/setup.py index 1e6b692..05ee5db 100644 --- a/src/neurostack/cli/setup.py +++ b/src/neurostack/cli/setup.py @@ -9,7 +9,6 @@ from .. import __version__ from ..config import CONFIG_PATH, get_config -from .cloud import _cmd_cloud_device_login, cmd_cloud_push from .utils import _get_vault_template_dir @@ -166,7 +165,7 @@ def _do_init(vault_root, cfg, profession_name=None, run_index=False): label = d.split("/")[-1].replace("-", " ").title() idx.write_text(f"# {label}\n\n") - # Write config — preserve existing [cloud] section + # Write config try: import tomllib as _tomllib except ImportError: @@ -406,11 +405,8 @@ def cmd_init(args): if args.path or args.profession or not sys.stdin.isatty(): vault_root = Path(args.path) if args.path else cfg.vault_root mode = getattr(args, "mode", None) or "lite" - use_cloud = getattr(args, "cloud", False) - if use_cloud: - mode = "lite" - cfg.mode = "cloud" if use_cloud else "local" + cfg.mode = "local" if mode == "full": uv_bin = _find_uv() if uv_bin: @@ -459,62 +455,47 @@ def cmd_init(args): " https://astral.sh/uv/install.sh | sh") sys.exit(1) - # ── Step 1: Cloud or Local? ── - print() - setup_choices = [ - ("cloud", "Cloud — Gemini indexes your vault, no GPU needed"), - ("local", "Local — self-hosted with Ollama"), - ] - setup = _prompt( - "How do you want to run NeuroStack?", - default="cloud", choices=setup_choices, - ) - use_cloud = setup == "cloud" - + # ── Step 1: Lite or Full? ── mode = "lite" pull_models = False embed_model = cfg.embed_model llm_model = cfg.llm_model - if use_cloud: - mode = "lite" - else: - # ── Step 2: Lite or Full? ── - _print_hardware_recommendation() - print() - mode_choices = [ - ("lite", - "Lite — FTS5 search + graph, no ML (~130 MB)"), - ("full", - "Full — + embeddings, summaries, communities (~560 MB)"), - ] - mode = _prompt( - "Installation mode", - default="full", choices=mode_choices, - ) + _print_hardware_recommendation() + print() + mode_choices = [ + ("lite", + "Lite — FTS5 search + graph, no ML (~130 MB)"), + ("full", + "Full — + embeddings, summaries, communities (~560 MB)"), + ] + mode = _prompt( + "Installation mode", + default="full", choices=mode_choices, + ) - if mode == "full": - print("\n \033[1mOllama Models\033[0m") - print(" Full mode uses Ollama for embeddings" - " and summaries.") - pull_models = _confirm( - "Pull Ollama models now?", default=True, + if mode == "full": + print("\n \033[1mOllama Models\033[0m") + print(" Full mode uses Ollama for embeddings" + " and summaries.") + pull_models = _confirm( + "Pull Ollama models now?", default=True, + ) + if pull_models: + embed_model = _prompt( + "Embedding model", default=cfg.embed_model, + ) + model_choices = [ + ("phi3.5", "phi3.5 — MIT, fast, 3.8B"), + ("qwen3:8b", "qwen3:8b — Apache 2.0, strong"), + ("llama3.1:8b", "llama3.1:8b — Meta license"), + ("mistral:7b", "mistral:7b — Apache 2.0"), + ] + llm_model = _prompt( + "LLM model", + default=cfg.llm_model, + choices=model_choices, ) - if pull_models: - embed_model = _prompt( - "Embedding model", default=cfg.embed_model, - ) - model_choices = [ - ("phi3.5", "phi3.5 — MIT, fast, 3.8B"), - ("qwen3:8b", "qwen3:8b — Apache 2.0, strong"), - ("llama3.1:8b", "llama3.1:8b — Meta license"), - ("mistral:7b", "mistral:7b — Apache 2.0"), - ] - llm_model = _prompt( - "LLM model", - default=cfg.llm_model, - choices=model_choices, - ) # ── Step 3: Vault path ── print() @@ -561,7 +542,7 @@ def cmd_init(args): # ── Summary ── print("\n \033[1m━━━ Plan ━━━\033[0m\n") - print(f" Mode: {'cloud' if use_cloud else mode}") + print(f" Mode: {mode}") print(f" Vault: {vault_root}") print(f" Profession: {profession}") if mode == "full": @@ -573,8 +554,6 @@ def cmd_init(args): auth_label = "yes" if (llm_api_key or embed_api_key) else "no" print(f" API auth: {auth_label}") print(" Index: full (summaries + triples + communities)") - elif use_cloud: - print(" Index: cloud (Gemini)") else: print(" Index: lite (FTS5 only)") @@ -597,7 +576,7 @@ def cmd_init(args): _setup_ollama(pull_models, embed_model, llm_model, cfg) # 3. Apply config + create vault structure - cfg.mode = "cloud" if use_cloud else "local" + cfg.mode = "local" cfg.vault_root = vault_root cfg.embed_url = embed_url cfg.llm_url = llm_url @@ -610,43 +589,10 @@ def cmd_init(args): _do_init(vault_root, cfg, profession_name=profession, run_index=False) _full_index_pipeline(vault_root, cfg) else: - # Lite/cloud: create vault with FTS5-only index + # Lite: create vault with FTS5-only index _do_init(vault_root, cfg, profession_name=profession, run_index=True) - # 4. Cloud path: login + push - if use_cloud: - print("\n \033[1m━━━ Cloud Login ━━━\033[0m\n") - _cmd_cloud_device_login() - - from ..cloud.config import load_cloud_config - cloud_cfg = load_cloud_config() - if cloud_cfg.cloud_api_key: - print("\n \033[32m✓\033[0m Logged in") - from ..cloud.client import CloudClient - tier = "Free" - try: - client = CloudClient(cloud_cfg) - info = client.status() - tier = (info.get("tier") or "free").capitalize() - except Exception: - pass - print(f" Plan: {tier}") - - if sys.stdin.isatty() and _confirm( - "\n Push vault to cloud now?", default=True, - ): - print() - cmd_cloud_push(args) - print() - print(" Check progress:" - " https://app.neurostack.sh") - else: - print("\n Run later: neurostack cloud push") - else: - print("\n \033[33m!\033[0m Login skipped.") - print(" Run later: neurostack cloud login") - - # 5. PATH check + # 4. PATH check local_bin = str(Path.home() / ".local" / "bin") if local_bin not in os.environ.get("PATH", ""): print("\n \033[33m!\033[0m Add to PATH:" @@ -1244,7 +1190,7 @@ def cmd_update(args): def cmd_install(args): - """Streamlined installation: local or cloud, deps, and setup. + """Streamlined installation: deps and setup. DEPRECATED: Use 'neurostack init' instead, which combines installation and vault setup into a single command. @@ -1264,7 +1210,6 @@ def cmd_install(args): pull_models = args.pull_models embed_model = args.embed_model or cfg.embed_model llm_model = args.llm_model or cfg.llm_model - use_cloud = False else: # ── Interactive wizard ── print("\n \033[1m━━━ NeuroStack Install ━━━\033[0m\n") @@ -1304,87 +1249,69 @@ def cmd_install(args): ) sys.exit(1) - # 2. Local or Cloud? - setup_choices = [ - ("cloud", "Cloud — Gemini indexes your vault, no GPU needed"), - ("local", "Local — self-hosted with Ollama (requires GPU)"), + # 2. Lite or Full? + # Detect current mode + current_mode = "lite" + try: + import numpy # noqa: F401 + current_mode = "full" + except ImportError: + pass + print(f" Current: {current_mode} mode\n") + + mode_choices = [ + ("lite", + "Lite — FTS5 search + graph, no ML (~130 MB)"), + ("full", + "Full — + embeddings, summaries, communities (~560 MB)"), ] - setup = _prompt( - "How do you want to run NeuroStack?", - default="cloud", choices=setup_choices, + mode = _prompt( + "Installation mode", + default=current_mode, choices=mode_choices, ) - use_cloud = setup == "cloud" - - if use_cloud: - # ── Cloud path: lite deps, then login ── - mode = "lite" - pull_models = False - embed_model = cfg.embed_model - llm_model = cfg.llm_model - else: - # ── Local path: existing flow ── - # Detect current mode - current_mode = "lite" - try: - import numpy # noqa: F401 - current_mode = "full" - except ImportError: - pass - print(f" Current: {current_mode} mode\n") - - mode_choices = [ - ("lite", - "Lite — FTS5 search + graph, no ML (~130 MB)"), - ("full", - "Full — + embeddings, summaries, communities (~560 MB)"), - ] - mode = _prompt( - "Installation mode", - default=current_mode, choices=mode_choices, - ) - pull_models = False - embed_model = cfg.embed_model - llm_model = cfg.llm_model - if mode == "full": - print("\n \033[1mOllama Models\033[0m") - print( - " Full mode uses Ollama for embeddings" - " and summaries." + pull_models = False + embed_model = cfg.embed_model + llm_model = cfg.llm_model + if mode == "full": + print("\n \033[1mOllama Models\033[0m") + print( + " Full mode uses Ollama for embeddings" + " and summaries." + ) + pull_models = _confirm( + "Pull Ollama models now?", default=True, + ) + if pull_models: + embed_model = _prompt( + "Embedding model", default=cfg.embed_model, ) - pull_models = _confirm( - "Pull Ollama models now?", default=True, + model_choices = [ + ("phi3.5", + "phi3.5 — MIT, fast, 3.8B"), + ("qwen3:8b", + "qwen3:8b — Apache 2.0, strong"), + ("llama3.1:8b", + "llama3.1:8b — Meta license"), + ("mistral:7b", + "mistral:7b — Apache 2.0"), + ] + llm_model = _prompt( + "LLM model", + default=cfg.llm_model, + choices=model_choices, ) - if pull_models: - embed_model = _prompt( - "Embedding model", default=cfg.embed_model, - ) - model_choices = [ - ("phi3.5", - "phi3.5 — MIT, fast, 3.8B"), - ("qwen3:8b", - "qwen3:8b — Apache 2.0, strong"), - ("llama3.1:8b", - "llama3.1:8b — Meta license"), - ("mistral:7b", - "mistral:7b — Apache 2.0"), - ] - llm_model = _prompt( - "LLM model", - default=cfg.llm_model, - choices=model_choices, - ) - print("\n \033[1m━━━ Plan ━━━\033[0m\n") - print(f" Mode: {mode}") - if pull_models: - print(f" Embed: ollama pull {embed_model}") - print(f" LLM: ollama pull {llm_model}") - else: - print(" Models: skip") - if not _confirm("\n Proceed?", default=True): - print("\n Cancelled.") - return + print("\n \033[1m━━━ Plan ━━━\033[0m\n") + print(f" Mode: {mode}") + if pull_models: + print(f" Embed: ollama pull {embed_model}") + print(f" LLM: ollama pull {llm_model}") + else: + print(" Models: skip") + if not _confirm("\n Proceed?", default=True): + print("\n Cancelled.") + return # ── Execute installation ── print() @@ -1496,49 +1423,7 @@ def cmd_install(args): print("\n \033[33m!\033[0m Add to PATH:" " export PATH=\"$HOME/.local/bin:$PATH\"") - # 5. Cloud setup (if cloud path was chosen) - if use_cloud: - print("\n \033[32m✓\033[0m Dependencies installed (lite)") - print("\n \033[1m━━━ Cloud Login ━━━\033[0m\n") - _cmd_cloud_device_login() - - # Check if login succeeded - from ..cloud.config import load_cloud_config - cloud_cfg = load_cloud_config() - if cloud_cfg.cloud_api_key: - print("\n \033[32m✓\033[0m Logged in") - - # Fetch tier - from ..cloud.client import CloudClient - tier = "Free" - try: - client = CloudClient(cloud_cfg) - info = client.status() - tier = (info.get("tier") or "free").capitalize() - except Exception: - pass - print(f" Plan: {tier}") - print(" Dashboard:" - " https://app.neurostack.sh") - - # Auto-run init — set defaults for init args - print("\n \033[1m━━━ Vault Setup ━━━\033[0m\n") - args.path = None - args.profession = None - args.index = True - cmd_init(args) - else: - print("\n \033[33m!\033[0m Login skipped") - print("\n \033[32mInstalled!\033[0m (lite)" - " Run this next:") - print(" neurostack init" - " # Set up your vault") - print(" neurostack cloud login" - " # Sign in later") - print() - return - - # Summary (local path) + # Summary print(f"\n \033[32mInstalled!\033[0m ({mode} mode)") print() print(" Next steps:") diff --git a/src/neurostack/cloud/__init__.py b/src/neurostack/cloud/__init__.py deleted file mode 100644 index 583de67..0000000 --- a/src/neurostack/cloud/__init__.py +++ /dev/null @@ -1,7 +0,0 @@ -# SPDX-License-Identifier: Apache-2.0 -# Copyright (c) 2024-2026 Raphael Southall -"""NeuroStack Cloud client — connect to NeuroStack Cloud for hosted indexing. - -Client-side modules for `neurostack cloud push/pull/query` CLI commands. -Server-side infrastructure lives in the separate neurostack-cloud package. -""" diff --git a/src/neurostack/cloud/client.py b/src/neurostack/cloud/client.py deleted file mode 100644 index 10c3bba..0000000 --- a/src/neurostack/cloud/client.py +++ /dev/null @@ -1,450 +0,0 @@ -# SPDX-License-Identifier: Apache-2.0 -# Copyright (c) 2024-2026 Raphael Southall -"""HTTP client for NeuroStack Cloud API with Bearer token authentication. - -Covers all 21 MCP tools. Methods with dedicated REST endpoints use them -directly; the rest route through ``POST /v1/vault/tools/{name}``. -""" - -from __future__ import annotations - -from typing import Any - -import httpx - -from .config import CloudConfig - - -class CloudClient: - """HTTP client for NeuroStack Cloud API with Bearer token authentication. - - Wraps httpx for synchronous HTTP calls (CLI is synchronous). - All authenticated requests include ``Authorization: Bearer {api_key}``. - """ - - def __init__(self, config: CloudConfig) -> None: - self._config = config - self._base_url = config.cloud_api_url.rstrip("/") - - @property - def is_configured(self) -> bool: - """True if both cloud URL and API key are set.""" - return bool(self._config.cloud_api_url and self._config.cloud_api_key) - - def _auth_headers(self) -> dict[str, str]: - """Build Bearer auth header from stored API key.""" - if self._config.cloud_api_key: - return {"Authorization": f"Bearer {self._config.cloud_api_key}"} - return {} - - def _post(self, path: str, body: dict, *, timeout: float = 60.0) -> httpx.Response: - """Authenticated POST request.""" - return httpx.post( - f"{self._base_url}{path}", - json=body, - headers=self._auth_headers(), - timeout=timeout, - ) - - def _get(self, path: str, *, timeout: float = 30.0) -> httpx.Response: - """Authenticated GET request.""" - return httpx.get( - f"{self._base_url}{path}", - headers=self._auth_headers(), - timeout=timeout, - ) - - def _handle_response(self, resp: httpx.Response) -> dict: - """Common response handling: 404 -> FileNotFoundError, else raise.""" - if resp.status_code == 404: - detail = "No database found" - try: - detail = resp.json().get("detail", detail) - except Exception: - pass - raise FileNotFoundError(detail) - resp.raise_for_status() - return resp.json() - - def _tool_call(self, tool_name: str, **kwargs: Any) -> dict: - """Generic tool invocation via POST /v1/vault/tools/{name}. - - Used for tools that don't have a dedicated REST endpoint. - The cloud API dispatches to the same tool implementation. - """ - body = {k: v for k, v in kwargs.items() if v is not None} - resp = self._post(f"/v1/vault/tools/{tool_name}", body) - return self._handle_response(resp) - - # ----------------------------------------------------------------- - # Infrastructure - # ----------------------------------------------------------------- - - def health(self) -> dict: - """Check cloud API health. No auth required.""" - url = f"{self._base_url}/health" - try: - resp = httpx.get(url, timeout=10.0) - resp.raise_for_status() - return resp.json() - except httpx.ConnectError: - raise ConnectionError( - f"Cannot reach cloud API at {self._base_url}. " - "Check your cloud_api_url setting." - ) - except httpx.TimeoutException: - raise ConnectionError( - f"Cloud API at {self._base_url} timed out." - ) - - def validate_key(self) -> bool: - """Validate stored API key. Returns True on 200, False on 401.""" - url = f"{self._base_url}/v1/usage" - try: - resp = httpx.get(url, headers=self._auth_headers(), timeout=10.0) - except httpx.ConnectError: - raise ConnectionError( - f"Cannot reach cloud API at {self._base_url}. " - "Check your cloud_api_url setting." - ) - except httpx.TimeoutException: - raise ConnectionError( - f"Cloud API at {self._base_url} timed out." - ) - if resp.status_code == 401: - return False - return resp.status_code == 200 - - def status(self) -> dict: - """Get authenticated status including tier and usage.""" - url = f"{self._base_url}/v1/usage" - try: - resp = httpx.get(url, headers=self._auth_headers(), timeout=10.0) - except (httpx.ConnectError, httpx.TimeoutException): - return { - "authenticated": False, "tier": None, - "cloud_url": self._base_url, "usage": None, - } - if resp.status_code == 401: - return { - "authenticated": False, "tier": None, - "cloud_url": self._base_url, "usage": None, - } - if resp.status_code == 200: - data = resp.json() - return { - "authenticated": True, - "tier": data.get("tier", "free"), - "cloud_url": self._base_url, - "usage": data.get("usage"), - } - return { - "authenticated": False, "tier": None, - "cloud_url": self._base_url, "usage": None, - } - - # ----------------------------------------------------------------- - # Search & Retrieval (dedicated endpoints) - # ----------------------------------------------------------------- - - def vault_search( - self, - query: str, - *, - top_k: int = 5, - mode: str = "hybrid", - depth: str = "auto", - context: str | None = None, - workspace: str | None = None, - ) -> dict: - """Hybrid search with tiered retrieval depth.""" - body: dict = {"query": query, "top_k": top_k, "depth": depth, "mode": mode} - if workspace: - body["workspace"] = workspace - if context: - body["context"] = context - resp = self._post("/v1/vault/query", body) - return self._handle_response(resp) - - def vault_triples( - self, - query: str, - *, - top_k: int = 10, - mode: str = "hybrid", - workspace: str | None = None, - ) -> dict: - """Search knowledge graph triples.""" - body: dict = {"query": query, "top_k": top_k} - if mode != "hybrid": - body["mode"] = mode - if workspace: - body["workspace"] = workspace - resp = self._post("/v1/vault/triples", body) - data = self._handle_response(resp) - # Normalize: REST returns {"triples": [...]}, tool expects dict - if "triples" in data and not any( - k for k in data if k != "triples" - ): - return data - return data - - def vault_summary(self, path_or_query: str) -> dict: - """Get pre-computed note summary.""" - resp = self._post("/v1/vault/summary", {"note_path": path_or_query}) - if resp.status_code == 404: - return {"error": f"No summary found for: {path_or_query}"} - resp.raise_for_status() - return resp.json() - - def vault_stats(self) -> dict: - """Get vault index health statistics.""" - resp = self._get("/v1/vault/stats") - stats = self._handle_response(resp) - # Also fetch health metrics - try: - health_resp = self._get("/v1/vault/health") - if health_resp.status_code == 200: - stats.update(health_resp.json()) - except Exception: - pass - return stats - - def vault_graph( - self, - note: str, - *, - depth: int = 1, - workspace: str | None = None, - ) -> dict: - """Get wiki-link neighborhood of a note.""" - return self._tool_call( - "vault_graph", note=note, depth=depth, workspace=workspace, - ) - - def vault_related( - self, - note: str, - *, - top_k: int = 10, - workspace: str | None = None, - ) -> dict: - """Find semantically similar notes.""" - return self._tool_call( - "vault_related", note=note, top_k=top_k, workspace=workspace, - ) - - def vault_ask( - self, - question: str, - *, - top_k: int = 8, - workspace: str | None = None, - ) -> dict: - """RAG Q&A with inline citations.""" - return self._tool_call( - "vault_ask", question=question, top_k=top_k, workspace=workspace, - ) - - def vault_communities( - self, - query: str, - *, - top_k: int = 6, - level: int = 0, - map_reduce: bool = True, - workspace: str | None = None, - ) -> dict: - """GraphRAG global queries across topic clusters.""" - return self._tool_call( - "vault_communities", - query=query, top_k=top_k, level=level, - map_reduce=map_reduce, workspace=workspace, - ) - - def vault_context( - self, - task: str, - *, - token_budget: int = 2000, - workspace: str | None = None, - include_memories: bool = True, - include_triples: bool = True, - ) -> dict: - """Task-scoped context recovery.""" - return self._tool_call( - "vault_context", - task=task, token_budget=token_budget, workspace=workspace, - include_memories=include_memories, include_triples=include_triples, - ) - - def session_brief(self, *, workspace: str | None = None) -> dict: - """Compact session briefing.""" - return self._tool_call("session_brief", workspace=workspace) - - def vault_record_usage(self, note_paths: list[str]) -> dict: - """Record note usage for hotness scoring.""" - return self._tool_call("vault_record_usage", note_paths=note_paths) - - def vault_prediction_errors( - self, - *, - error_type: str | None = None, - limit: int = 20, - resolve: list[str] | None = None, - workspace: str | None = None, - ) -> dict: - """Find notes flagged as poor retrieval fit.""" - return self._tool_call( - "vault_prediction_errors", - error_type=error_type, limit=limit, - resolve=resolve, workspace=workspace, - ) - - # ----------------------------------------------------------------- - # Memories (write operations — cloud stores in Firestore) - # ----------------------------------------------------------------- - - def vault_remember( - self, - content: str, - *, - tags: list[str] | None = None, - entity_type: str = "observation", - source_agent: str | None = None, - workspace: str | None = None, - ttl_hours: float | None = None, - session_id: int | None = None, - ) -> dict: - """Save a memory.""" - return self._tool_call( - "vault_remember", - content=content, tags=tags, entity_type=entity_type, - source_agent=source_agent, workspace=workspace, - ttl_hours=ttl_hours, session_id=session_id, - ) - - def vault_forget(self, memory_id: int) -> dict: - """Delete a memory.""" - return self._tool_call("vault_forget", memory_id=memory_id) - - def vault_update_memory( - self, - memory_id: int, - *, - content: str | None = None, - tags: list[str] | None = None, - add_tags: list[str] | None = None, - remove_tags: list[str] | None = None, - entity_type: str | None = None, - workspace: str | None = None, - ttl_hours: float | None = None, - ) -> dict: - """Update a memory.""" - return self._tool_call( - "vault_update_memory", - memory_id=memory_id, content=content, tags=tags, - add_tags=add_tags, remove_tags=remove_tags, - entity_type=entity_type, workspace=workspace, - ttl_hours=ttl_hours, - ) - - def vault_merge(self, target_id: int, source_id: int) -> dict: - """Merge two memories.""" - return self._tool_call( - "vault_merge", target_id=target_id, source_id=source_id, - ) - - def vault_memories( - self, - *, - query: str | None = None, - entity_type: str | None = None, - workspace: str | None = None, - limit: int = 20, - ) -> dict: - """Search or list memories.""" - return self._tool_call( - "vault_memories", - query=query, entity_type=entity_type, - workspace=workspace, limit=limit, - ) - - # ----------------------------------------------------------------- - # Sessions - # ----------------------------------------------------------------- - - def vault_session_start( - self, - *, - source_agent: str | None = None, - workspace: str | None = None, - ) -> dict: - """Begin a memory session.""" - return self._tool_call( - "vault_session_start", - source_agent=source_agent, workspace=workspace, - ) - - def vault_session_end( - self, - session_id: int, - *, - summarize: bool = True, - auto_harvest: bool = True, - ) -> dict: - """End a memory session.""" - return self._tool_call( - "vault_session_end", - session_id=session_id, summarize=summarize, - auto_harvest=auto_harvest, - ) - - def vault_harvest( - self, - *, - sessions: int = 1, - dry_run: bool = False, - provider: str | None = None, - ) -> dict: - """Extract insights from Claude Code sessions.""" - return self._tool_call( - "vault_harvest", - sessions=sessions, dry_run=dry_run, provider=provider, - ) - - # ----------------------------------------------------------------- - # Backwards compatibility aliases - # ----------------------------------------------------------------- - - def query( - self, - query: str, - *, - top_k: int = 10, - depth: str = "auto", - mode: str = "hybrid", - workspace: str | None = None, - ) -> dict: - """Alias for vault_search (backwards compat with existing callers).""" - return self.vault_search( - query, top_k=top_k, depth=depth, mode=mode, workspace=workspace, - ) - - def triples( - self, - query: str, - *, - top_k: int = 10, - workspace: str | None = None, - ) -> list[dict]: - """Alias for vault_triples (backwards compat).""" - result = self.vault_triples(query, top_k=top_k, workspace=workspace) - return result.get("triples", []) if isinstance(result, dict) else result - - def summary(self, note_path: str) -> dict | None: - """Alias for vault_summary (backwards compat).""" - result = self.vault_summary(note_path) - if isinstance(result, dict) and "error" in result: - return None - return result diff --git a/src/neurostack/cloud/config.py b/src/neurostack/cloud/config.py deleted file mode 100644 index ea44bd4..0000000 --- a/src/neurostack/cloud/config.py +++ /dev/null @@ -1,112 +0,0 @@ -# SPDX-License-Identifier: Apache-2.0 -# Copyright (c) 2024-2026 Raphael Southall -"""Cloud client configuration for NeuroStack. - -Client-side settings for connecting to NeuroStack Cloud. -Loaded from defaults -> config.toml [cloud] section -> environment variables. -""" - -from __future__ import annotations - -import os -from dataclasses import dataclass - -try: - import tomllib -except ImportError: - import tomli as tomllib # Python 3.10 fallback - -import tomli_w - - -def _get_config_path(): - """Get CONFIG_PATH lazily to avoid circular import with neurostack.config.""" - from neurostack.config import CONFIG_PATH - return CONFIG_PATH - - -@dataclass -class CloudConfig: - """Client-side cloud settings for connecting to NeuroStack Cloud.""" - - cloud_api_url: str = "" - cloud_api_key: str = "" - consent_given: bool = False - consent_date: str = "" - - -def _read_toml() -> dict: - """Read existing config.toml or return empty dict.""" - config_path = _get_config_path() - if config_path.exists(): - with open(config_path, "rb") as f: - return tomllib.load(f) - return {} - - -def _write_toml(data: dict) -> None: - """Write config dict to config.toml, creating parent dirs if needed.""" - config_path = _get_config_path() - config_path.parent.mkdir(parents=True, exist_ok=True) - with open(config_path, "wb") as f: - tomli_w.dump(data, f) - - -def save_cloud_config(cloud_api_url: str, cloud_api_key: str) -> None: - """Persist cloud API URL and key to config.toml [cloud] section.""" - data = _read_toml() - cloud = data.get("cloud", {}) - cloud["cloud_api_url"] = cloud_api_url - cloud["cloud_api_key"] = cloud_api_key - data["cloud"] = cloud - _write_toml(data) - - -def save_consent() -> None: - """Record that the user has granted cloud consent in config.toml.""" - from datetime import datetime, timezone - - data = _read_toml() - cloud = data.get("cloud", {}) - cloud["consent_given"] = True - cloud["consent_date"] = datetime.now(timezone.utc).isoformat() - data["cloud"] = cloud - _write_toml(data) - - -def clear_cloud_credentials() -> None: - """Remove cloud_api_key from config.toml but preserve cloud_api_url.""" - data = _read_toml() - cloud = data.get("cloud", {}) - cloud["cloud_api_key"] = "" - data["cloud"] = cloud - _write_toml(data) - - -def load_cloud_config() -> CloudConfig: - """Load cloud client config: defaults -> config.toml [cloud] -> env vars.""" - cfg = CloudConfig() - - # Layer 2: TOML [cloud] section - data = _read_toml() - cloud_data = data.get("cloud", {}) - - for key in ("cloud_api_url", "cloud_api_key", "consent_date"): - if key in cloud_data: - setattr(cfg, key, cloud_data[key]) - - if "consent_given" in cloud_data: - cfg.consent_given = bool(cloud_data["consent_given"]) - - # Layer 3: env var overrides (highest priority) - env_map = { - "NEUROSTACK_CLOUD_API_URL": "cloud_api_url", - "NEUROSTACK_CLOUD_API_KEY": "cloud_api_key", - } - - for env_key, attr in env_map.items(): - val = os.environ.get(env_key) - if val is not None: - setattr(cfg, attr, val) - - return cfg diff --git a/src/neurostack/cloud/dispatch.py b/src/neurostack/cloud/dispatch.py deleted file mode 100644 index d083837..0000000 --- a/src/neurostack/cloud/dispatch.py +++ /dev/null @@ -1,85 +0,0 @@ -# SPDX-License-Identifier: Apache-2.0 -# Copyright (c) 2024-2026 Raphael Southall -"""Cloud backend dispatch — replaces local tool functions with cloud proxies. - -When ``mode=cloud``, each registered tool is re-pointed to a CloudClient -method that forwards the call to the cloud REST API. The registry stays -protocol-agnostic: MCP, OpenAI, and REST adapters all pick up the change -automatically. - -Usage (called once at startup): - from neurostack.cloud.dispatch import enable_cloud_dispatch - enable_cloud_dispatch(registry, cloud_client) -""" - -from __future__ import annotations - -import logging -from typing import TYPE_CHECKING - -if TYPE_CHECKING: - from neurostack.cloud.client import CloudClient - from neurostack.tools.registry import ToolRegistry - -log = logging.getLogger("neurostack.cloud.dispatch") - -# Map: registry tool name -> CloudClient method name. -# Most are 1:1, but a few differ (e.g. query -> vault_search). -_TOOL_TO_METHOD = { - "vault_search": "vault_search", - "vault_triples": "vault_triples", - "vault_summary": "vault_summary", - "vault_stats": "vault_stats", - "vault_graph": "vault_graph", - "vault_related": "vault_related", - "vault_ask": "vault_ask", - "vault_communities": "vault_communities", - "vault_context": "vault_context", - "session_brief": "session_brief", - "vault_record_usage": "vault_record_usage", - "vault_prediction_errors": "vault_prediction_errors", - "vault_remember": "vault_remember", - "vault_forget": "vault_forget", - "vault_update_memory": "vault_update_memory", - "vault_merge": "vault_merge", - "vault_memories": "vault_memories", - "vault_session_start": "vault_session_start", - "vault_session_end": "vault_session_end", - "vault_harvest": "vault_harvest", -} - - -def enable_cloud_dispatch(registry: ToolRegistry, client: CloudClient) -> int: - """Replace local tool functions with cloud-proxied versions. - - For each tool in the registry that has a corresponding CloudClient method, - swaps the ``fn`` on the frozen ToolDef with a lambda that calls the client. - - Args: - registry: The populated tool registry (after ensure_registered()). - client: A configured CloudClient instance. - - Returns: - Number of tools successfully re-pointed to cloud. - """ - count = 0 - for tool_name, method_name in _TOOL_TO_METHOD.items(): - tool_def = registry.get(tool_name) - if tool_def is None: - log.debug("Tool %r not in registry — skipping cloud dispatch", tool_name) - continue - - cloud_method = getattr(client, method_name, None) - if cloud_method is None: - log.warning( - "CloudClient missing method %r for tool %r", method_name, tool_name, - ) - continue - - # ToolDef is frozen, so we replace via object.__setattr__ - object.__setattr__(tool_def, "fn", cloud_method) - count += 1 - log.debug("Cloud dispatch: %s -> CloudClient.%s", tool_name, method_name) - - log.info("Cloud dispatch enabled for %d/%d tools", count, len(_TOOL_TO_METHOD)) - return count diff --git a/src/neurostack/cloud/hooks.py b/src/neurostack/cloud/hooks.py deleted file mode 100644 index 92252e6..0000000 --- a/src/neurostack/cloud/hooks.py +++ /dev/null @@ -1,172 +0,0 @@ -# SPDX-License-Identifier: Apache-2.0 -# Copyright (c) 2024-2026 Raphael Southall -"""Git hooks installer for automatic cloud sync. - -Installs post-commit and post-merge hooks in the vault's git repository -that run `neurostack cloud sync` after each commit/merge. -""" -from __future__ import annotations - -import stat -from pathlib import Path - -HOOK_MARKER = "# neurostack-cloud-sync" - -POST_COMMIT_HOOK = f"""\ -#!/bin/sh -{HOOK_MARKER} -# Auto-sync vault to NeuroStack Cloud after commit -neurostack cloud sync --quiet 2>/dev/null & -""" - -POST_MERGE_HOOK = f"""\ -#!/bin/sh -{HOOK_MARKER} -# Auto-sync vault to NeuroStack Cloud after merge -neurostack cloud sync --quiet 2>/dev/null & -""" - - -def find_git_dir(vault_root: Path) -> Path | None: - """Find the .git directory for the vault. Returns None if not a git repo.""" - git_dir = vault_root / ".git" - if git_dir.is_dir(): - return git_dir - # Could be a git worktree — .git is a file with "gitdir: " - if git_dir.is_file(): - content = git_dir.read_text().strip() - if content.startswith("gitdir:"): - return Path(content.split(":", 1)[1].strip()) - return None - - -def install_hooks(vault_root: Path) -> dict: - """Install post-commit and post-merge hooks. - - Returns dict with results: - { - "installed": ["post-commit", "post-merge"], - "skipped": [], # already installed - "git_dir": str, - } - """ - git_dir = find_git_dir(vault_root) - if git_dir is None: - raise ValueError(f"Not a git repository: {vault_root}") - - hooks_dir = git_dir / "hooks" - hooks_dir.mkdir(exist_ok=True) - - installed = [] - skipped = [] - - for hook_name, hook_content in [ - ("post-commit", POST_COMMIT_HOOK), - ("post-merge", POST_MERGE_HOOK), - ]: - hook_path = hooks_dir / hook_name - - if hook_path.exists(): - existing = hook_path.read_text() - if HOOK_MARKER in existing: - skipped.append(hook_name) - continue - # Append to existing hook - hook_path.write_text(existing.rstrip() + "\n\n" + hook_content) - else: - hook_path.write_text(hook_content) - - # Make executable - hook_path.chmod(hook_path.stat().st_mode | stat.S_IXUSR | stat.S_IXGRP | stat.S_IXOTH) - installed.append(hook_name) - - return { - "installed": installed, - "skipped": skipped, - "git_dir": str(git_dir), - } - - -def uninstall_hooks(vault_root: Path) -> dict: - """Remove neurostack sync hooks. - - Returns dict with results: - { - "removed": ["post-commit", "post-merge"], - "not_found": [], - } - """ - git_dir = find_git_dir(vault_root) - if git_dir is None: - raise ValueError(f"Not a git repository: {vault_root}") - - hooks_dir = git_dir / "hooks" - removed = [] - not_found = [] - - for hook_name in ("post-commit", "post-merge"): - hook_path = hooks_dir / hook_name - - if not hook_path.exists(): - not_found.append(hook_name) - continue - - content = hook_path.read_text() - if HOOK_MARKER not in content: - not_found.append(hook_name) - continue - - # Remove the neurostack section - lines = content.split("\n") - new_lines = [] - skip = False - for line in lines: - if HOOK_MARKER in line: - skip = True - continue - if skip and line.startswith("#"): - continue - if skip and line.strip().startswith("neurostack"): - skip = False - continue - skip = False - new_lines.append(line) - - remaining = "\n".join(new_lines).strip() - if remaining and remaining != "#!/bin/sh": - hook_path.write_text(remaining + "\n") - else: - hook_path.unlink() - - removed.append(hook_name) - - return { - "removed": removed, - "not_found": not_found, - } - - -def hooks_status(vault_root: Path) -> dict: - """Check if hooks are installed. - - Returns: - { - "git_repo": bool, - "post_commit": bool, - "post_merge": bool, - } - """ - git_dir = find_git_dir(vault_root) - if git_dir is None: - return {"git_repo": False, "post_commit": False, "post_merge": False} - - hooks_dir = git_dir / "hooks" - result = {"git_repo": True, "post_commit": False, "post_merge": False} - - for hook_name in ("post_commit", "post_merge"): - file_name = hook_name.replace("_", "-") - hook_path = hooks_dir / file_name - if hook_path.exists() and HOOK_MARKER in hook_path.read_text(): - result[hook_name] = True - - return result diff --git a/src/neurostack/cloud/manifest.py b/src/neurostack/cloud/manifest.py deleted file mode 100644 index 9a65fe4..0000000 --- a/src/neurostack/cloud/manifest.py +++ /dev/null @@ -1,128 +0,0 @@ -# SPDX-License-Identifier: Apache-2.0 -# Copyright (c) 2024-2026 Raphael Southall -"""Content-hash manifest for incremental vault sync. - -Tracks SHA-256 hashes of vault markdown files so only changed content -is uploaded to the cloud indexer. -""" - -from __future__ import annotations - -import hashlib -import json -import os -from dataclasses import dataclass, field -from pathlib import Path - -import pathspec - - -@dataclass -class SyncDiff: - """Result of comparing two manifests.""" - - added: list[str] = field(default_factory=list) - """New files not in previous manifest.""" - - changed: list[str] = field(default_factory=list) - """Files whose content hash changed.""" - - removed: list[str] = field(default_factory=list) - """Files no longer in vault.""" - - @property - def has_changes(self) -> bool: - """True if any files were added, changed, or removed.""" - return bool(self.added or self.changed or self.removed) - - @property - def upload_files(self) -> list[str]: - """Files that need uploading (added + changed).""" - return self.added + self.changed - - -class Manifest: - """Content-hash manifest for vault files. - - Stores {relative_path: sha256_hex} mappings and supports diffing - two manifests to determine what changed. - """ - - def __init__(self, entries: dict[str, str] | None = None) -> None: - self.entries: dict[str, str] = entries or {} - - @staticmethod - def scan_vault(vault_root: Path, ignore_file: Path | None = None) -> Manifest: - """Walk vault_root, compute SHA-256 for each .md file. - - Skips directories starting with '.' (.obsidian, .git, .neurostack). - Uses forward slashes for cross-platform consistency. - - If *ignore_file* is provided and exists, patterns are parsed using - gitignore syntax (via ``pathspec``) and matching files are excluded. - """ - entries: dict[str, str] = {} - root = str(vault_root) - - # Build ignore spec from .neurostackignore if available - spec: pathspec.PathSpec | None = None - if ignore_file is not None and ignore_file.exists(): - patterns = ignore_file.read_text().splitlines() - patterns = [p for p in patterns if p.strip() and not p.strip().startswith("#")] - if patterns: - spec = pathspec.PathSpec.from_lines("gitignore", patterns) - - for dirpath, dirnames, filenames in os.walk(root): - # Filter out dot-directories in-place (prevents os.walk from descending) - dirnames[:] = [d for d in dirnames if not d.startswith(".")] - - for fname in filenames: - if not fname.endswith(".md"): - continue - - full_path = os.path.join(dirpath, fname) - # Relative path with forward slashes - rel_path = os.path.relpath(full_path, root).replace(os.sep, "/") - - if spec and spec.match_file(rel_path): - continue # Skip ignored files - - sha = hashlib.sha256() - with open(full_path, "rb") as f: - while chunk := f.read(65536): - sha.update(chunk) - - entries[rel_path] = sha.hexdigest() - - return Manifest(entries) - - @staticmethod - def load(path: Path) -> Manifest: - """Load manifest from JSON file. Returns empty Manifest if not found.""" - if not path.exists(): - return Manifest() - - with open(path) as f: - data = json.load(f) - - return Manifest(data) - - def save(self, path: Path) -> None: - """Save manifest to JSON file. Creates parent directories.""" - path.parent.mkdir(parents=True, exist_ok=True) - with open(path, "w") as f: - json.dump(self.entries, f, indent=2, sort_keys=True) - - @staticmethod - def diff(old: Manifest, new: Manifest) -> SyncDiff: - """Compute what changed between two manifests.""" - old_keys = set(old.entries) - new_keys = set(new.entries) - - added = sorted(new_keys - old_keys) - removed = sorted(old_keys - new_keys) - changed = sorted( - k for k in old_keys & new_keys if old.entries[k] != new.entries[k] - ) - - return SyncDiff(added=added, changed=changed, removed=removed) diff --git a/src/neurostack/cloud/sync.py b/src/neurostack/cloud/sync.py deleted file mode 100644 index 05da22d..0000000 --- a/src/neurostack/cloud/sync.py +++ /dev/null @@ -1,622 +0,0 @@ -# SPDX-License-Identifier: Apache-2.0 -# Copyright (c) 2024-2026 Raphael Southall -"""Vault sync engine for push, pull, and cloud query operations. - -Orchestrates all cloud API interactions: uploading vault files, -polling indexing status, downloading the indexed DB, and sending -remote queries. -""" - -from __future__ import annotations - -import hashlib -import io -import json -import logging -import tarfile -import tempfile -import time -from collections.abc import Callable -from datetime import datetime, timezone -from pathlib import Path - -import httpx - -from .manifest import Manifest - -logger = logging.getLogger(__name__) - - -NEUROSTACKIGNORE_FILE = ".neurostackignore" - - -class SyncError(Exception): - """Raised when a sync operation fails.""" - - -class ConsentError(SyncError): - """Raised when cloud consent has not been given.""" - - -class VaultSyncEngine: - """Orchestrates vault push, DB pull, and cloud query. - - Uses httpx directly for tar.gz upload, polling, and streaming - download. All HTTP calls include Bearer auth headers. - """ - - def __init__( - self, - cloud_api_url: str, - cloud_api_key: str, - vault_root: Path, - db_dir: Path, - manifest_path: Path | None = None, - poll_interval: float = 5.0, - poll_timeout: float = 3600.0, - consent_given: bool = True, - ) -> None: - self._api_url = cloud_api_url.rstrip("/") - self._api_key = cloud_api_key - self._vault_root = vault_root - self._db_dir = db_dir - self._manifest_path = manifest_path or ( - vault_root / ".neurostack" / "cloud-manifest.json" - ) - self._poll_interval = poll_interval - self._poll_timeout = poll_timeout - self._consent_given = consent_given - - def _headers(self) -> dict[str, str]: - """Build Bearer auth header.""" - return {"Authorization": f"Bearer {self._api_key}"} - - def _build_tar_archive( - self, upload_files: list[str], diff: object - ) -> bytes: - """Pack upload files and manifest into a tar.gz archive. - - Creates a tar.gz containing: - - ``_manifest.json`` with format_version, removed list, and file_hashes - - Each .md file from *upload_files* - - Args: - upload_files: Relative paths of files to include. - diff: A ``SyncDiff`` instance (uses ``.removed``). - - Returns: - The tar.gz archive as raw bytes. - """ - buf = io.BytesIO() - file_hashes: dict[str, str] = {} - - with tarfile.open(fileobj=buf, mode="w:gz") as tar: - # Add each vault file - for rel_path in upload_files: - full_path = self._vault_root / rel_path - content = full_path.read_bytes() - file_hashes[rel_path] = ( - "sha256:" + hashlib.sha256(content).hexdigest() - ) - info = tarfile.TarInfo(name=rel_path) - info.size = len(content) - tar.addfile(info, io.BytesIO(content)) - - # Build and add _manifest.json - manifest_data = { - "format_version": 1, - "removed": list(getattr(diff, "removed", [])), - "file_hashes": file_hashes, - } - manifest_bytes = json.dumps(manifest_data).encode() - info = tarfile.TarInfo(name="_manifest.json") - info.size = len(manifest_bytes) - tar.addfile(info, io.BytesIO(manifest_bytes)) - - return buf.getvalue() - - def _push_changes( - self, - progress_callback: Callable[[str], None] | None = None, - ) -> dict: - """Shared push logic: scan, diff, upload, poll, save manifest. - - Returns a result dict with status/upload_stats. Used by both - ``push()`` and the push phase of ``sync()``. - """ - ignore_path = self._vault_root / NEUROSTACKIGNORE_FILE - new_manifest = Manifest.scan_vault( - self._vault_root, - ignore_file=ignore_path if ignore_path.exists() else None, - ) - old_manifest = Manifest.load(self._manifest_path) - diff = Manifest.diff(old_manifest, new_manifest) - - if not diff.has_changes: - logger.info("No changes detected, skipping upload") - if progress_callback: - progress_callback("No changes detected") - return { - "status": "no_changes", - "message": "Vault is up to date", - "upload_stats": { - "files_uploaded": 0, - "raw_bytes": 0, - "compressed_bytes": 0, - "compression_ratio": 0.0, - }, - } - - upload_files = diff.upload_files - logger.info( - "Uploading %d files (%d added, %d changed, %d removed)", - len(upload_files), - len(diff.added), - len(diff.changed), - len(diff.removed), - ) - - archive_data = self._build_tar_archive(upload_files, diff) - - total_raw_bytes = sum( - (self._vault_root / rel_path).stat().st_size for rel_path in upload_files - ) - archive_bytes = len(archive_data) - compression_ratio = ( - (1 - archive_bytes / total_raw_bytes) * 100 if total_raw_bytes > 0 else 0 - ) - - if progress_callback: - progress_callback( - f"Uploading {len(upload_files)} files " - f"({archive_bytes / 1024:.1f} KB, " - f"{compression_ratio:.0f}% compression)" - ) - - with httpx.Client(headers=self._headers(), timeout=300.0) as client: - headers = { - "Content-Type": "application/gzip", - "X-Upload-Format": "tar.gz", - } - resp = client.post( - f"{self._api_url}/v1/vault/upload", - content=archive_data, - headers=headers, - ) - resp.raise_for_status() - upload_data = resp.json() - - job_id = upload_data["job_id"] - logger.info("Upload accepted, job_id=%s", job_id) - - if progress_callback: - progress_callback(f"Upload complete ({archive_bytes / 1024:.1f} KB sent)") - - if progress_callback: - progress_callback(f"Upload accepted, polling job {job_id}...") - - result = self._poll_job(client, job_id, progress_callback) - - new_manifest.save(self._manifest_path) - logger.info("Manifest saved to %s", self._manifest_path) - - result["upload_stats"] = { - "files_uploaded": len(upload_files), - "raw_bytes": total_raw_bytes, - "compressed_bytes": archive_bytes, - "compression_ratio": round(compression_ratio, 1), - } - - return result - - def push( - self, *, progress_callback: Callable[[str], None] | None = None - ) -> dict: - """Upload changed vault files and wait for indexing.""" - if not self._consent_given: - raise ConsentError( - "Cloud consent not given. Run `neurostack cloud consent` " - "or `neurostack init --cloud` to grant consent." - ) - return self._push_changes(progress_callback) - - def _poll_job( - self, - client: httpx.Client, - job_id: str, - progress_callback: Callable[[str], None] | None = None, - ) -> dict: - """Poll job status until terminal state. - - Raises SyncError on failure or timeout. - """ - start = time.monotonic() - url = f"{self._api_url}/v1/vault/status/{job_id}" - - while True: - elapsed = time.monotonic() - start - if elapsed >= self._poll_timeout: - raise SyncError( - f"Indexing timed out after {self._poll_timeout}s " - f"for job {job_id}" - ) - - resp = client.get(url) - resp.raise_for_status() - data = resp.json() - - status = data.get("status", "unknown") - logger.info( - "Job %s: status=%s, progress=%s", - job_id, - status, - data.get("progress"), - ) - - if progress_callback: - progress = data.get("progress") - msg = f"Job {job_id}: {status}" - if progress is not None: - msg += f" ({progress:.0%})" - progress_callback(msg) - - if status == "complete": - return data - - if status == "failed": - error = data.get("error", "Unknown indexing error") - raise SyncError(f"Indexing failed for job {job_id}: {error}") - - time.sleep(self._poll_interval) - - def pull(self, *, db_path: Path | None = None) -> Path: - """Download indexed DB from cloud. - - Steps: - 1. GET /v1/vault/download -> presigned URL - 2. Stream download to temp file (separate client, no auth headers) - 3. Verify downloaded size matches Content-Length - 4. Atomic rename to db_dir/neurostack.db - 5. Return path to downloaded DB - """ - target = db_path or (self._db_dir / "neurostack.db") - target.parent.mkdir(parents=True, exist_ok=True) - - # Step 1: Get presigned download URL (authenticated) - with httpx.Client(headers=self._headers(), timeout=60.0) as client: - resp = client.get(f"{self._api_url}/v1/vault/download") - resp.raise_for_status() - download_info = resp.json() - download_url = download_info["download_url"] - - logger.info("Downloading DB from presigned URL...") - - # Step 2: Stream download with a SEPARATE client -- no auth headers - # (GCS signed URLs reject extra Authorization headers) and a longer - # timeout suitable for large files (72MB+). - download_timeout = httpx.Timeout( - connect=30.0, read=300.0, write=30.0, pool=30.0 - ) - bytes_written = 0 - with tempfile.NamedTemporaryFile( - dir=str(target.parent), delete=False, suffix=".tmp" - ) as tmp_file: - tmp_path = Path(tmp_file.name) - try: - with httpx.Client(timeout=download_timeout) as dl_client: - with dl_client.stream("GET", download_url) as stream: - stream.raise_for_status() - expected_size = stream.headers.get("content-length") - for chunk in stream.iter_bytes(chunk_size=131072): - tmp_file.write(chunk) - bytes_written += len(chunk) - except Exception: - # Clean up temp file on failure - tmp_path.unlink(missing_ok=True) - raise - - # Step 3: Verify download integrity - if expected_size is not None: - expected = int(expected_size) - if bytes_written != expected: - tmp_path.unlink(missing_ok=True) - raise SyncError( - f"Download incomplete: got {bytes_written} bytes, " - f"expected {expected} bytes" - ) - - logger.info( - "Downloaded %d bytes to temp file", bytes_written - ) - - # Step 4: Remove stale WAL/SHM from previous DB before replacing - for suffix in ("-wal", "-shm"): - stale = target.with_name(target.name + suffix) - if stale.exists(): - stale.unlink() - - # Step 5: Atomic rename - tmp_path.rename(target) - logger.info("DB saved to %s (%d bytes)", target, bytes_written) - - return target - - def query( - self, - search_text: str, - *, - top_k: int = 10, - depth: str = "auto", - mode: str = "hybrid", - workspace: str | None = None, - ) -> dict: - """Query the cloud-indexed vault. - - Sends POST /v1/vault/query with search params. - Returns dict with triples, summaries, chunks, and depth_used. - """ - with httpx.Client(headers=self._headers(), timeout=30.0) as client: - body: dict = { - "query": search_text, - "top_k": top_k, - "depth": depth, - "mode": mode, - } - if workspace: - body["workspace"] = workspace - - resp = client.post(f"{self._api_url}/v1/vault/query", json=body) - - try: - resp.raise_for_status() - except httpx.HTTPStatusError: - if resp.status_code == 501: - raise SyncError( - "Cloud query API not yet available. " - "This feature is coming in a future release." - ) - raise - - return resp.json() - - def sync( - self, - *, - progress_callback: Callable[[str], None] | None = None, - ) -> dict: - """Push vault changes and fetch new memories from cloud. - - Steps: - 1. Check consent - 2. Push changes (shared with push()) - 3. Fetch new memories from cloud since last sync - 4. Merge memories into local SQLite - 5. Return combined result dict - """ - if not self._consent_given: - raise ConsentError( - "Cloud consent not given. Run `neurostack cloud consent` " - "or `neurostack init --cloud` to grant consent." - ) - - push_result = self._push_changes(progress_callback) - - # --- Memory fetch phase --- - if progress_callback: - progress_callback("Fetching new memories from cloud...") - - last_sync_time = self._load_last_sync_time() - memories_fetched = 0 - server_time: str | None = None - - try: - with httpx.Client(headers=self._headers(), timeout=30.0) as client: - params: dict[str, str | int] = {"limit": 100} - if last_sync_time: - params["after"] = last_sync_time - - resp = client.get( - f"{self._api_url}/v1/vault/memories/since", - params=params, - ) - resp.raise_for_status() - mem_data = resp.json() - - memories = mem_data.get("memories", []) - server_time = mem_data.get("server_time") - memories_fetched = len(memories) - - if memories: - self._merge_memories(memories) - logger.info("Merged %d memories into local DB", memories_fetched) - else: - logger.info("No new memories from cloud") - - if progress_callback: - progress_callback(f"Fetched {memories_fetched} memories") - - except httpx.HTTPStatusError as exc: - if exc.response.status_code == 404: - logger.warning("Memories endpoint not available (404), skipping") - if progress_callback: - progress_callback("Memories endpoint not available, skipped") - else: - raise - - # Save sync time - if server_time: - self._save_last_sync_time(server_time) - - return { - **push_result, - "memories_fetched": memories_fetched, - } - - def _sync_meta_path(self) -> Path: - """Path to the cloud sync metadata file.""" - return Path.home() / ".local" / "share" / "neurostack" / "cloud-sync-meta.json" - - def _load_last_sync_time(self) -> str | None: - """Load last_sync_time from sync metadata file.""" - meta_path = self._sync_meta_path() - if not meta_path.exists(): - return None - try: - data = json.loads(meta_path.read_text()) - return data.get("last_sync_time") - except (json.JSONDecodeError, OSError): - return None - - def _save_last_sync_time(self, server_time: str) -> None: - """Save last_sync_time to sync metadata file.""" - meta_path = self._sync_meta_path() - meta_path.parent.mkdir(parents=True, exist_ok=True) - meta_path.write_text(json.dumps({"last_sync_time": server_time})) - - def get_staleness(self) -> dict: - """Compute staleness between local vault and last cloud sync. - - Compares the latest modification time of any .md file in the vault - against the last_sync_time stored in cloud-sync-meta.json. Uses - stat-only checks (no file reads) for performance. - - Returns: - Dict with is_stale, local_latest, last_sync, stale_since, - stale_files_count, and behind_hours fields. - """ - # Find the latest mtime of any .md file in the vault - local_latest_dt: datetime | None = None - stale_files_count = 0 - stale_since_dt: datetime | None = None - - last_sync_str = self._load_last_sync_time() - last_sync_dt: datetime | None = None - if last_sync_str: - try: - last_sync_dt = datetime.fromisoformat(last_sync_str) - if last_sync_dt.tzinfo is None: - last_sync_dt = last_sync_dt.replace(tzinfo=timezone.utc) - except (ValueError, TypeError): - last_sync_dt = None - - for md_file in self._vault_root.rglob("*.md"): - # Skip dot-directories (.obsidian, .git, .neurostack) - if any(part.startswith(".") for part in md_file.relative_to(self._vault_root).parts): - continue - mtime = datetime.fromtimestamp(md_file.stat().st_mtime, tz=timezone.utc) - if local_latest_dt is None or mtime > local_latest_dt: - local_latest_dt = mtime - - if last_sync_dt is not None and mtime > last_sync_dt: - stale_files_count += 1 - if stale_since_dt is None or mtime < stale_since_dt: - stale_since_dt = mtime - - # No sync meta at all => stale (never synced) - if last_sync_dt is None: - return { - "is_stale": True, - "local_latest": local_latest_dt.isoformat() if local_latest_dt else None, - "last_sync": None, - "stale_since": None, - "stale_files_count": 0, - "behind_hours": None, - } - - is_stale = local_latest_dt is not None and local_latest_dt > last_sync_dt - - behind_hours: float | None = None - if is_stale and local_latest_dt is not None: - behind_hours = round( - (local_latest_dt - last_sync_dt).total_seconds() / 3600, 1 - ) - - return { - "is_stale": is_stale, - "local_latest": local_latest_dt.isoformat() if local_latest_dt else None, - "last_sync": last_sync_str, - "stale_since": stale_since_dt.isoformat() if stale_since_dt else None, - "stale_files_count": stale_files_count, - "behind_hours": behind_hours, - } - - def _check_neurostackignore_exists(self) -> bool: - """Check whether a .neurostackignore file exists in the vault root.""" - return (self._vault_root / NEUROSTACKIGNORE_FILE).is_file() - - def _load_ignore_patterns(self) -> list[str]: - """Load ignore patterns from .neurostackignore in vault root. - - Returns a list of pattern strings (one per non-empty, non-comment line). - """ - ignore_path = self._vault_root / NEUROSTACKIGNORE_FILE - if not ignore_path.is_file(): - return [] - - patterns: list[str] = [] - for line in ignore_path.read_text().splitlines(): - stripped = line.strip() - if stripped and not stripped.startswith("#"): - patterns.append(stripped) - return patterns - - def _merge_memories(self, memories: list[dict]) -> None: - """Merge fetched memories into local SQLite DB. - - Uses INSERT ... ON CONFLICT to upsert by UUID, preserving - existing fields (embedding, revision_count, merge_count, - merged_from) not present in the cloud response. - Deletes memories marked with deleted=true. - """ - from ..schema import get_db - - db_path = self._db_dir / "neurostack.db" - if not db_path.exists(): - logger.warning("Local DB not found at %s, skipping memory merge", db_path) - return - - conn = get_db(db_path) - try: - for mem in memories: - if mem.get("deleted"): - conn.execute( - "DELETE FROM memories WHERE uuid = ?", - (mem["uuid"],), - ) - else: - tags = mem.get("tags") - if isinstance(tags, list): - tags = json.dumps(tags) - conn.execute( - "INSERT INTO memories " - "(uuid, content, entity_type, tags, workspace, " - "source_agent, session_id, expires_at, " - "created_at, updated_at, file_path) " - "VALUES (?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?) " - "ON CONFLICT(uuid) DO UPDATE SET " - "content = excluded.content, " - "entity_type = excluded.entity_type, " - "tags = excluded.tags, " - "workspace = excluded.workspace, " - "source_agent = excluded.source_agent, " - "session_id = excluded.session_id, " - "expires_at = excluded.expires_at, " - "updated_at = excluded.updated_at, " - "file_path = excluded.file_path", - ( - mem["uuid"], - mem.get("content", ""), - mem.get("entity_type", "observation"), - tags, - mem.get("workspace"), - mem.get("source_agent"), - mem.get("session_id"), - mem.get("expires_at"), - mem.get("created_at"), - mem.get("updated_at"), - mem.get("file_path"), - ), - ) - conn.commit() - finally: - conn.close() diff --git a/src/neurostack/cloud/timer.py b/src/neurostack/cloud/timer.py deleted file mode 100644 index 45ed8a1..0000000 --- a/src/neurostack/cloud/timer.py +++ /dev/null @@ -1,203 +0,0 @@ -# SPDX-License-Identifier: Apache-2.0 -# Copyright (c) 2024-2026 Raphael Southall -"""systemd user timer for periodic cloud sync. - -Installs a systemd user service and timer that runs -``neurostack cloud sync --quiet`` on a configurable interval. -""" -from __future__ import annotations - -import subprocess -from pathlib import Path - -SERVICE_NAME = "neurostack-cloud-sync" - -SERVICE_UNIT = """\ -[Unit] -Description=NeuroStack Cloud Sync -After=network-online.target -Wants=network-online.target - -[Service] -Type=oneshot -ExecStart={neurostack_path} cloud sync --quiet -Environment=PATH=/usr/local/bin:/usr/bin:/bin -""" - -TIMER_UNIT = """\ -[Unit] -Description=NeuroStack Cloud Sync Timer - -[Timer] -OnBootSec=5min -OnUnitActiveSec={interval} -Persistent=true - -[Install] -WantedBy=timers.target -""" - - -def _systemd_user_dir() -> Path: - """Return ~/.config/systemd/user/ directory.""" - return Path.home() / ".config" / "systemd" / "user" - - -def _find_neurostack_path() -> str: - """Find the neurostack executable path.""" - import shutil - path = shutil.which("neurostack") - if path: - return path - # Fallback: try common locations - for candidate in [ - Path.home() / ".local" / "bin" / "neurostack", - Path("/usr/local/bin/neurostack"), - Path("/usr/bin/neurostack"), - ]: - if candidate.exists(): - return str(candidate) - return "neurostack" # Hope it's on PATH at runtime - - -def install_timer(interval: str = "15min") -> dict: - """Install systemd user timer for periodic sync. - - Args: - interval: Sync interval (systemd time format: 5min, 1h, 30s, etc.) - - Returns: - { - "service_path": str, - "timer_path": str, - "interval": str, - "enabled": bool, - } - """ - user_dir = _systemd_user_dir() - user_dir.mkdir(parents=True, exist_ok=True) - - neurostack_path = _find_neurostack_path() - - service_path = user_dir / f"{SERVICE_NAME}.service" - timer_path = user_dir / f"{SERVICE_NAME}.timer" - - # Write service unit - service_path.write_text( - SERVICE_UNIT.format(neurostack_path=neurostack_path) - ) - - # Write timer unit - timer_path.write_text( - TIMER_UNIT.format(interval=interval) - ) - - # Reload systemd and enable timer - enabled = False - try: - subprocess.run( - ["systemctl", "--user", "daemon-reload"], - check=True, capture_output=True, - ) - subprocess.run( - ["systemctl", "--user", "enable", "--now", f"{SERVICE_NAME}.timer"], - check=True, capture_output=True, - ) - enabled = True - except (subprocess.CalledProcessError, FileNotFoundError): - pass # systemctl may not be available in containers/CI - - return { - "service_path": str(service_path), - "timer_path": str(timer_path), - "interval": interval, - "enabled": enabled, - } - - -def uninstall_timer() -> dict: - """Remove systemd user timer. - - Returns: - {"removed": bool, "paths": list[str]} - """ - user_dir = _systemd_user_dir() - service_path = user_dir / f"{SERVICE_NAME}.service" - timer_path = user_dir / f"{SERVICE_NAME}.timer" - - # Stop and disable - try: - subprocess.run( - ["systemctl", "--user", "disable", "--now", f"{SERVICE_NAME}.timer"], - check=False, capture_output=True, - ) - subprocess.run( - ["systemctl", "--user", "daemon-reload"], - check=False, capture_output=True, - ) - except FileNotFoundError: - pass - - removed_paths = [] - for path in (service_path, timer_path): - if path.exists(): - path.unlink() - removed_paths.append(str(path)) - - return { - "removed": len(removed_paths) > 0, - "paths": removed_paths, - } - - -def timer_status() -> dict: - """Check if the systemd timer is installed and active. - - Returns: - { - "installed": bool, - "active": bool, - "interval": str | None, - "next_run": str | None, - } - """ - user_dir = _systemd_user_dir() - timer_path = user_dir / f"{SERVICE_NAME}.timer" - - if not timer_path.exists(): - return {"installed": False, "active": False, "interval": None, "next_run": None} - - # Parse interval from timer file - interval = None - for line in timer_path.read_text().splitlines(): - if line.startswith("OnUnitActiveSec="): - interval = line.split("=", 1)[1].strip() - break - - # Check if active - active = False - next_run = None - try: - result = subprocess.run( - ["systemctl", "--user", "is-active", f"{SERVICE_NAME}.timer"], - capture_output=True, text=True, - ) - active = result.returncode == 0 - - if active: - result = subprocess.run( - ["systemctl", "--user", "show", f"{SERVICE_NAME}.timer", - "--property=NextElapseUSecRealtime"], - capture_output=True, text=True, - ) - if result.returncode == 0: - next_run = result.stdout.strip().split("=", 1)[-1] - except FileNotFoundError: - pass - - return { - "installed": True, - "active": active, - "interval": interval, - "next_run": next_run, - } diff --git a/src/neurostack/config.py b/src/neurostack/config.py index 48e4617..5581cfc 100644 --- a/src/neurostack/config.py +++ b/src/neurostack/config.py @@ -34,7 +34,7 @@ def _config_dir() -> Path: class Config: """NeuroStack configuration with env var overrides.""" - mode: str = "local" # "local" | "cloud" + mode: str = "local" # "local" vault_root: Path = field(default_factory=lambda: Path.home() / "brain") db_dir: Path = field(default_factory=_data_dir) embed_url: str = "http://localhost:11434" @@ -52,10 +52,6 @@ class Config: api_key: str = "" cooccurrence_boost_weight: float = 0.1 - @property - def is_cloud(self) -> bool: - return self.mode == "cloud" - @property def db_path(self) -> Path: return self.db_dir / "neurostack.db" diff --git a/src/neurostack/tools/mcp_adapter.py b/src/neurostack/tools/mcp_adapter.py index ec79199..389bbf6 100644 --- a/src/neurostack/tools/mcp_adapter.py +++ b/src/neurostack/tools/mcp_adapter.py @@ -26,9 +26,6 @@ def create_mcp_server(name: str = "neurostack", **fastmcp_kwargs) -> FastMCP: """Create a FastMCP server with all registry tools auto-registered. - If config mode is "cloud", tool functions are replaced with CloudClient - proxies before registration — all adapters pick up the change. - Args: name: MCP server name **fastmcp_kwargs: Passed through to FastMCP constructor @@ -36,28 +33,6 @@ def create_mcp_server(name: str = "neurostack", **fastmcp_kwargs) -> FastMCP: mcp = FastMCP(name, **fastmcp_kwargs) registry = ensure_registered() - # Cloud dispatch: swap local tool functions with cloud API proxies - from neurostack.config import get_config - cfg = get_config() - if cfg.is_cloud: - try: - from neurostack.cloud.client import CloudClient - from neurostack.cloud.config import load_cloud_config - from neurostack.cloud.dispatch import enable_cloud_dispatch - - cloud_cfg = load_cloud_config() - client = CloudClient(cloud_cfg) - if client.is_configured: - enable_cloud_dispatch(registry, client) - else: - log.warning( - "Cloud mode enabled but no cloud credentials configured. " - "Run 'neurostack cloud login' to authenticate. " - "Falling back to local tools." - ) - except Exception: - log.exception("Failed to enable cloud dispatch — falling back to local") - for tool_def in registry.list_tools(): @functools.wraps(tool_def.fn) async def wrapper(_td=tool_def, **kwargs): @@ -70,13 +45,11 @@ async def wrapper(_td=tool_def, **kwargs): mcp_annotations = None if tool_def.annotations: hints = tool_def.annotations - # In cloud mode, all tools contact an external service - open_world = True if cfg.is_cloud else hints.open_world mcp_annotations = ToolAnnotations( readOnlyHint=hints.read_only, destructiveHint=hints.destructive, idempotentHint=hints.idempotent, - openWorldHint=open_world, + openWorldHint=hints.open_world, ) mcp.tool(annotations=mcp_annotations)(wrapper) diff --git a/src/neurostack/watcher.py b/src/neurostack/watcher.py index 3640378..5c0afbb 100644 --- a/src/neurostack/watcher.py +++ b/src/neurostack/watcher.py @@ -1,12 +1,6 @@ # SPDX-License-Identifier: Apache-2.0 # Copyright (c) 2024-2026 Raphael Southall -"""Watchdog-based file watcher with debounce for vault indexing. - -Supports optional cloud sync via ``--cloud`` flag or ``cloud.auto_push`` -config toggle. When enabled, vault changes are pushed to NeuroStack Cloud -after an idle period (no file changes for ``CLOUD_IDLE_SECONDS``). The push -runs in a background thread so it never blocks local indexing. -""" +"""Watchdog-based file watcher with debounce for vault indexing.""" import hashlib import json @@ -15,7 +9,7 @@ import time from datetime import datetime, timezone from pathlib import Path -from threading import Lock, Thread, Timer +from threading import Lock, Timer from watchdog.events import FileSystemEventHandler from watchdog.observers import Observer @@ -45,8 +39,6 @@ log = logging.getLogger("neurostack.indexer") DEBOUNCE_SECONDS = 2.0 -CLOUD_IDLE_SECONDS = 60.0 # Push to cloud after 60s of no file changes -CLOUD_RETRY_DELAYS = (5, 15, 45) # Exponential backoff for push failures def _vault_root(): @@ -54,96 +46,6 @@ def _vault_root(): return get_config().vault_root -class CloudPusher: - """Background cloud push with idle detection and retry. - - Tracks file changes and triggers a cloud push after the vault has been - idle for ``CLOUD_IDLE_SECONDS``. The push runs in a daemon thread so - it never blocks local indexing. Retries with exponential backoff on - transient failures; the 15-min systemd timer acts as ultimate fallback. - """ - - def __init__(self, vault_root: Path) -> None: - self._vault_root = vault_root - self._idle_timer: Timer | None = None - self._lock = Lock() - self._push_in_progress = False - - def notify_change(self) -> None: - """Called by DebouncedHandler on every file event. Resets idle timer.""" - with self._lock: - if self._idle_timer is not None: - self._idle_timer.cancel() - self._idle_timer = Timer(CLOUD_IDLE_SECONDS, self._trigger_push) - self._idle_timer.daemon = True - self._idle_timer.start() - - def _trigger_push(self) -> None: - """Fire a cloud push in a background thread.""" - with self._lock: - if self._push_in_progress: - log.debug("Cloud push already in progress, skipping") - return - self._push_in_progress = True - - thread = Thread(target=self._do_push, daemon=True) - thread.start() - - def _do_push(self) -> None: - """Execute cloud push with retry. Runs in a daemon thread.""" - try: - from .cloud.config import load_cloud_config - from .cloud.sync import ConsentError, SyncError, VaultSyncEngine - from .config import get_config - - cfg = get_config() - cloud_cfg = load_cloud_config() - if not cloud_cfg.cloud_api_url or not cloud_cfg.cloud_api_key: - log.debug("Cloud not configured, skipping push") - return - - engine = VaultSyncEngine( - cloud_api_url=cloud_cfg.cloud_api_url, - cloud_api_key=cloud_cfg.cloud_api_key, - vault_root=self._vault_root, - db_dir=Path(cfg.db_path).parent if hasattr(cfg, "db_path") else self._vault_root, - ) - - for attempt, delay in enumerate(CLOUD_RETRY_DELAYS): - try: - result = engine.push( - progress_callback=lambda msg: log.info("Cloud: %s", msg), - ) - status = result.get("status", "unknown") - if status == "no_changes": - log.debug("Cloud push: no changes") - else: - uploaded = result.get("upload_stats", {}).get("files_uploaded", 0) - log.info("Cloud push complete: %d files synced", uploaded) - return - except ConsentError: - log.debug("Cloud consent not given, skipping push") - return - except (SyncError, Exception) as exc: - log.warning( - "Cloud push attempt %d/%d failed: %s", - attempt + 1, len(CLOUD_RETRY_DELAYS), exc, - ) - if attempt < len(CLOUD_RETRY_DELAYS) - 1: - time.sleep(delay) - - log.error( - "Cloud push failed after %d attempts (timer will retry)", - len(CLOUD_RETRY_DELAYS), - ) - - except Exception as exc: - log.error("Cloud push error: %s", exc) - finally: - with self._lock: - self._push_in_progress = False - - class DebouncedHandler(FileSystemEventHandler): """Debounces file changes and triggers indexing.""" @@ -153,7 +55,6 @@ def __init__( embed_url: str, summarize_url: str, exclude_dirs: list[str] | None = None, - cloud_pusher: CloudPusher | None = None, ): self.vault_root = vault_root self.embed_url = embed_url @@ -163,7 +64,6 @@ def __init__( self._exclude_dirs = set( exclude_dirs or [], ) - self._cloud_pusher = cloud_pusher # Reuse a single DB connection across events (WAL mode is safe for # concurrent reads). The connection is created lazily on first use. self._conn = None @@ -185,10 +85,6 @@ def on_any_event(self, event): if not self._should_process(path): return - # Notify cloud pusher (resets idle timer) - if self._cloud_pusher is not None: - self._cloud_pusher.notify_change() - # Debounce local indexing with self._timers_lock: if path in self._timers: @@ -975,41 +871,17 @@ def run_watcher( embed_url: str = None, summarize_url: str = None, exclude_dirs: list[str] | None = None, - cloud: bool = False, ): - """Run the watchdog file watcher. - - Args: - cloud: Enable automatic cloud push after idle period. Can also - be enabled via ``cloud.auto_push = true`` in config.toml. - """ + """Run the watchdog file watcher.""" vault_root = vault_root or _vault_root() embed_url = embed_url or get_config().embed_url summarize_url = summarize_url or get_config().llm_url - # Check config toggle if flag not passed - if not cloud: - try: - from .cloud.config import load_cloud_config - cfg = load_cloud_config() - cloud = bool(getattr(cfg, "auto_push", False)) - except Exception: - pass - - cloud_pusher = None - if cloud: - cloud_pusher = CloudPusher(vault_root) - log.info( - "Watching %s for changes (cloud sync enabled, %ds idle push)...", - vault_root, int(CLOUD_IDLE_SECONDS), - ) - else: - log.info(f"Watching {vault_root} for changes...") + log.info(f"Watching {vault_root} for changes...") handler = DebouncedHandler( vault_root, embed_url, summarize_url, exclude_dirs=exclude_dirs, - cloud_pusher=cloud_pusher, ) observer = Observer() observer.schedule(handler, str(vault_root), recursive=True) diff --git a/tests/conftest.py b/tests/conftest.py index aa4e47e..da85918 100644 --- a/tests/conftest.py +++ b/tests/conftest.py @@ -3,7 +3,6 @@ import json import sqlite3 import textwrap -from unittest.mock import MagicMock, patch import pytest @@ -144,214 +143,3 @@ def populated_db(in_memory_db, tmp_vault): conn.commit() return conn - - -# --------------------------------------------------------------------------- -# Firebase / Firestore test fixtures -# --------------------------------------------------------------------------- - -MOCK_FIREBASE_DECODED_TOKEN = { - "uid": "firebase-user-123", - "email": "test@example.com", - "firebase": { - "sign_in_provider": "google.com", - }, -} - - -@pytest.fixture -def mock_firebase_admin(): - """Patch firebase_admin.auth.verify_id_token to return a mock decoded token. - - Also patches firebase_admin.initialize_app to avoid real init. - """ - with ( - patch("firebase_admin.auth.verify_id_token", return_value=MOCK_FIREBASE_DECODED_TOKEN), - patch("firebase_admin.initialize_app", return_value=MagicMock()), - ): - # Reset the singleton so it re-initializes with mock - import neurostack.cloud.firebase_init as fi - old_app = fi._app - fi._app = None - yield MOCK_FIREBASE_DECODED_TOKEN - fi._app = old_app - - -class MockFirestoreDoc: - """Mock Firestore document snapshot.""" - - def __init__(self, data: dict | None, doc_id: str = "doc", parent=None): - self._data = data - self.exists = data is not None - self.id = doc_id - self.reference = MagicMock() - self.reference.id = doc_id - if parent: - self.reference.parent = MagicMock() - self.reference.parent.parent = parent - - def to_dict(self): - return self._data - - -class MockFirestoreCollection: - """In-memory mock of a Firestore collection/subcollection. - - Supports document CRUD, auto-generated IDs, subcollections, - and async streaming. - """ - - _auto_id_counter = 0 - - def __init__( - self, data: dict | None = None, *, - parent_data: dict | None = None, sub_key: str | None = None, - ): - self._data = data or {} # {doc_id: {field: value}} - # Track parent so subcollection writes propagate - self._parent_data = parent_data - self._sub_key = sub_key - - def _sync_to_parent(self): - """Write subcollection data back to parent's _sub_X key.""" - if self._parent_data is not None and self._sub_key is not None: - self._parent_data[self._sub_key] = self._data - - def document(self, doc_id=None): - if doc_id is None: - MockFirestoreCollection._auto_id_counter += 1 - doc_id = f"auto-{MockFirestoreCollection._auto_id_counter}" - - col = self # capture for closures - doc_ref = MagicMock() - doc_ref.id = doc_id - - async def _get(): - if doc_id in col._data: - return MockFirestoreDoc(col._data[doc_id], doc_id=doc_id) - return MockFirestoreDoc(None, doc_id=doc_id) - - async def _set(data, merge=False): - if merge and doc_id in col._data: - col._data[doc_id].update(data) - else: - col._data[doc_id] = dict(data) - col._sync_to_parent() - - async def _update(data): - if doc_id in col._data: - col._data[doc_id].update(data) - else: - col._data[doc_id] = dict(data) - col._sync_to_parent() - - async def _delete(): - col._data.pop(doc_id, None) - col._sync_to_parent() - - doc_ref.get = _get - doc_ref.set = _set - doc_ref.update = _update - doc_ref.delete = _delete - doc_ref.collection = lambda name: self._get_subcollection(doc_id, name) - return doc_ref - - def _get_subcollection(self, doc_id, name): - """Return a subcollection backed by the parent doc's _sub_X dict.""" - if doc_id not in self._data: - self._data[doc_id] = {} - parent_doc = self._data[doc_id] - sub_key = f"_sub_{name}" - if sub_key not in parent_doc: - parent_doc[sub_key] = {} - return MockFirestoreCollection( - parent_doc[sub_key], - parent_data=parent_doc, - sub_key=sub_key, - ) - - def stream(self): - return self._stream() - - async def _stream(self): - """Yield all documents in the collection.""" - for doc_id, data in list(self._data.items()): - if not doc_id.startswith("_sub_"): - yield MockFirestoreDoc(data, doc_id=doc_id) - - def collection(self, name): - return MockFirestoreCollection() - - -class MockFirestoreClient: - """In-memory mock of google.cloud.firestore_v1.AsyncClient.""" - - def __init__(self): - self._collections = {} # {name: MockFirestoreCollection} - - def collection(self, name): - if name not in self._collections: - self._collections[name] = MockFirestoreCollection() - return self._collections[name] - - def collection_group(self, name): - """Mock collection group query.""" - return MockCollectionGroupQuery(self, name) - - -class MockCollectionGroupQuery: - """Mock collection group query for api_keys.""" - - def __init__(self, client: MockFirestoreClient, collection_name: str): - self._client = client - self._collection_name = collection_name - self._filters = [] - - def where(self, field, op, value): - self._filters.append((field, op, value)) - return self - - def stream(self): - return self._stream() - - async def _stream(self): - """Yield matching docs from all subcollections.""" - users_col = self._client._collections.get("users") - if not users_col: - return - - for uid, user_data in users_col._data.items(): - sub_key = f"_sub_{self._collection_name}" - if sub_key not in user_data: - continue - sub_data = user_data[sub_key] - for key_id, key_data in sub_data.items(): - if self._matches(key_data): - parent_ref = MagicMock() - parent_ref.id = uid - parent_mock = MagicMock() - parent_mock.parent = parent_ref - yield MockFirestoreDoc( - key_data, - doc_id=key_id, - parent=parent_ref, - ) - - def _matches(self, data: dict) -> bool: - for field, op, value in self._filters: - if op == "==": - if data.get(field) != value: - return False - return True - - -@pytest.fixture -def mock_firestore(): - """Provide an in-memory MockFirestoreClient and patch the user_store module.""" - client = MockFirestoreClient() - - # Patch the singleton in user_store - with patch("neurostack.cloud.user_store._db", client): - # Also patch _get_db to return our mock - with patch("neurostack.cloud.user_store._get_db", return_value=client): - yield client diff --git a/tests/test_cloud_cli.py b/tests/test_cloud_cli.py deleted file mode 100644 index 3703355..0000000 --- a/tests/test_cloud_cli.py +++ /dev/null @@ -1,385 +0,0 @@ -# SPDX-License-Identifier: Apache-2.0 -# Copyright (c) 2024-2026 Raphael Southall -"""Tests for neurostack cloud CLI subcommands (login, logout, status, setup).""" - -from __future__ import annotations - -import json -from argparse import Namespace -from unittest.mock import MagicMock, patch - -import pytest - -from neurostack.cloud.config import CloudConfig - -# --------------------------------------------------------------------------- -# Helpers -# --------------------------------------------------------------------------- - -def _make_args(**kwargs) -> Namespace: - """Build a minimal argparse Namespace for cmd_cloud.""" - defaults = { - "command": "cloud", - "cloud_command": None, - "json": False, - "key": None, - } - defaults.update(kwargs) - return Namespace(**defaults) - - -def _mock_cloud_config(url: str = "", key: str = "") -> CloudConfig: - return CloudConfig(cloud_api_url=url, cloud_api_key=key) - - -# --------------------------------------------------------------------------- -# Test 1: login --key with valid key saves config -# --------------------------------------------------------------------------- - -class TestCloudLogin: - @patch("neurostack.cli.cloud.CloudClient") - @patch("neurostack.cli.cloud.save_cloud_config") - @patch("neurostack.cli.cloud.load_cloud_config") - def test_login_valid_key_saves(self, mock_load, mock_save, mock_client_cls, capsys): - """Login with --key and valid key prints success and saves.""" - mock_load.return_value = _mock_cloud_config( - url="https://neurostack-api-911077737485.us-central1.run.app" - ) - mock_client = MagicMock() - mock_client.validate_key.return_value = True - mock_client_cls.return_value = mock_client - - from neurostack.cli.cloud import cmd_cloud - args = _make_args(cloud_command="login", key="sk-test123") - cmd_cloud(args) - - out = capsys.readouterr().out - assert "Logged in" in out - mock_save.assert_called_once() - call_kwargs = mock_save.call_args - assert call_kwargs[1]["cloud_api_key"] == "sk-test123" - - # Test 2: login --key with invalid key does NOT save - @patch("neurostack.cli.cloud.CloudClient") - @patch("neurostack.cli.cloud.save_cloud_config") - @patch("neurostack.cli.cloud.load_cloud_config") - def test_login_invalid_key_errors(self, mock_load, mock_save, mock_client_cls, capsys): - """Login with invalid key prints error, does not save, exits 1.""" - mock_load.return_value = _mock_cloud_config( - url="https://neurostack-api-911077737485.us-central1.run.app" - ) - mock_client = MagicMock() - mock_client.validate_key.return_value = False - mock_client_cls.return_value = mock_client - - from neurostack.cli.cloud import cmd_cloud - args = _make_args(cloud_command="login", key="sk-bad") - with pytest.raises(SystemExit, match="1"): - cmd_cloud(args) - - out = capsys.readouterr().out - assert "Invalid API key" in out - mock_save.assert_not_called() - - # Test 3: login without --key triggers device code flow - @patch("neurostack.cli.cloud._cmd_cloud_device_login") - def test_login_no_key_triggers_device_flow(self, mock_device_login, capsys): - """Login without --key delegates to device code flow.""" - from neurostack.cli.cloud import cmd_cloud - args = _make_args(cloud_command="login", key=None) - cmd_cloud(args) - - mock_device_login.assert_called_once() - - -# --------------------------------------------------------------------------- -# Test 4: logout clears credentials -# --------------------------------------------------------------------------- - -class TestCloudLogout: - @patch("neurostack.cli.cloud.clear_cloud_credentials") - def test_logout_clears_and_confirms(self, mock_clear, capsys): - """Logout calls clear_cloud_credentials and prints confirmation.""" - from neurostack.cli.cloud import cmd_cloud - args = _make_args(cloud_command="logout") - cmd_cloud(args) - - mock_clear.assert_called_once() - out = capsys.readouterr().out - assert "Logged out" in out - assert "credentials cleared" in out.lower() - - -# --------------------------------------------------------------------------- -# Tests 5-7: status command -# --------------------------------------------------------------------------- - -class TestCloudStatus: - # Test 5: status when authenticated - @patch("neurostack.cli.cloud.CloudClient") - @patch("neurostack.cli.cloud.load_cloud_config") - def test_status_authenticated(self, mock_load, mock_client_cls, capsys): - """Status shows Authenticated, cloud URL, and tier when configured.""" - mock_load.return_value = _mock_cloud_config( - url="https://neurostack-api-911077737485.us-central1.run.app", - key="sk-valid", - ) - mock_client = MagicMock() - mock_client.is_configured = True - mock_client.status.return_value = { - "authenticated": True, - "tier": "free", - "cloud_url": "https://neurostack-api-911077737485.us-central1.run.app", - } - mock_client_cls.return_value = mock_client - - from neurostack.cli.cloud import cmd_cloud - args = _make_args(cloud_command="status") - cmd_cloud(args) - - out = capsys.readouterr().out - assert "Authenticated" in out - assert "neurostack-api" in out - assert "free" in out - - # Test 6: status when NOT authenticated - @patch("neurostack.cli.cloud.CloudClient") - @patch("neurostack.cli.cloud.load_cloud_config") - def test_status_not_authenticated(self, mock_load, mock_client_cls, capsys): - """Status shows 'Not authenticated' when no key is stored.""" - mock_load.return_value = _mock_cloud_config(url="", key="") - mock_client = MagicMock() - mock_client.is_configured = False - mock_client_cls.return_value = mock_client - - from neurostack.cli.cloud import cmd_cloud - args = _make_args(cloud_command="status") - cmd_cloud(args) - - out = capsys.readouterr().out - assert "Not authenticated" in out - - # Test 7: status --json returns machine-readable JSON - @patch("neurostack.cli.cloud.CloudClient") - @patch("neurostack.cli.cloud.load_cloud_config") - def test_status_json_output(self, mock_load, mock_client_cls, capsys): - """status --json returns JSON with authenticated, cloud_url, tier.""" - mock_load.return_value = _mock_cloud_config( - url="https://neurostack-api-911077737485.us-central1.run.app", - key="sk-valid", - ) - mock_client = MagicMock() - mock_client.is_configured = True - mock_client.status.return_value = { - "authenticated": True, - "tier": "free", - "cloud_url": "https://neurostack-api-911077737485.us-central1.run.app", - } - mock_client_cls.return_value = mock_client - - from neurostack.cli.cloud import cmd_cloud - args = _make_args(cloud_command="status", json=True) - cmd_cloud(args) - - out = capsys.readouterr().out - data = json.loads(out) - assert data["authenticated"] is True - assert "cloud_url" in data - assert data["tier"] == "free" - - -# --------------------------------------------------------------------------- -# Test 8: setup prompts for URL and key, validates, saves -# --------------------------------------------------------------------------- - -class TestCloudSetup: - @patch("neurostack.cli.cloud.CloudClient") - @patch("neurostack.cli.cloud.save_cloud_config") - @patch("neurostack.cli.cloud.load_cloud_config") - @patch("builtins.input", side_effect=["https://custom.api.dev", "sk-setup-key"]) - def test_setup_interactive(self, mock_input, mock_load, mock_save, mock_client_cls, capsys): - """Setup prompts for URL and key, validates, saves both.""" - mock_load.return_value = _mock_cloud_config() - mock_client = MagicMock() - mock_client.validate_key.return_value = True - mock_client_cls.return_value = mock_client - - from neurostack.cli.cloud import cmd_cloud - args = _make_args(cloud_command="setup") - cmd_cloud(args) - - out = capsys.readouterr().out - assert "configured" in out.lower() or "Authenticated" in out - mock_save.assert_called_once_with( - cloud_api_url="https://custom.api.dev", - cloud_api_key="sk-setup-key", - ) - - -# --------------------------------------------------------------------------- -# Test 9: no subcommand prints usage -# --------------------------------------------------------------------------- - -class TestCloudNoSubcommand: - def test_no_subcommand_prints_usage(self, capsys): - """cloud with no subcommand prints usage help.""" - from neurostack.cli.cloud import cmd_cloud - args = _make_args(cloud_command=None) - cmd_cloud(args) - - out = capsys.readouterr().out - assert "login" in out - assert "logout" in out - assert "status" in out - assert "setup" in out - - -# --------------------------------------------------------------------------- -# Tests 10-12: device code login flow -# --------------------------------------------------------------------------- - -class TestCloudDeviceLogin: - """Tests for the device code (browser-based) login flow.""" - - @patch("neurostack.cli.cloud.webbrowser", create=True) - @patch("neurostack.cli.cloud.time", create=True) - @patch("neurostack.cli.cloud.httpx", create=True) - @patch("neurostack.cli.cloud.save_cloud_config") - @patch("neurostack.cli.cloud.load_cloud_config") - def test_cloud_login_device_code_success( - self, mock_load, mock_save, mock_httpx, mock_time, mock_wb, capsys - ): - """Device code login: polls, gets 200 with api_key, saves config.""" - mock_load.return_value = _mock_cloud_config( - url="https://neurostack-api-911077737485.us-central1.run.app" - ) - - # Mock device-code response - device_resp = MagicMock() - device_resp.status_code = 200 - device_resp.json.return_value = { - "device_code": "dc-123", - "user_code": "ABCD-EFGH", - "verification_uri": "https://app.neurostack.sh/device", - "expires_in": 600, - "interval": 1, - } - device_resp.raise_for_status = MagicMock() - - # Mock token response: first pending (428), then success (200) - pending_resp = MagicMock() - pending_resp.status_code = 428 - - success_resp = MagicMock() - success_resp.status_code = 200 - success_resp.json.return_value = { - "api_key": "nsk-device-key-123", - "key_id": "kid-1", - "name": "CLI Device", - } - - mock_httpx.post.side_effect = [device_resp, pending_resp, success_resp] - mock_httpx.ConnectError = Exception - mock_httpx.TimeoutException = Exception - mock_httpx.HTTPStatusError = Exception - - # Mock time to avoid real sleeps - mock_time.monotonic.side_effect = [0, 0, 1, 2] # start, loop check, after sleep, loop check - mock_time.sleep = MagicMock() - - from neurostack.cli.cloud import _cmd_cloud_device_login - _cmd_cloud_device_login() - - out = capsys.readouterr().out - assert "Login successful" in out - mock_save.assert_called_once() - saved_kwargs = mock_save.call_args - assert saved_kwargs[1]["cloud_api_key"] == "nsk-device-key-123" - - @patch("neurostack.cli.cloud.webbrowser", create=True) - @patch("neurostack.cli.cloud.time", create=True) - @patch("neurostack.cli.cloud.httpx", create=True) - @patch("neurostack.cli.cloud.save_cloud_config") - @patch("neurostack.cli.cloud.load_cloud_config") - def test_cloud_login_device_code_expired( - self, mock_load, mock_save, mock_httpx, mock_time, mock_wb, capsys - ): - """Device code login: 400 expired code exits with error.""" - mock_load.return_value = _mock_cloud_config( - url="https://neurostack-api-911077737485.us-central1.run.app" - ) - - device_resp = MagicMock() - device_resp.status_code = 200 - device_resp.json.return_value = { - "device_code": "dc-456", - "user_code": "WXYZ-1234", - "verification_uri": "https://app.neurostack.sh/device", - "expires_in": 600, - "interval": 1, - } - device_resp.raise_for_status = MagicMock() - - expired_resp = MagicMock() - expired_resp.status_code = 400 - - mock_httpx.post.side_effect = [device_resp, expired_resp] - mock_httpx.ConnectError = Exception - mock_httpx.TimeoutException = Exception - mock_httpx.HTTPStatusError = Exception - - mock_time.monotonic.side_effect = [0, 0, 1] - mock_time.sleep = MagicMock() - - from neurostack.cli.cloud import _cmd_cloud_device_login - with pytest.raises(SystemExit, match="1"): - _cmd_cloud_device_login() - - out = capsys.readouterr().out - assert "expired" in out.lower() - mock_save.assert_not_called() - - @patch("neurostack.cli.cloud.webbrowser", create=True) - @patch("neurostack.cli.cloud.time", create=True) - @patch("neurostack.cli.cloud.httpx", create=True) - @patch("neurostack.cli.cloud.save_cloud_config") - @patch("neurostack.cli.cloud.load_cloud_config") - def test_cloud_login_device_code_timeout( - self, mock_load, mock_save, mock_httpx, mock_time, mock_wb, capsys - ): - """Device code login: timeout after expires_in seconds.""" - mock_load.return_value = _mock_cloud_config( - url="https://neurostack-api-911077737485.us-central1.run.app" - ) - - device_resp = MagicMock() - device_resp.status_code = 200 - device_resp.json.return_value = { - "device_code": "dc-789", - "user_code": "TIME-OUT1", - "verification_uri": "https://app.neurostack.sh/device", - "expires_in": 5, - "interval": 1, - } - device_resp.raise_for_status = MagicMock() - - pending_resp = MagicMock() - pending_resp.status_code = 428 - - mock_httpx.post.side_effect = [device_resp, pending_resp, pending_resp] - mock_httpx.ConnectError = Exception - mock_httpx.TimeoutException = Exception - mock_httpx.HTTPStatusError = Exception - - # Simulate time passing past deadline - # start, first check, after poll, past deadline - mock_time.monotonic.side_effect = [0, 0, 3, 999] - mock_time.sleep = MagicMock() - - from neurostack.cli.cloud import _cmd_cloud_device_login - with pytest.raises(SystemExit, match="1"): - _cmd_cloud_device_login() - - out = capsys.readouterr().out - assert "timed out" in out.lower() - mock_save.assert_not_called() diff --git a/tests/test_cloud_client.py b/tests/test_cloud_client.py deleted file mode 100644 index 88ad35c..0000000 --- a/tests/test_cloud_client.py +++ /dev/null @@ -1,398 +0,0 @@ -# SPDX-License-Identifier: Apache-2.0 -# Copyright (c) 2024-2026 Raphael Southall -"""Tests for cloud config persistence and CloudClient HTTP wrapper.""" - -from __future__ import annotations - -from unittest.mock import MagicMock, patch - -import pytest - -try: - import tomllib -except ImportError: - import tomli as tomllib - - -# --------------------------------------------------------------------------- -# Config persistence tests -# --------------------------------------------------------------------------- - - -class TestSaveCloudConfig: - """Tests for save_cloud_config() TOML persistence.""" - - def test_save_writes_cloud_section(self, tmp_path, monkeypatch): - """save_cloud_config(url, key) writes [cloud] section to config.toml.""" - config_path = tmp_path / "config.toml" - monkeypatch.setattr("neurostack.cloud.config._get_config_path", lambda: config_path) - - from neurostack.cloud.config import save_cloud_config - - save_cloud_config( - cloud_api_url="https://api.neurostack.sh", - cloud_api_key="ns_test_key_123", - ) - - assert config_path.exists() - with open(config_path, "rb") as f: - data = tomllib.load(f) - assert data["cloud"]["cloud_api_url"] == "https://api.neurostack.sh" - assert data["cloud"]["cloud_api_key"] == "ns_test_key_123" - - def test_save_preserves_existing_sections(self, tmp_path, monkeypatch): - """save_cloud_config() preserves existing non-cloud config sections.""" - config_path = tmp_path / "config.toml" - config_path.write_text('vault_root = "~/my-vault"\nembed_url = "http://gpu:11435"\n') - monkeypatch.setattr("neurostack.cloud.config._get_config_path", lambda: config_path) - - from neurostack.cloud.config import save_cloud_config - - save_cloud_config( - cloud_api_url="https://api.neurostack.sh", - cloud_api_key="ns_key", - ) - - with open(config_path, "rb") as f: - data = tomllib.load(f) - assert data["vault_root"] == "~/my-vault" - assert data["embed_url"] == "http://gpu:11435" - assert data["cloud"]["cloud_api_url"] == "https://api.neurostack.sh" - - def test_save_creates_directory(self, tmp_path, monkeypatch): - """save_cloud_config() creates ~/.config/neurostack/ directory if missing.""" - config_path = tmp_path / "subdir" / "nested" / "config.toml" - monkeypatch.setattr("neurostack.cloud.config._get_config_path", lambda: config_path) - - from neurostack.cloud.config import save_cloud_config - - save_cloud_config(cloud_api_url="https://api.neurostack.sh", cloud_api_key="key") - - assert config_path.exists() - - -class TestClearCloudCredentials: - """Tests for clear_cloud_credentials().""" - - def test_clear_removes_key_preserves_url(self, tmp_path, monkeypatch): - """clear_cloud_credentials() removes cloud_api_key but preserves cloud_api_url.""" - config_path = tmp_path / "config.toml" - config_path.write_text( - '[cloud]\ncloud_api_url = "https://api.neurostack.sh"\n' - 'cloud_api_key = "ns_secret"\n' - ) - monkeypatch.setattr("neurostack.cloud.config._get_config_path", lambda: config_path) - - from neurostack.cloud.config import clear_cloud_credentials - - clear_cloud_credentials() - - with open(config_path, "rb") as f: - data = tomllib.load(f) - assert data["cloud"]["cloud_api_url"] == "https://api.neurostack.sh" - assert data["cloud"]["cloud_api_key"] == "" - - -class TestLoadCloudConfigToml: - """Tests for load_cloud_config() TOML integration.""" - - def test_load_reads_from_toml(self, tmp_path, monkeypatch): - """load_cloud_config() reads cloud_api_url and cloud_api_key from config.toml [cloud].""" - config_path = tmp_path / "config.toml" - config_path.write_text( - '[cloud]\ncloud_api_url = "https://api.neurostack.sh"\n' - 'cloud_api_key = "ns_toml_key"\n' - ) - monkeypatch.setattr("neurostack.cloud.config._get_config_path", lambda: config_path) - # Clear env vars that would override - monkeypatch.delenv("NEUROSTACK_CLOUD_API_URL", raising=False) - monkeypatch.delenv("NEUROSTACK_CLOUD_API_KEY", raising=False) - - from neurostack.cloud.config import load_cloud_config - - cfg = load_cloud_config() - assert cfg.cloud_api_url == "https://api.neurostack.sh" - assert cfg.cloud_api_key == "ns_toml_key" - - def test_env_vars_override_toml(self, tmp_path, monkeypatch): - """Env vars NEUROSTACK_CLOUD_API_URL/KEY override TOML values.""" - config_path = tmp_path / "config.toml" - config_path.write_text( - '[cloud]\ncloud_api_url = "https://toml.example.com"\n' - 'cloud_api_key = "toml_key"\n' - ) - monkeypatch.setattr("neurostack.cloud.config._get_config_path", lambda: config_path) - monkeypatch.setenv("NEUROSTACK_CLOUD_API_URL", "https://env.example.com") - monkeypatch.setenv("NEUROSTACK_CLOUD_API_KEY", "env_key") - - from neurostack.cloud.config import load_cloud_config - - cfg = load_cloud_config() - assert cfg.cloud_api_url == "https://env.example.com" - assert cfg.cloud_api_key == "env_key" - - -# --------------------------------------------------------------------------- -# CloudClient tests -# --------------------------------------------------------------------------- - - -class TestCloudClientInit: - """Tests for CloudClient initialization.""" - - def test_init_stores_config_no_network(self): - """CloudClient.__init__ stores config and does not make network calls.""" - from neurostack.cloud.client import CloudClient - from neurostack.cloud.config import CloudConfig - - cfg = CloudConfig( - cloud_api_url="https://api.neurostack.sh", - cloud_api_key="ns_key_123", - ) - client = CloudClient(cfg) - assert client._config is cfg - assert client._base_url == "https://api.neurostack.sh" - - def test_init_strips_trailing_slash(self): - """CloudClient strips trailing slash from base URL.""" - from neurostack.cloud.client import CloudClient - from neurostack.cloud.config import CloudConfig - - cfg = CloudConfig(cloud_api_url="https://api.neurostack.sh/") - client = CloudClient(cfg) - assert client._base_url == "https://api.neurostack.sh" - - def test_is_configured_true(self): - """is_configured returns True when both url and key are set.""" - from neurostack.cloud.client import CloudClient - from neurostack.cloud.config import CloudConfig - - cfg = CloudConfig( - cloud_api_url="https://api.neurostack.sh", - cloud_api_key="ns_key", - ) - assert CloudClient(cfg).is_configured is True - - def test_is_configured_false_missing_key(self): - """is_configured returns False when api_key is empty.""" - from neurostack.cloud.client import CloudClient - from neurostack.cloud.config import CloudConfig - - cfg = CloudConfig(cloud_api_url="https://api.neurostack.sh", cloud_api_key="") - assert CloudClient(cfg).is_configured is False - - def test_is_configured_false_missing_url(self): - """is_configured returns False when cloud_api_url is empty.""" - from neurostack.cloud.client import CloudClient - from neurostack.cloud.config import CloudConfig - - cfg = CloudConfig(cloud_api_url="", cloud_api_key="ns_key") - assert CloudClient(cfg).is_configured is False - - -class TestCloudClientHealth: - """Tests for CloudClient.health() — no auth required.""" - - def test_health_returns_status(self): - """health() calls GET /health and returns server status dict.""" - from neurostack.cloud.client import CloudClient - from neurostack.cloud.config import CloudConfig - - cfg = CloudConfig(cloud_api_url="https://api.neurostack.sh") - client = CloudClient(cfg) - - mock_response = MagicMock() - mock_response.status_code = 200 - mock_response.json.return_value = {"status": "ok", "version": "0.8.0"} - mock_response.raise_for_status = MagicMock() - - with patch("neurostack.cloud.client.httpx") as mock_httpx: - mock_httpx.get.return_value = mock_response - result = client.health() - - assert result == {"status": "ok", "version": "0.8.0"} - mock_httpx.get.assert_called_once_with( - "https://api.neurostack.sh/health", - timeout=10.0, - ) - - def test_health_no_auth_header(self): - """health() does not send Authorization header.""" - from neurostack.cloud.client import CloudClient - from neurostack.cloud.config import CloudConfig - - cfg = CloudConfig( - cloud_api_url="https://api.neurostack.sh", - cloud_api_key="ns_secret", - ) - client = CloudClient(cfg) - - mock_response = MagicMock() - mock_response.status_code = 200 - mock_response.json.return_value = {"status": "ok"} - mock_response.raise_for_status = MagicMock() - - with patch("neurostack.cloud.client.httpx") as mock_httpx: - mock_httpx.get.return_value = mock_response - client.health() - - # Verify no headers kwarg (no auth) - call_kwargs = mock_httpx.get.call_args - assert "headers" not in call_kwargs.kwargs - - def test_health_connection_error(self): - """health() raises ConnectionError when server unreachable.""" - import httpx as real_httpx - - from neurostack.cloud.client import CloudClient - from neurostack.cloud.config import CloudConfig - - cfg = CloudConfig(cloud_api_url="https://api.neurostack.sh") - client = CloudClient(cfg) - - with patch("neurostack.cloud.client.httpx") as mock_httpx: - mock_httpx.ConnectError = real_httpx.ConnectError - mock_httpx.TimeoutException = real_httpx.TimeoutException - mock_httpx.get.side_effect = real_httpx.ConnectError("Connection refused") - with pytest.raises(ConnectionError, match="Cannot reach cloud API"): - client.health() - - -class TestCloudClientValidateKey: - """Tests for CloudClient.validate_key().""" - - def test_validate_key_returns_true_on_200(self): - """validate_key() returns True when server responds 200.""" - from neurostack.cloud.client import CloudClient - from neurostack.cloud.config import CloudConfig - - cfg = CloudConfig( - cloud_api_url="https://api.neurostack.sh", - cloud_api_key="ns_valid_key", - ) - client = CloudClient(cfg) - - mock_response = MagicMock() - mock_response.status_code = 200 - mock_response.json.return_value = {"status": "ok"} - - with patch("neurostack.cloud.client.httpx") as mock_httpx: - mock_httpx.get.return_value = mock_response - result = client.validate_key() - - assert result is True - call_kwargs = mock_httpx.get.call_args - assert call_kwargs.kwargs["headers"]["Authorization"] == "Bearer ns_valid_key" - - def test_validate_key_returns_false_on_401(self): - """validate_key() returns False on 401 Unauthorized.""" - from neurostack.cloud.client import CloudClient - from neurostack.cloud.config import CloudConfig - - cfg = CloudConfig( - cloud_api_url="https://api.neurostack.sh", - cloud_api_key="ns_bad_key", - ) - client = CloudClient(cfg) - - mock_response = MagicMock() - mock_response.status_code = 401 - - with patch("neurostack.cloud.client.httpx") as mock_httpx: - mock_httpx.get.return_value = mock_response - result = client.validate_key() - - assert result is False - - def test_validate_key_connection_error(self): - """validate_key() raises ConnectionError when server unreachable.""" - import httpx as real_httpx - - from neurostack.cloud.client import CloudClient - from neurostack.cloud.config import CloudConfig - - cfg = CloudConfig( - cloud_api_url="https://api.neurostack.sh", - cloud_api_key="ns_key", - ) - client = CloudClient(cfg) - - with patch("neurostack.cloud.client.httpx") as mock_httpx: - mock_httpx.ConnectError = real_httpx.ConnectError - mock_httpx.TimeoutException = real_httpx.TimeoutException - mock_httpx.get.side_effect = real_httpx.ConnectError("Connection refused") - with pytest.raises(ConnectionError): - client.validate_key() - - -class TestCloudClientStatus: - """Tests for CloudClient.status().""" - - def test_status_authenticated(self): - """status() returns auth state and tier when key is valid.""" - from neurostack.cloud.client import CloudClient - from neurostack.cloud.config import CloudConfig - - cfg = CloudConfig( - cloud_api_url="https://api.neurostack.sh", - cloud_api_key="ns_valid_key", - ) - client = CloudClient(cfg) - - mock_response = MagicMock() - mock_response.status_code = 200 - mock_response.json.return_value = {"status": "ok", "version": "0.8.0"} - - with patch("neurostack.cloud.client.httpx") as mock_httpx: - mock_httpx.get.return_value = mock_response - result = client.status() - - assert result["authenticated"] is True - assert result["cloud_url"] == "https://api.neurostack.sh" - - def test_status_unauthenticated(self): - """status() returns authenticated=False when key is invalid.""" - from neurostack.cloud.client import CloudClient - from neurostack.cloud.config import CloudConfig - - cfg = CloudConfig( - cloud_api_url="https://api.neurostack.sh", - cloud_api_key="ns_bad_key", - ) - client = CloudClient(cfg) - - mock_response = MagicMock() - mock_response.status_code = 401 - - with patch("neurostack.cloud.client.httpx") as mock_httpx: - mock_httpx.get.return_value = mock_response - result = client.status() - - assert result["authenticated"] is False - assert result["tier"] is None - - -class TestCloudClientAuthHeaders: - """Tests for Bearer auth header construction.""" - - def test_auth_headers_set_on_authenticated_requests(self): - """CloudClient sets Authorization: Bearer {api_key} on authenticated requests.""" - from neurostack.cloud.client import CloudClient - from neurostack.cloud.config import CloudConfig - - cfg = CloudConfig( - cloud_api_url="https://api.neurostack.sh", - cloud_api_key="ns_my_secret_key", - ) - client = CloudClient(cfg) - headers = client._auth_headers() - assert headers == {"Authorization": "Bearer ns_my_secret_key"} - - def test_auth_headers_empty_when_no_key(self): - """_auth_headers() returns empty dict when no API key configured.""" - from neurostack.cloud.client import CloudClient - from neurostack.cloud.config import CloudConfig - - cfg = CloudConfig(cloud_api_url="https://api.neurostack.sh", cloud_api_key="") - client = CloudClient(cfg) - assert client._auth_headers() == {} diff --git a/tests/test_cloud_consent.py b/tests/test_cloud_consent.py deleted file mode 100644 index fd31d65..0000000 --- a/tests/test_cloud_consent.py +++ /dev/null @@ -1,201 +0,0 @@ -# SPDX-License-Identifier: Apache-2.0 -# Copyright (c) 2024-2026 Raphael Southall -"""Tests for cloud privacy consent and .neurostackignore.""" - -from __future__ import annotations - -from unittest.mock import patch - -import pytest - - -class TestSaveConsent: - """Tests for save_consent() writing to config.toml.""" - - def test_save_consent_writes_to_config(self, tmp_path): - """save_consent() writes consent_given=true and consent_date to config.""" - config_path = tmp_path / "config.toml" - - with patch("neurostack.cloud.config._get_config_path", return_value=config_path): - from neurostack.cloud.config import save_consent - - save_consent() - - # Read back the file and verify - import tomllib - - with open(config_path, "rb") as f: - data = tomllib.load(f) - - assert data["cloud"]["consent_given"] is True - assert "consent_date" in data["cloud"] - assert len(data["cloud"]["consent_date"]) > 0 - - def test_load_consent_from_config(self, tmp_path): - """load_cloud_config() reads consent fields from config.toml.""" - config_path = tmp_path / "config.toml" - - import tomli_w - - data = { - "cloud": { - "cloud_api_url": "https://example.com", - "cloud_api_key": "test-key", - "consent_given": True, - "consent_date": "2026-03-26T12:00:00+00:00", - } - } - with open(config_path, "wb") as f: - tomli_w.dump(data, f) - - with patch("neurostack.cloud.config._get_config_path", return_value=config_path): - from neurostack.cloud.config import load_cloud_config - - cfg = load_cloud_config() - - assert cfg.consent_given is True - assert cfg.consent_date == "2026-03-26T12:00:00+00:00" - - -class TestConsentCheck: - """Tests for consent enforcement in push/sync.""" - - def test_push_raises_consent_error_when_not_given(self, tmp_path): - """push() raises ConsentError when consent_given=False.""" - from neurostack.cloud.sync import ConsentError, VaultSyncEngine - - vault = tmp_path / "vault" - vault.mkdir() - (vault / "note.md").write_text("hello") - db_dir = tmp_path / "db" - db_dir.mkdir() - - engine = VaultSyncEngine( - cloud_api_url="https://example.com", - cloud_api_key="test-key", - vault_root=vault, - db_dir=db_dir, - consent_given=False, - ) - - with pytest.raises(ConsentError, match="Cloud consent not given"): - engine.push() - - def test_push_succeeds_with_consent(self, tmp_path): - """push() proceeds normally when consent_given=True (no ConsentError).""" - from unittest.mock import MagicMock - - from neurostack.cloud.sync import VaultSyncEngine - - vault = tmp_path / "vault" - vault.mkdir() - (vault / "note.md").write_text("hello world") - db_dir = tmp_path / "db" - db_dir.mkdir() - - engine = VaultSyncEngine( - cloud_api_url="https://example.com", - cloud_api_key="test-key", - vault_root=vault, - db_dir=db_dir, - consent_given=True, - ) - - # Mock httpx.Client to avoid real HTTP calls - mock_response = MagicMock() - mock_response.json.return_value = {"job_id": "test-job-123"} - mock_response.raise_for_status = MagicMock() - - mock_status_response = MagicMock() - mock_status_response.json.return_value = { - "status": "complete", - "message": "Indexing complete", - } - mock_status_response.raise_for_status = MagicMock() - - mock_client = MagicMock() - mock_client.post.return_value = mock_response - mock_client.get.return_value = mock_status_response - mock_client.__enter__ = MagicMock(return_value=mock_client) - mock_client.__exit__ = MagicMock(return_value=False) - - with patch("httpx.Client", return_value=mock_client): - result = engine.push() - - assert result["status"] == "complete" - - def test_sync_raises_consent_error_when_not_given(self, tmp_path): - """sync() raises ConsentError when consent_given=False.""" - from neurostack.cloud.sync import ConsentError, VaultSyncEngine - - vault = tmp_path / "vault" - vault.mkdir() - (vault / "note.md").write_text("hello") - db_dir = tmp_path / "db" - db_dir.mkdir() - - engine = VaultSyncEngine( - cloud_api_url="https://example.com", - cloud_api_key="test-key", - vault_root=vault, - db_dir=db_dir, - consent_given=False, - ) - - with pytest.raises(ConsentError, match="Cloud consent not given"): - engine.sync() - - -class TestNeurostackIgnore: - """Tests for .neurostackignore loading.""" - - def test_neurostackignore_load_patterns(self, tmp_path): - """_load_ignore_patterns() reads patterns from .neurostackignore file.""" - from neurostack.cloud.sync import VaultSyncEngine - - vault = tmp_path / "vault" - vault.mkdir() - db_dir = tmp_path / "db" - db_dir.mkdir() - - # Write a .neurostackignore file with patterns, comments, and blank lines - ignore_file = vault / ".neurostackignore" - ignore_file.write_text( - "# Ignore private notes\n" - "private/\n" - "\n" - "*.draft.md\n" - "# Another comment\n" - "journal/personal/*\n" - ) - - engine = VaultSyncEngine( - cloud_api_url="https://example.com", - cloud_api_key="test-key", - vault_root=vault, - db_dir=db_dir, - ) - - patterns = engine._load_ignore_patterns() - - assert patterns == ["private/", "*.draft.md", "journal/personal/*"] - assert engine._check_neurostackignore_exists() is True - - def test_neurostackignore_returns_empty_when_missing(self, tmp_path): - """_load_ignore_patterns() returns empty list when file doesn't exist.""" - from neurostack.cloud.sync import VaultSyncEngine - - vault = tmp_path / "vault" - vault.mkdir() - db_dir = tmp_path / "db" - db_dir.mkdir() - - engine = VaultSyncEngine( - cloud_api_url="https://example.com", - cloud_api_key="test-key", - vault_root=vault, - db_dir=db_dir, - ) - - assert engine._load_ignore_patterns() == [] - assert engine._check_neurostackignore_exists() is False diff --git a/tests/test_cloud_hooks.py b/tests/test_cloud_hooks.py deleted file mode 100644 index 5295b02..0000000 --- a/tests/test_cloud_hooks.py +++ /dev/null @@ -1,107 +0,0 @@ -"""Tests for git hooks installer (neurostack cloud install-hooks).""" -from __future__ import annotations - -import stat - -import pytest - -from neurostack.cloud.hooks import ( - HOOK_MARKER, - hooks_status, - install_hooks, - uninstall_hooks, -) - - -@pytest.fixture -def git_vault(tmp_path): - """Create a tmp_path that looks like a git repo.""" - (tmp_path / ".git").mkdir() - return tmp_path - - -def test_install_hooks_creates_post_commit(git_vault): - result = install_hooks(git_vault) - hook = git_vault / ".git" / "hooks" / "post-commit" - assert hook.exists() - assert HOOK_MARKER in hook.read_text() - assert "post-commit" in result["installed"] - - -def test_install_hooks_creates_post_merge(git_vault): - result = install_hooks(git_vault) - hook = git_vault / ".git" / "hooks" / "post-merge" - assert hook.exists() - assert HOOK_MARKER in hook.read_text() - assert "post-merge" in result["installed"] - - -def test_hooks_are_executable(git_vault): - install_hooks(git_vault) - for name in ("post-commit", "post-merge"): - hook = git_vault / ".git" / "hooks" / name - mode = hook.stat().st_mode - assert mode & stat.S_IXUSR, f"{name} should be user-executable" - assert mode & stat.S_IXGRP, f"{name} should be group-executable" - assert mode & stat.S_IXOTH, f"{name} should be other-executable" - - -def test_install_hooks_skips_existing(git_vault): - first = install_hooks(git_vault) - assert len(first["installed"]) == 2 - assert len(first["skipped"]) == 0 - - second = install_hooks(git_vault) - assert len(second["installed"]) == 0 - assert set(second["skipped"]) == {"post-commit", "post-merge"} - - -def test_install_hooks_not_git_repo(tmp_path): - with pytest.raises(ValueError, match="Not a git repository"): - install_hooks(tmp_path) - - -def test_uninstall_hooks_removes(git_vault): - install_hooks(git_vault) - result = uninstall_hooks(git_vault) - assert set(result["removed"]) == {"post-commit", "post-merge"} - assert not (git_vault / ".git" / "hooks" / "post-commit").exists() - assert not (git_vault / ".git" / "hooks" / "post-merge").exists() - - -def test_hooks_status_reports_correctly(git_vault): - # Before install - status = hooks_status(git_vault) - assert status["git_repo"] is True - assert status["post_commit"] is False - assert status["post_merge"] is False - - # After install - install_hooks(git_vault) - status = hooks_status(git_vault) - assert status["post_commit"] is True - assert status["post_merge"] is True - - # Not a git repo - from pathlib import Path - status = hooks_status(Path("/tmp/not-a-repo-12345")) - assert status["git_repo"] is False - - -def test_hooks_append_to_existing(git_vault): - hooks_dir = git_vault / ".git" / "hooks" - hooks_dir.mkdir(exist_ok=True) - - # Write a pre-existing post-commit hook without the marker - existing_content = "#!/bin/sh\n# my custom hook\necho 'custom action'\n" - (hooks_dir / "post-commit").write_text(existing_content) - - install_hooks(git_vault) - - final = (hooks_dir / "post-commit").read_text() - # Original content preserved - assert "my custom hook" in final - assert "echo 'custom action'" in final - # Neurostack content appended - assert HOOK_MARKER in final - assert "neurostack cloud sync" in final diff --git a/tests/test_cloud_ignore.py b/tests/test_cloud_ignore.py deleted file mode 100644 index 260a52f..0000000 --- a/tests/test_cloud_ignore.py +++ /dev/null @@ -1,113 +0,0 @@ -# SPDX-License-Identifier: Apache-2.0 -# Copyright (c) 2024-2026 Raphael Southall -"""Tests for .neurostackignore support in manifest scanning and sync.""" - -from __future__ import annotations - -from unittest.mock import MagicMock, patch - -from neurostack.cloud.manifest import Manifest - - -class TestNeurostackIgnore: - """Tests for .neurostackignore pattern exclusion in Manifest.scan_vault().""" - - def test_scan_excludes_ignored_file(self, tmp_path): - """Files listed in .neurostackignore are excluded from the manifest.""" - (tmp_path / "normal.md").write_text("visible note") - (tmp_path / "secret.md").write_text("hidden note") - (tmp_path / ".neurostackignore").write_text("secret.md\n") - - ignore_file = tmp_path / ".neurostackignore" - manifest = Manifest.scan_vault(tmp_path, ignore_file=ignore_file) - - assert "normal.md" in manifest.entries - assert "secret.md" not in manifest.entries - assert len(manifest.entries) == 1 - - def test_scan_excludes_glob_pattern(self, tmp_path): - """Glob patterns like 'private/*.md' exclude matching subdirectory files.""" - (tmp_path / "note.md").write_text("public note") - (tmp_path / "private").mkdir() - (tmp_path / "private" / "diary.md").write_text("private diary") - (tmp_path / "private" / "journal.md").write_text("private journal") - (tmp_path / ".neurostackignore").write_text("private/*.md\n") - - ignore_file = tmp_path / ".neurostackignore" - manifest = Manifest.scan_vault(tmp_path, ignore_file=ignore_file) - - assert "note.md" in manifest.entries - assert "private/diary.md" not in manifest.entries - assert "private/journal.md" not in manifest.entries - assert len(manifest.entries) == 1 - - def test_scan_ignores_comments_and_blank_lines(self, tmp_path): - """Comments (#) and blank lines in .neurostackignore are ignored.""" - (tmp_path / "keep.md").write_text("keep this") - (tmp_path / "drop.md").write_text("drop this") - ignore_content = "# This is a comment\n\n # Indented comment\n\ndrop.md\n\n" - (tmp_path / ".neurostackignore").write_text(ignore_content) - - ignore_file = tmp_path / ".neurostackignore" - manifest = Manifest.scan_vault(tmp_path, ignore_file=ignore_file) - - assert "keep.md" in manifest.entries - assert "drop.md" not in manifest.entries - assert len(manifest.entries) == 1 - - def test_scan_without_ignore_file_includes_all(self, tmp_path): - """Without an ignore_file, all .md files are included (backwards compatible).""" - (tmp_path / "a.md").write_text("alpha") - (tmp_path / "b.md").write_text("beta") - - manifest = Manifest.scan_vault(tmp_path) - - assert "a.md" in manifest.entries - assert "b.md" in manifest.entries - assert len(manifest.entries) == 2 - - def test_push_uses_ignore_file(self, tmp_path): - """VaultSyncEngine.push() skips files matching .neurostackignore patterns.""" - from neurostack.cloud.sync import VaultSyncEngine - - # Set up vault with ignore file - (tmp_path / "public.md").write_text("public content") - (tmp_path / "ignored.md").write_text("ignored content") - (tmp_path / ".neurostackignore").write_text("ignored.md\n") - - manifest_path = tmp_path / ".neurostack" / "cloud-manifest.json" - - engine = VaultSyncEngine( - cloud_api_url="https://fake.api", - cloud_api_key="fake-key", - vault_root=tmp_path, - db_dir=tmp_path, - manifest_path=manifest_path, - consent_given=True, - ) - - # Mock the HTTP calls so push doesn't actually contact a server - mock_response = MagicMock() - mock_response.json.return_value = {"job_id": "test-job"} - mock_response.raise_for_status = MagicMock() - - mock_poll_result = {"status": "complete"} - - with ( - patch.object(engine, "_build_tar_archive", return_value=b"fake") as mock_tar, - patch.object(engine, "_poll_job", return_value=mock_poll_result), - patch("httpx.Client") as mock_client_cls, - ): - mock_client = MagicMock() - mock_client.__enter__ = MagicMock(return_value=mock_client) - mock_client.__exit__ = MagicMock(return_value=False) - mock_client.post.return_value = mock_response - mock_client_cls.return_value = mock_client - - engine.push() - - # Verify _build_tar_archive was called with only "public.md" - call_args = mock_tar.call_args - upload_files = call_args[0][0] - assert "public.md" in upload_files - assert "ignored.md" not in upload_files diff --git a/tests/test_cloud_merge_memories.py b/tests/test_cloud_merge_memories.py deleted file mode 100644 index 4b7f223..0000000 --- a/tests/test_cloud_merge_memories.py +++ /dev/null @@ -1,394 +0,0 @@ -# SPDX-License-Identifier: Apache-2.0 -# Copyright (c) 2024-2026 Raphael Southall -"""Tests for cloud sync _merge_memories ON CONFLICT logic. - -Validates that: -1. New memories are inserted correctly -2. Existing memories are updated via ON CONFLICT(uuid) DO UPDATE -3. Existing fields not in cloud response (embedding, revision_count) are preserved -4. Deleted memories are removed -5. Uses get_db() instead of raw sqlite3.connect() -6. Column names match the actual schema (expires_at, not ttl_hours) -""" - -from __future__ import annotations - -import json -import sqlite3 -from pathlib import Path - -import pytest - - -@pytest.fixture -def sync_engine(tmp_path): - """Create a VaultSyncEngine with a real SQLite DB.""" - from neurostack.cloud.sync import VaultSyncEngine - from neurostack.schema import SCHEMA_SQL, SCHEMA_VERSION - - db_dir = tmp_path / "data" - db_dir.mkdir() - db_path = db_dir / "neurostack.db" - - # Create real schema - conn = sqlite3.connect(str(db_path)) - conn.execute("PRAGMA foreign_keys=ON") - conn.execute("PRAGMA journal_mode=WAL") - conn.executescript(SCHEMA_SQL) - conn.execute( - "INSERT OR REPLACE INTO schema_version VALUES (?)", (SCHEMA_VERSION,) - ) - conn.commit() - conn.close() - - engine = VaultSyncEngine.__new__(VaultSyncEngine) - engine._db_dir = db_dir - engine._vault_root = tmp_path / "vault" - engine._vault_root.mkdir() - return engine - - -def _read_memory(db_path: Path, uuid: str) -> dict | None: - """Read a memory by UUID from the DB.""" - conn = sqlite3.connect(str(db_path)) - conn.row_factory = sqlite3.Row - row = conn.execute( - "SELECT * FROM memories WHERE uuid = ?", (uuid,) - ).fetchone() - conn.close() - if row is None: - return None - return dict(row) - - -def _insert_memory(db_path: Path, **kwargs): - """Insert a memory directly into the DB.""" - defaults = { - "uuid": "test-uuid-1", - "content": "original content", - "entity_type": "observation", - "tags": json.dumps(["tag1"]), - "workspace": None, - "source_agent": "test-agent", - "session_id": None, - "expires_at": None, - "created_at": "2026-03-25T00:00:00", - "updated_at": "2026-03-25T00:00:00", - "file_path": None, - } - defaults.update(kwargs) - - conn = sqlite3.connect(str(db_path)) - conn.execute("PRAGMA journal_mode=WAL") - cols = ", ".join(defaults.keys()) - placeholders = ", ".join("?" for _ in defaults) - conn.execute( - f"INSERT INTO memories ({cols}) VALUES ({placeholders})", - tuple(defaults.values()), - ) - conn.commit() - conn.close() - - -# --------------------------------------------------------------------------- -# Insert tests -# --------------------------------------------------------------------------- - - -class TestMergeMemoriesInsert: - """Tests for inserting new memories via _merge_memories.""" - - def test_inserts_new_memory(self, sync_engine): - """A memory with a new UUID is inserted.""" - memories = [ - { - "uuid": "new-uuid-1", - "content": "a new memory", - "entity_type": "decision", - "tags": ["tag-a", "tag-b"], - "workspace": "work/project", - "source_agent": "claude-code", - "session_id": None, - "expires_at": None, - "created_at": "2026-03-27T10:00:00", - "updated_at": "2026-03-27T10:00:00", - "file_path": None, - } - ] - - sync_engine._merge_memories(memories) - - row = _read_memory(sync_engine._db_dir / "neurostack.db", "new-uuid-1") - assert row is not None - assert row["content"] == "a new memory" - assert row["entity_type"] == "decision" - assert json.loads(row["tags"]) == ["tag-a", "tag-b"] - assert row["workspace"] == "work/project" - - def test_inserts_memory_with_expires_at(self, sync_engine): - """expires_at field is correctly stored (not ttl_hours).""" - memories = [ - { - "uuid": "expiring-uuid", - "content": "temporary memory", - "entity_type": "context", - "tags": [], - "workspace": None, - "source_agent": "test", - "session_id": None, - "expires_at": "2026-04-01T00:00:00", - "created_at": "2026-03-27T10:00:00", - "updated_at": "2026-03-27T10:00:00", - "file_path": None, - } - ] - - sync_engine._merge_memories(memories) - - row = _read_memory(sync_engine._db_dir / "neurostack.db", "expiring-uuid") - assert row is not None - assert row["expires_at"] == "2026-04-01T00:00:00" - - -# --------------------------------------------------------------------------- -# Update (ON CONFLICT) tests -# --------------------------------------------------------------------------- - - -class TestMergeMemoriesUpdate: - """Tests for updating existing memories via ON CONFLICT.""" - - def test_updates_content_on_conflict(self, sync_engine): - """Existing memory content is updated when UUID matches.""" - db_path = sync_engine._db_dir / "neurostack.db" - _insert_memory(db_path, uuid="existing-uuid", content="old content") - - memories = [ - { - "uuid": "existing-uuid", - "content": "updated content", - "entity_type": "decision", - "tags": ["new-tag"], - "workspace": None, - "source_agent": "claude-code", - "session_id": None, - "expires_at": None, - "created_at": "2026-03-25T00:00:00", - "updated_at": "2026-03-27T12:00:00", - "file_path": None, - } - ] - - sync_engine._merge_memories(memories) - - row = _read_memory(db_path, "existing-uuid") - assert row["content"] == "updated content" - assert row["entity_type"] == "decision" - assert row["updated_at"] == "2026-03-27T12:00:00" - - def test_preserves_embedding_on_update(self, sync_engine): - """Existing embedding blob is NOT overwritten by ON CONFLICT update.""" - db_path = sync_engine._db_dir / "neurostack.db" - - # Insert with a fake embedding - fake_embedding = b"\x00\x01\x02\x03" * 192 # 768 bytes - _insert_memory(db_path, uuid="embed-uuid", content="has embedding") - - # Manually set the embedding - conn = sqlite3.connect(str(db_path)) - conn.execute( - "UPDATE memories SET embedding = ? WHERE uuid = ?", - (fake_embedding, "embed-uuid"), - ) - conn.commit() - conn.close() - - # Merge update (cloud response has no embedding field) - memories = [ - { - "uuid": "embed-uuid", - "content": "updated content", - "entity_type": "observation", - "tags": [], - "workspace": None, - "source_agent": "cloud", - "session_id": None, - "expires_at": None, - "created_at": "2026-03-25T00:00:00", - "updated_at": "2026-03-27T12:00:00", - "file_path": None, - } - ] - - sync_engine._merge_memories(memories) - - row = _read_memory(db_path, "embed-uuid") - assert row["content"] == "updated content" - assert row["embedding"] == fake_embedding, ( - "Embedding was overwritten by ON CONFLICT — should be preserved" - ) - - def test_preserves_revision_count_on_update(self, sync_engine): - """revision_count is not reset by ON CONFLICT update.""" - db_path = sync_engine._db_dir / "neurostack.db" - _insert_memory(db_path, uuid="rev-uuid") - - # Set revision_count manually - conn = sqlite3.connect(str(db_path)) - conn.execute( - "UPDATE memories SET revision_count = 5 WHERE uuid = ?", - ("rev-uuid",), - ) - conn.commit() - conn.close() - - memories = [ - { - "uuid": "rev-uuid", - "content": "revised", - "entity_type": "observation", - "tags": [], - "workspace": None, - "source_agent": "test", - "session_id": None, - "expires_at": None, - "created_at": "2026-03-25T00:00:00", - "updated_at": "2026-03-27T12:00:00", - "file_path": None, - } - ] - - sync_engine._merge_memories(memories) - - row = _read_memory(db_path, "rev-uuid") - assert row["revision_count"] == 5, ( - "revision_count was reset — ON CONFLICT should preserve it" - ) - - -# --------------------------------------------------------------------------- -# Delete tests -# --------------------------------------------------------------------------- - - -class TestMergeMemoriesDelete: - """Tests for deleting memories marked as deleted.""" - - def test_deletes_memory_with_deleted_flag(self, sync_engine): - """Memories with deleted=true are removed from local DB.""" - db_path = sync_engine._db_dir / "neurostack.db" - _insert_memory(db_path, uuid="to-delete") - - # Verify it exists - assert _read_memory(db_path, "to-delete") is not None - - memories = [{"uuid": "to-delete", "deleted": True}] - sync_engine._merge_memories(memories) - - assert _read_memory(db_path, "to-delete") is None - - def test_delete_nonexistent_is_safe(self, sync_engine): - """Deleting a UUID that doesn't exist locally is a no-op.""" - memories = [{"uuid": "never-existed", "deleted": True}] - sync_engine._merge_memories(memories) - # Should not raise - - -# --------------------------------------------------------------------------- -# Edge cases -# --------------------------------------------------------------------------- - - -class TestMergeMemoriesEdgeCases: - """Edge case tests for _merge_memories.""" - - def test_empty_list_is_noop(self, sync_engine): - """Empty memory list does nothing.""" - sync_engine._merge_memories([]) - - def test_tags_as_list_serialised_to_json(self, sync_engine): - """Tags provided as list are JSON-serialised before insert.""" - memories = [ - { - "uuid": "tags-uuid", - "content": "test", - "entity_type": "observation", - "tags": ["alpha", "beta"], - "workspace": None, - "source_agent": "test", - "session_id": None, - "expires_at": None, - "created_at": "2026-03-27T10:00:00", - "updated_at": "2026-03-27T10:00:00", - "file_path": None, - } - ] - - sync_engine._merge_memories(memories) - - row = _read_memory(sync_engine._db_dir / "neurostack.db", "tags-uuid") - parsed = json.loads(row["tags"]) - assert parsed == ["alpha", "beta"] - - def test_tags_as_string_stored_directly(self, sync_engine): - """Tags already serialised as JSON string are stored as-is.""" - memories = [ - { - "uuid": "str-tags-uuid", - "content": "test", - "entity_type": "observation", - "tags": '["gamma"]', - "workspace": None, - "source_agent": "test", - "session_id": None, - "expires_at": None, - "created_at": "2026-03-27T10:00:00", - "updated_at": "2026-03-27T10:00:00", - "file_path": None, - } - ] - - sync_engine._merge_memories(memories) - - row = _read_memory(sync_engine._db_dir / "neurostack.db", "str-tags-uuid") - parsed = json.loads(row["tags"]) - assert parsed == ["gamma"] - - def test_missing_db_skips_gracefully(self, tmp_path): - """Missing DB file logs warning and returns without error.""" - from neurostack.cloud.sync import VaultSyncEngine - - engine = VaultSyncEngine.__new__(VaultSyncEngine) - engine._db_dir = tmp_path / "nonexistent" - engine._db_dir.mkdir() - # No neurostack.db created - - # Should not raise - engine._merge_memories([{"uuid": "x", "content": "y"}]) - - def test_multiple_memories_in_single_call(self, sync_engine): - """Multiple memories are processed in a single commit.""" - memories = [ - { - "uuid": f"batch-{i}", - "content": f"memory {i}", - "entity_type": "observation", - "tags": [], - "workspace": None, - "source_agent": "test", - "session_id": None, - "expires_at": None, - "created_at": "2026-03-27T10:00:00", - "updated_at": "2026-03-27T10:00:00", - "file_path": None, - } - for i in range(5) - ] - - sync_engine._merge_memories(memories) - - db_path = sync_engine._db_dir / "neurostack.db" - for i in range(5): - row = _read_memory(db_path, f"batch-{i}") - assert row is not None - assert row["content"] == f"memory {i}" diff --git a/tests/test_cloud_progress.py b/tests/test_cloud_progress.py deleted file mode 100644 index 1e977ba..0000000 --- a/tests/test_cloud_progress.py +++ /dev/null @@ -1,146 +0,0 @@ -# SPDX-License-Identifier: Apache-2.0 -# Copyright (c) 2024-2026 Raphael Southall -"""Tests for upload progress reporting in push() and sync().""" - -from __future__ import annotations - -from unittest.mock import MagicMock, patch - -import pytest - - -@pytest.fixture() -def vault_env(tmp_path): - """Set up vault dir, db dir, and sample .md files for progress tests.""" - vault_root = tmp_path / "vault" - vault_root.mkdir() - db_dir = tmp_path / "db" - db_dir.mkdir() - manifest_path = tmp_path / "manifest" / "cloud-manifest.json" - - # Create sample vault files with known content - (vault_root / "note1.md").write_text("first note content") - (vault_root / "sub").mkdir() - (vault_root / "sub" / "note2.md").write_text("second note content") - - return { - "vault_root": vault_root, - "db_dir": db_dir, - "manifest_path": manifest_path, - "api_url": "https://api.neurostack.sh", - "api_key": "ns_test_key_123", - } - - -def _make_engine(vault_env, **overrides): - """Create a VaultSyncEngine from vault_env fixture.""" - from neurostack.cloud.sync import VaultSyncEngine - - kwargs = { - "cloud_api_url": vault_env["api_url"], - "cloud_api_key": vault_env["api_key"], - "vault_root": vault_env["vault_root"], - "db_dir": vault_env["db_dir"], - "manifest_path": vault_env["manifest_path"], - "poll_interval": 0.01, - "poll_timeout": 1.0, - } - kwargs.update(overrides) - return VaultSyncEngine(**kwargs) - - -def _mock_http_client(): - """Build a mock httpx.Client that accepts upload and returns complete.""" - mock_upload_resp = MagicMock() - mock_upload_resp.status_code = 202 - mock_upload_resp.json.return_value = { - "job_id": "job-progress", - "status": "queued", - "message": "Upload received", - } - mock_upload_resp.raise_for_status = MagicMock() - - mock_status_resp = MagicMock() - mock_status_resp.status_code = 200 - mock_status_resp.json.return_value = { - "job_id": "job-progress", - "status": "complete", - "progress": 1.0, - } - mock_status_resp.raise_for_status = MagicMock() - - mock_client = MagicMock() - mock_client.post.return_value = mock_upload_resp - mock_client.get.return_value = mock_status_resp - mock_client.__enter__ = MagicMock(return_value=mock_client) - mock_client.__exit__ = MagicMock(return_value=False) - - return mock_client - - -class TestPushProgressReporting: - """Tests for progress reporting in push().""" - - def test_push_reports_file_count(self, vault_env): - """progress_callback receives message with file count.""" - engine = _make_engine(vault_env) - mock_client = _mock_http_client() - messages: list[str] = [] - - with patch("neurostack.cloud.sync.httpx.Client", return_value=mock_client): - engine.push(progress_callback=messages.append) - - # Find the uploading message that includes file count - upload_msgs = [m for m in messages if "Uploading 2 files" in m] - assert len(upload_msgs) >= 1, f"Expected 'Uploading 2 files' in messages: {messages}" - - def test_push_reports_compression_ratio(self, vault_env): - """progress_callback receives message with compression percentage.""" - engine = _make_engine(vault_env) - mock_client = _mock_http_client() - messages: list[str] = [] - - with patch("neurostack.cloud.sync.httpx.Client", return_value=mock_client): - engine.push(progress_callback=messages.append) - - # Find messages with compression info - compression_msgs = [m for m in messages if "compression" in m.lower()] - assert len(compression_msgs) >= 1, f"Expected compression info in messages: {messages}" - # Should contain a percentage - msg = compression_msgs[0] - assert "%" in msg, f"Expected percentage in compression message: {msg}" - - def test_push_result_includes_upload_stats(self, vault_env): - """push() return dict has upload_stats key with correct fields.""" - engine = _make_engine(vault_env) - mock_client = _mock_http_client() - - with patch("neurostack.cloud.sync.httpx.Client", return_value=mock_client): - result = engine.push() - - assert "upload_stats" in result - stats = result["upload_stats"] - assert stats["files_uploaded"] == 2 - assert stats["raw_bytes"] > 0 - assert stats["compressed_bytes"] > 0 - assert isinstance(stats["compression_ratio"], float) - - def test_push_no_changes_reports_zero_stats(self, vault_env): - """When no changes, upload_stats has all zeros.""" - engine = _make_engine(vault_env) - mock_client = _mock_http_client() - - # First push to establish manifest - with patch("neurostack.cloud.sync.httpx.Client", return_value=mock_client): - engine.push() - - # Second push should detect no changes - result = engine.push() - - assert result["status"] == "no_changes" - assert "upload_stats" in result - stats = result["upload_stats"] - assert stats["files_uploaded"] == 0 - assert stats["raw_bytes"] == 0 - assert stats["compressed_bytes"] == 0 - assert stats["compression_ratio"] == 0.0 diff --git a/tests/test_cloud_sync.py b/tests/test_cloud_sync.py deleted file mode 100644 index 3622d5e..0000000 --- a/tests/test_cloud_sync.py +++ /dev/null @@ -1,985 +0,0 @@ -# SPDX-License-Identifier: Apache-2.0 -# Copyright (c) 2024-2026 Raphael Southall -"""Tests for cloud manifest tracking and vault sync engine.""" - -from __future__ import annotations - -import hashlib -import json -from unittest.mock import MagicMock, patch - -import pytest - -# --------------------------------------------------------------------------- -# Manifest tests -# --------------------------------------------------------------------------- - - -class TestManifestScanVault: - """Tests for Manifest.scan_vault() SHA-256 hashing.""" - - def test_scan_returns_hashes_for_md_files(self, tmp_path): - """scan_vault returns {relative_path: sha256_hex} for all .md files.""" - from neurostack.cloud.manifest import Manifest - - (tmp_path / "note1.md").write_text("hello world") - (tmp_path / "subdir").mkdir() - (tmp_path / "subdir" / "note2.md").write_text("second note") - - manifest = Manifest.scan_vault(tmp_path) - - assert "note1.md" in manifest.entries - assert "subdir/note2.md" in manifest.entries - assert len(manifest.entries) == 2 - - # Verify actual SHA-256 - expected = hashlib.sha256(b"hello world").hexdigest() - assert manifest.entries["note1.md"] == expected - - def test_scan_skips_non_md_files(self, tmp_path): - """scan_vault skips non-.md files (images, .obsidian, etc.).""" - from neurostack.cloud.manifest import Manifest - - (tmp_path / "note.md").write_text("content") - (tmp_path / "image.png").write_bytes(b"\x89PNG") - (tmp_path / "data.json").write_text("{}") - - manifest = Manifest.scan_vault(tmp_path) - - assert "note.md" in manifest.entries - assert "image.png" not in manifest.entries - assert "data.json" not in manifest.entries - assert len(manifest.entries) == 1 - - def test_scan_skips_dot_directories(self, tmp_path): - """scan_vault skips directories starting with . (.obsidian, .git, .neurostack).""" - from neurostack.cloud.manifest import Manifest - - (tmp_path / "note.md").write_text("content") - (tmp_path / ".obsidian").mkdir() - (tmp_path / ".obsidian" / "config.md").write_text("obsidian config") - (tmp_path / ".git").mkdir() - (tmp_path / ".git" / "HEAD.md").write_text("ref") - (tmp_path / ".neurostack").mkdir() - (tmp_path / ".neurostack" / "manifest.md").write_text("data") - - manifest = Manifest.scan_vault(tmp_path) - - assert len(manifest.entries) == 1 - assert "note.md" in manifest.entries - - def test_content_hash_is_deterministic(self, tmp_path): - """Same file content always produces the same hash.""" - from neurostack.cloud.manifest import Manifest - - content = "deterministic content test" - (tmp_path / "a.md").write_text(content) - - m1 = Manifest.scan_vault(tmp_path) - m2 = Manifest.scan_vault(tmp_path) - - assert m1.entries["a.md"] == m2.entries["a.md"] - assert m1.entries["a.md"] == hashlib.sha256(content.encode()).hexdigest() - - -class TestManifestLoadSave: - """Tests for Manifest.load() and Manifest.save() JSON persistence.""" - - def test_load_returns_empty_if_no_file(self, tmp_path): - """load() returns empty Manifest if file does not exist.""" - from neurostack.cloud.manifest import Manifest - - manifest = Manifest.load(tmp_path / "nonexistent.json") - assert manifest.entries == {} - - def test_load_reads_saved_manifest(self, tmp_path): - """load() loads previously saved manifest from JSON.""" - from neurostack.cloud.manifest import Manifest - - path = tmp_path / "manifest.json" - data = {"note.md": "abc123", "sub/note2.md": "def456"} - path.write_text(json.dumps(data)) - - manifest = Manifest.load(path) - assert manifest.entries == data - - def test_save_writes_json(self, tmp_path): - """save() writes {filename: hash} JSON to disk.""" - from neurostack.cloud.manifest import Manifest - - path = tmp_path / "manifest.json" - manifest = Manifest({"note.md": "abc123"}) - manifest.save(path) - - data = json.loads(path.read_text()) - assert data == {"note.md": "abc123"} - - def test_save_creates_parent_dirs(self, tmp_path): - """save() creates parent directories if needed.""" - from neurostack.cloud.manifest import Manifest - - path = tmp_path / "deep" / "nested" / "manifest.json" - manifest = Manifest({"note.md": "abc123"}) - manifest.save(path) - - assert path.exists() - data = json.loads(path.read_text()) - assert data == {"note.md": "abc123"} - - -class TestManifestDiff: - """Tests for Manifest.diff() computing SyncDiff.""" - - def test_diff_detects_added_files(self): - """diff() identifies new files not in old manifest.""" - from neurostack.cloud.manifest import Manifest - - old = Manifest({}) - new = Manifest({"note.md": "abc123"}) - - diff = Manifest.diff(old, new) - assert "note.md" in diff.added - assert diff.changed == [] - assert diff.removed == [] - - def test_diff_detects_changed_files(self): - """diff() identifies files whose content hash changed.""" - from neurostack.cloud.manifest import Manifest - - old = Manifest({"note.md": "old_hash"}) - new = Manifest({"note.md": "new_hash"}) - - diff = Manifest.diff(old, new) - assert diff.added == [] - assert "note.md" in diff.changed - assert diff.removed == [] - - def test_diff_detects_removed_files(self): - """diff() identifies files no longer in vault.""" - from neurostack.cloud.manifest import Manifest - - old = Manifest({"note.md": "abc123"}) - new = Manifest({}) - - diff = Manifest.diff(old, new) - assert diff.added == [] - assert diff.changed == [] - assert "note.md" in diff.removed - - def test_diff_returns_empty_when_identical(self): - """diff() returns empty SyncDiff when old and new are identical.""" - from neurostack.cloud.manifest import Manifest - - entries = {"note.md": "abc123", "sub/note2.md": "def456"} - old = Manifest(dict(entries)) - new = Manifest(dict(entries)) - - diff = Manifest.diff(old, new) - assert diff.added == [] - assert diff.changed == [] - assert diff.removed == [] - assert not diff.has_changes - - def test_diff_has_changes_property(self): - """has_changes is True when diff is non-empty.""" - from neurostack.cloud.manifest import Manifest - - old = Manifest({}) - new = Manifest({"note.md": "abc123"}) - - diff = Manifest.diff(old, new) - assert diff.has_changes is True - - def test_diff_upload_files_property(self): - """upload_files returns added + changed files.""" - from neurostack.cloud.manifest import Manifest - - old = Manifest({"existing.md": "old_hash"}) - new = Manifest({"existing.md": "new_hash", "new.md": "abc123"}) - - diff = Manifest.diff(old, new) - assert set(diff.upload_files) == {"existing.md", "new.md"} - - -# --------------------------------------------------------------------------- -# VaultSyncEngine tests -# --------------------------------------------------------------------------- - - -@pytest.fixture -def vault_env(tmp_path): - """Set up vault dir, db dir, and sample .md files for sync tests.""" - vault_root = tmp_path / "vault" - vault_root.mkdir() - db_dir = tmp_path / "db" - db_dir.mkdir() - manifest_path = tmp_path / "manifest" / "cloud-manifest.json" - - # Create sample vault files - (vault_root / "note1.md").write_text("first note content") - (vault_root / "sub").mkdir() - (vault_root / "sub" / "note2.md").write_text("second note content") - - return { - "vault_root": vault_root, - "db_dir": db_dir, - "manifest_path": manifest_path, - "api_url": "https://api.neurostack.sh", - "api_key": "ns_test_key_123", - } - - -def _make_engine(vault_env, **overrides): - """Create a VaultSyncEngine from vault_env fixture.""" - from neurostack.cloud.sync import VaultSyncEngine - - kwargs = { - "cloud_api_url": vault_env["api_url"], - "cloud_api_key": vault_env["api_key"], - "vault_root": vault_env["vault_root"], - "db_dir": vault_env["db_dir"], - "manifest_path": vault_env["manifest_path"], - "poll_interval": 0.01, # fast polling for tests - "poll_timeout": 1.0, # short timeout for tests - } - kwargs.update(overrides) - return VaultSyncEngine(**kwargs) - - -class TestSyncEnginePush: - """Tests for VaultSyncEngine.push() — upload changed files.""" - - def test_push_uploads_changed_files(self, vault_env): - """push() scans vault, diffs, uploads only changed files via multipart POST.""" - engine = _make_engine(vault_env) - - mock_upload_resp = MagicMock() - mock_upload_resp.status_code = 202 - mock_upload_resp.json.return_value = { - "job_id": "job-abc", - "status": "queued", - "message": "Upload received", - } - mock_upload_resp.raise_for_status = MagicMock() - - mock_status_resp = MagicMock() - mock_status_resp.status_code = 200 - mock_status_resp.json.return_value = { - "job_id": "job-abc", - "status": "complete", - "progress": 1.0, - "note_count": 2, - } - mock_status_resp.raise_for_status = MagicMock() - - mock_client = MagicMock() - mock_client.post.return_value = mock_upload_resp - mock_client.get.return_value = mock_status_resp - mock_client.__enter__ = MagicMock(return_value=mock_client) - mock_client.__exit__ = MagicMock(return_value=False) - - with patch("neurostack.cloud.sync.httpx.Client", return_value=mock_client): - result = engine.push() - - # Verify upload was called - mock_client.post.assert_called_once() - post_call = mock_client.post.call_args - assert "/v1/vault/upload" in post_call.args[0] - - # Verify result contains job info - assert result["status"] == "complete" - assert result["job_id"] == "job-abc" - - def test_push_polls_until_complete(self, vault_env): - """push() polls status endpoint until status='complete'.""" - engine = _make_engine(vault_env) - - mock_upload_resp = MagicMock() - mock_upload_resp.status_code = 202 - mock_upload_resp.json.return_value = { - "job_id": "job-abc", - "status": "queued", - "message": "Upload received", - } - mock_upload_resp.raise_for_status = MagicMock() - - # First poll returns indexing, second returns complete - mock_indexing_resp = MagicMock() - mock_indexing_resp.status_code = 200 - mock_indexing_resp.json.return_value = { - "job_id": "job-abc", - "status": "indexing", - "progress": 0.5, - } - mock_indexing_resp.raise_for_status = MagicMock() - - mock_complete_resp = MagicMock() - mock_complete_resp.status_code = 200 - mock_complete_resp.json.return_value = { - "job_id": "job-abc", - "status": "complete", - "progress": 1.0, - "note_count": 2, - } - mock_complete_resp.raise_for_status = MagicMock() - - mock_client = MagicMock() - mock_client.post.return_value = mock_upload_resp - mock_client.get.side_effect = [mock_indexing_resp, mock_complete_resp] - mock_client.__enter__ = MagicMock(return_value=mock_client) - mock_client.__exit__ = MagicMock(return_value=False) - - with patch("neurostack.cloud.sync.httpx.Client", return_value=mock_client): - result = engine.push() - - # Should have polled twice - assert mock_client.get.call_count == 2 - assert result["status"] == "complete" - - def test_push_saves_manifest_on_success(self, vault_env): - """push() saves updated manifest to disk after successful upload + indexing.""" - engine = _make_engine(vault_env) - - mock_upload_resp = MagicMock() - mock_upload_resp.status_code = 202 - mock_upload_resp.json.return_value = { - "job_id": "job-abc", - "status": "queued", - "message": "OK", - } - mock_upload_resp.raise_for_status = MagicMock() - - mock_status_resp = MagicMock() - mock_status_resp.status_code = 200 - mock_status_resp.json.return_value = { - "job_id": "job-abc", - "status": "complete", - "progress": 1.0, - } - mock_status_resp.raise_for_status = MagicMock() - - mock_client = MagicMock() - mock_client.post.return_value = mock_upload_resp - mock_client.get.return_value = mock_status_resp - mock_client.__enter__ = MagicMock(return_value=mock_client) - mock_client.__exit__ = MagicMock(return_value=False) - - with patch("neurostack.cloud.sync.httpx.Client", return_value=mock_client): - engine.push() - - # Manifest should be saved - assert vault_env["manifest_path"].exists() - data = json.loads(vault_env["manifest_path"].read_text()) - assert "note1.md" in data - assert "sub/note2.md" in data - - def test_push_no_changes_skips_upload(self, vault_env): - """push() with no changes skips upload and returns early.""" - from neurostack.cloud.manifest import Manifest - - engine = _make_engine(vault_env) - - # Pre-save a manifest matching current vault state - current = Manifest.scan_vault(vault_env["vault_root"]) - current.save(vault_env["manifest_path"]) - - mock_client = MagicMock() - mock_client.__enter__ = MagicMock(return_value=mock_client) - mock_client.__exit__ = MagicMock(return_value=False) - - with patch("neurostack.cloud.sync.httpx.Client", return_value=mock_client): - result = engine.push() - - # No HTTP calls should have been made - mock_client.post.assert_not_called() - assert result["status"] == "no_changes" - - def test_push_raises_on_indexing_failure(self, vault_env): - """push() raises SyncError if indexing fails (status='failed').""" - from neurostack.cloud.sync import SyncError - - engine = _make_engine(vault_env) - - mock_upload_resp = MagicMock() - mock_upload_resp.status_code = 202 - mock_upload_resp.json.return_value = { - "job_id": "job-abc", - "status": "queued", - "message": "OK", - } - mock_upload_resp.raise_for_status = MagicMock() - - mock_failed_resp = MagicMock() - mock_failed_resp.status_code = 200 - mock_failed_resp.json.return_value = { - "job_id": "job-abc", - "status": "failed", - "error": "Indexing error: out of memory", - } - mock_failed_resp.raise_for_status = MagicMock() - - mock_client = MagicMock() - mock_client.post.return_value = mock_upload_resp - mock_client.get.return_value = mock_failed_resp - mock_client.__enter__ = MagicMock(return_value=mock_client) - mock_client.__exit__ = MagicMock(return_value=False) - - with patch("neurostack.cloud.sync.httpx.Client", return_value=mock_client): - with pytest.raises(SyncError, match="Indexing error"): - engine.push() - - def test_push_raises_on_poll_timeout(self, vault_env): - """push() raises SyncError when poll timeout is exceeded.""" - from neurostack.cloud.sync import SyncError - - engine = _make_engine(vault_env, poll_timeout=0.05, poll_interval=0.01) - - mock_upload_resp = MagicMock() - mock_upload_resp.status_code = 202 - mock_upload_resp.json.return_value = { - "job_id": "job-abc", - "status": "queued", - "message": "OK", - } - mock_upload_resp.raise_for_status = MagicMock() - - # Always return "indexing" to trigger timeout - mock_indexing_resp = MagicMock() - mock_indexing_resp.status_code = 200 - mock_indexing_resp.json.return_value = { - "job_id": "job-abc", - "status": "indexing", - "progress": 0.5, - } - mock_indexing_resp.raise_for_status = MagicMock() - - mock_client = MagicMock() - mock_client.post.return_value = mock_upload_resp - mock_client.get.return_value = mock_indexing_resp - mock_client.__enter__ = MagicMock(return_value=mock_client) - mock_client.__exit__ = MagicMock(return_value=False) - - with patch("neurostack.cloud.sync.httpx.Client", return_value=mock_client): - with pytest.raises(SyncError, match="timed out"): - engine.push() - - -class TestSyncEnginePushRemoved: - """Tests for push() transmitting diff.removed to the server.""" - - def _mock_success_client(self): - """Return a mock httpx.Client that accepts upload and returns complete.""" - mock_upload_resp = MagicMock() - mock_upload_resp.status_code = 202 - mock_upload_resp.json.return_value = { - "job_id": "job-abc", - "status": "queued", - "message": "OK", - } - mock_upload_resp.raise_for_status = MagicMock() - - mock_status_resp = MagicMock() - mock_status_resp.status_code = 200 - mock_status_resp.json.return_value = { - "job_id": "job-abc", - "status": "complete", - "progress": 1.0, - "note_count": 2, - } - mock_status_resp.raise_for_status = MagicMock() - - mock_client = MagicMock() - mock_client.post.return_value = mock_upload_resp - mock_client.get.return_value = mock_status_resp - mock_client.__enter__ = MagicMock(return_value=mock_client) - mock_client.__exit__ = MagicMock(return_value=False) - return mock_client - - def _get_tar_manifest(self, mock_client): - """Extract _manifest.json from the tar.gz body sent via POST.""" - import io - import tarfile - - body = mock_client.post.call_args.kwargs.get("content", b"") - tar = tarfile.open(fileobj=io.BytesIO(body), mode="r:gz") - try: - manifest_file = tar.extractfile("_manifest.json") - if manifest_file is None: - return None - return json.loads(manifest_file.read()) - finally: - tar.close() - - def _get_tar_file_names(self, mock_client): - """Extract list of .md filenames from the tar.gz body sent via POST.""" - import io - import tarfile - - body = mock_client.post.call_args.kwargs.get("content", b"") - tar = tarfile.open(fileobj=io.BytesIO(body), mode="r:gz") - try: - return [n for n in tar.getnames() if n != "_manifest.json"] - finally: - tar.close() - - def test_push_includes_removed_files_in_payload(self, vault_env): - """When diff has removed files, the tar.gz manifest includes a 'removed' list.""" - from neurostack.cloud.manifest import Manifest - - engine = _make_engine(vault_env) - - # Save manifest with an extra file that no longer exists - old = Manifest.scan_vault(vault_env["vault_root"]) - old.entries["deleted.md"] = "fakehash" - old.save(vault_env["manifest_path"]) - - mock_client = self._mock_success_client() - - with patch("neurostack.cloud.sync.httpx.Client", return_value=mock_client): - engine.push() - - manifest = self._get_tar_manifest(mock_client) - assert manifest is not None - assert len(manifest["removed"]) > 0, "Expected 'removed' list in tar manifest" - - def test_push_removed_field_contains_correct_paths(self, vault_env): - """The removed list in tar manifest matches diff.removed.""" - from neurostack.cloud.manifest import Manifest - - engine = _make_engine(vault_env) - - old = Manifest.scan_vault(vault_env["vault_root"]) - old.entries["deleted.md"] = "fakehash" - old.save(vault_env["manifest_path"]) - - mock_client = self._mock_success_client() - - with patch("neurostack.cloud.sync.httpx.Client", return_value=mock_client): - engine.push() - - manifest = self._get_tar_manifest(mock_client) - assert manifest["removed"] == ["deleted.md"] - - def test_push_no_removed_files_skips_removed_field(self, vault_env): - """When diff.removed is empty, the tar manifest has an empty removed list.""" - engine = _make_engine(vault_env) - # No pre-saved manifest -> everything is 'added', nothing 'removed' - - mock_client = self._mock_success_client() - - with patch("neurostack.cloud.sync.httpx.Client", return_value=mock_client): - engine.push() - - manifest = self._get_tar_manifest(mock_client) - assert manifest["removed"] == [], "Should have empty 'removed' when nothing removed" - - def test_push_only_removals_still_sends_request(self, vault_env): - """If only files were deleted (no added/changed), push() still sends the POST.""" - from neurostack.cloud.manifest import Manifest - - engine = _make_engine(vault_env) - - # Save manifest with current files + extras - old = Manifest.scan_vault(vault_env["vault_root"]) - old.entries["ghost1.md"] = "hash1" - old.entries["ghost2.md"] = "hash2" - old.save(vault_env["manifest_path"]) - - # Now delete vault files so diff has ONLY removals (current files match old) - # Actually, old manifest already has the current files, so those won't be - # added/changed. Only ghost1.md and ghost2.md are removed. - mock_client = self._mock_success_client() - - with patch("neurostack.cloud.sync.httpx.Client", return_value=mock_client): - result = engine.push() - - # POST was sent (not skipped) - mock_client.post.assert_called_once() - assert result["status"] == "complete" - - # Removed list is present in tar manifest - manifest = self._get_tar_manifest(mock_client) - assert manifest is not None - assert sorted(manifest["removed"]) == ["ghost1.md", "ghost2.md"] - - def test_push_mixed_changes_includes_both_files_and_removed(self, vault_env): - """Added + changed files + removed files all in same tar.gz request.""" - from neurostack.cloud.manifest import Manifest - - engine = _make_engine(vault_env) - - # Old manifest: note1.md with different hash (-> changed), plus deleted.md - old = Manifest({"note1.md": "oldhash", "deleted.md": "fakehash"}) - old.save(vault_env["manifest_path"]) - - mock_client = self._mock_success_client() - - with patch("neurostack.cloud.sync.httpx.Client", return_value=mock_client): - engine.push() - - # Check tar has .md files - file_names = self._get_tar_file_names(mock_client) - assert len(file_names) >= 1 - - # Check manifest has removed - manifest = self._get_tar_manifest(mock_client) - assert manifest["removed"] == ["deleted.md"] - - def test_push_sends_gzip_content_type(self, vault_env): - """The POST request uses Content-Type: application/gzip.""" - from neurostack.cloud.manifest import Manifest - - engine = _make_engine(vault_env) - - old = Manifest.scan_vault(vault_env["vault_root"]) - old.entries["gone.md"] = "hash" - old.save(vault_env["manifest_path"]) - - mock_client = self._mock_success_client() - - with patch("neurostack.cloud.sync.httpx.Client", return_value=mock_client): - engine.push() - - headers = mock_client.post.call_args.kwargs.get("headers", {}) - assert headers.get("Content-Type") == "application/gzip" - - def test_push_saves_manifest_after_removal_sync(self, vault_env): - """Manifest is updated after push — removed files no longer in saved manifest.""" - from neurostack.cloud.manifest import Manifest - - engine = _make_engine(vault_env) - - old = Manifest.scan_vault(vault_env["vault_root"]) - old.entries["ghost.md"] = "hash" - old.save(vault_env["manifest_path"]) - - mock_client = self._mock_success_client() - - with patch("neurostack.cloud.sync.httpx.Client", return_value=mock_client): - engine.push() - - # Manifest should be re-saved without ghost.md - saved = json.loads(vault_env["manifest_path"].read_text()) - assert "ghost.md" not in saved - assert "note1.md" in saved - assert "sub/note2.md" in saved - - def test_push_removed_multiple_files(self, vault_env): - """Multiple files in removed list are all included.""" - from neurostack.cloud.manifest import Manifest - - engine = _make_engine(vault_env) - - old = Manifest.scan_vault(vault_env["vault_root"]) - old.entries["a.md"] = "h1" - old.entries["b.md"] = "h2" - old.entries["deep/c.md"] = "h3" - old.save(vault_env["manifest_path"]) - - mock_client = self._mock_success_client() - - with patch("neurostack.cloud.sync.httpx.Client", return_value=mock_client): - engine.push() - - manifest = self._get_tar_manifest(mock_client) - assert sorted(manifest["removed"]) == ["a.md", "b.md", "deep/c.md"] - - def test_push_removed_logs_count(self, vault_env): - """Logger output includes removed file count.""" - from neurostack.cloud.manifest import Manifest - - engine = _make_engine(vault_env) - - old = Manifest.scan_vault(vault_env["vault_root"]) - old.entries["gone1.md"] = "h1" - old.entries["gone2.md"] = "h2" - old.save(vault_env["manifest_path"]) - - mock_client = self._mock_success_client() - - with patch("neurostack.cloud.sync.httpx.Client", return_value=mock_client): - with patch("neurostack.cloud.sync.logger") as mock_logger: - engine.push() - - # Find the log call that mentions removed count - found = False - for call in mock_logger.info.call_args_list: - args = call.args - if len(args) >= 5 and "removed" in str(args[0]).lower(): - # The format string has %d removed as 4th positional - assert args[4] == 2, f"Expected 2 removed, got {args[4]}" - found = True - break - assert found, "Expected a log message mentioning removed file count" - - -class TestSyncEnginePull: - """Tests for VaultSyncEngine.pull() — download indexed DB.""" - - def test_pull_downloads_db(self, vault_env): - """pull() downloads DB from presigned URL and saves to db_dir.""" - engine = _make_engine(vault_env) - - # Client 1: API client that gets the download URL - mock_download_info_resp = MagicMock() - mock_download_info_resp.status_code = 200 - mock_download_info_resp.json.return_value = { - "download_url": "https://storage.example.com/db.sqlite?sig=abc", - "expires_in": 3600, - } - mock_download_info_resp.raise_for_status = MagicMock() - - mock_api_client = MagicMock() - mock_api_client.get.return_value = mock_download_info_resp - mock_api_client.__enter__ = MagicMock(return_value=mock_api_client) - mock_api_client.__exit__ = MagicMock(return_value=False) - - # Client 2: Download client that streams the DB file - db_content = b"SQLite format 3\x00fake-db-content" - mock_db_resp = MagicMock() - mock_db_resp.status_code = 200 - mock_db_resp.headers = {"content-length": str(len(db_content))} - mock_db_resp.raise_for_status = MagicMock() - mock_db_resp.iter_bytes = MagicMock(return_value=iter([db_content])) - mock_db_resp.__enter__ = MagicMock(return_value=mock_db_resp) - mock_db_resp.__exit__ = MagicMock(return_value=False) - - mock_dl_client = MagicMock() - mock_dl_client.stream.return_value = mock_db_resp - mock_dl_client.__enter__ = MagicMock(return_value=mock_dl_client) - mock_dl_client.__exit__ = MagicMock(return_value=False) - - # First call creates API client, second creates download client - with patch( - "neurostack.cloud.sync.httpx.Client", - side_effect=[mock_api_client, mock_dl_client], - ): - result_path = engine.pull() - - expected_db = vault_env["db_dir"] / "neurostack.db" - assert result_path == expected_db - assert expected_db.exists() - assert expected_db.read_bytes() == db_content - - def test_pull_creates_db_dir_if_missing(self, vault_env): - """pull() creates db_dir if it does not exist.""" - new_db_dir = vault_env["db_dir"] / "new_subdir" - engine = _make_engine(vault_env, db_dir=new_db_dir) - - mock_download_info_resp = MagicMock() - mock_download_info_resp.status_code = 200 - mock_download_info_resp.json.return_value = { - "download_url": "https://storage.example.com/db.sqlite?sig=abc", - "expires_in": 3600, - } - mock_download_info_resp.raise_for_status = MagicMock() - - mock_api_client = MagicMock() - mock_api_client.get.return_value = mock_download_info_resp - mock_api_client.__enter__ = MagicMock(return_value=mock_api_client) - mock_api_client.__exit__ = MagicMock(return_value=False) - - db_content = b"SQLite format 3\x00fake-db" - mock_db_resp = MagicMock() - mock_db_resp.status_code = 200 - mock_db_resp.headers = {"content-length": str(len(db_content))} - mock_db_resp.iter_bytes = MagicMock(return_value=iter([db_content])) - mock_db_resp.__enter__ = MagicMock(return_value=mock_db_resp) - mock_db_resp.__exit__ = MagicMock(return_value=False) - mock_db_resp.raise_for_status = MagicMock() - - mock_dl_client = MagicMock() - mock_dl_client.stream.return_value = mock_db_resp - mock_dl_client.__enter__ = MagicMock(return_value=mock_dl_client) - mock_dl_client.__exit__ = MagicMock(return_value=False) - - with patch( - "neurostack.cloud.sync.httpx.Client", - side_effect=[mock_api_client, mock_dl_client], - ): - result_path = engine.pull() - - assert new_db_dir.exists() - assert result_path.exists() - - def test_pull_detects_incomplete_download(self, vault_env): - """pull() raises SyncError when downloaded bytes don't match Content-Length.""" - from neurostack.cloud.sync import SyncError - - engine = _make_engine(vault_env) - - mock_download_info_resp = MagicMock() - mock_download_info_resp.status_code = 200 - mock_download_info_resp.json.return_value = { - "download_url": "https://storage.example.com/db.sqlite?sig=abc", - } - mock_download_info_resp.raise_for_status = MagicMock() - - mock_api_client = MagicMock() - mock_api_client.get.return_value = mock_download_info_resp - mock_api_client.__enter__ = MagicMock(return_value=mock_api_client) - mock_api_client.__exit__ = MagicMock(return_value=False) - - # Claim 1000 bytes but only deliver 10 - mock_db_resp = MagicMock() - mock_db_resp.status_code = 200 - mock_db_resp.headers = {"content-length": "1000"} - mock_db_resp.raise_for_status = MagicMock() - mock_db_resp.iter_bytes = MagicMock(return_value=iter([b"short data"])) - mock_db_resp.__enter__ = MagicMock(return_value=mock_db_resp) - mock_db_resp.__exit__ = MagicMock(return_value=False) - - mock_dl_client = MagicMock() - mock_dl_client.stream.return_value = mock_db_resp - mock_dl_client.__enter__ = MagicMock(return_value=mock_dl_client) - mock_dl_client.__exit__ = MagicMock(return_value=False) - - with patch( - "neurostack.cloud.sync.httpx.Client", - side_effect=[mock_api_client, mock_dl_client], - ): - with pytest.raises(SyncError, match="Download incomplete"): - engine.pull() - - def test_pull_download_uses_no_auth_headers(self, vault_env): - """pull() download client must NOT include Bearer auth headers.""" - engine = _make_engine(vault_env) - - mock_download_info_resp = MagicMock() - mock_download_info_resp.status_code = 200 - mock_download_info_resp.json.return_value = { - "download_url": "https://storage.example.com/db.sqlite?sig=abc", - } - mock_download_info_resp.raise_for_status = MagicMock() - - db_content = b"SQLite format 3\x00test" - mock_db_resp = MagicMock() - mock_db_resp.status_code = 200 - mock_db_resp.headers = {"content-length": str(len(db_content))} - mock_db_resp.raise_for_status = MagicMock() - mock_db_resp.iter_bytes = MagicMock(return_value=iter([db_content])) - mock_db_resp.__enter__ = MagicMock(return_value=mock_db_resp) - mock_db_resp.__exit__ = MagicMock(return_value=False) - - mock_dl_client = MagicMock() - mock_dl_client.stream.return_value = mock_db_resp - mock_dl_client.__enter__ = MagicMock(return_value=mock_dl_client) - mock_dl_client.__exit__ = MagicMock(return_value=False) - - mock_api_client = MagicMock() - mock_api_client.get.return_value = mock_download_info_resp - mock_api_client.__enter__ = MagicMock(return_value=mock_api_client) - mock_api_client.__exit__ = MagicMock(return_value=False) - - with patch( - "neurostack.cloud.sync.httpx.Client", - side_effect=[mock_api_client, mock_dl_client], - ) as mock_cls: - engine.pull() - - # First call (API client) should have auth headers - api_call = mock_cls.call_args_list[0] - assert "Authorization" in api_call.kwargs.get("headers", {}) - - # Second call (download client) should NOT have auth headers - dl_call = mock_cls.call_args_list[1] - dl_headers = dl_call.kwargs.get("headers", {}) - assert "Authorization" not in dl_headers - - -class TestSyncEngineQuery: - """Tests for VaultSyncEngine.query() — cloud search.""" - - def test_query_sends_search_request(self, vault_env): - """query() sends POST /v1/vault/query with search params.""" - engine = _make_engine(vault_env) - - mock_resp = MagicMock() - mock_resp.status_code = 200 - mock_resp.json.return_value = { - "results": [ - {"title": "note1.md", "score": 0.95, "snippet": "hello"}, - ] - } - mock_resp.raise_for_status = MagicMock() - - mock_client = MagicMock() - mock_client.post.return_value = mock_resp - mock_client.__enter__ = MagicMock(return_value=mock_client) - mock_client.__exit__ = MagicMock(return_value=False) - - with patch("neurostack.cloud.sync.httpx.Client", return_value=mock_client): - engine.query("hello world") - - mock_client.post.assert_called_once() - post_call = mock_client.post.call_args - assert "/v1/vault/query" in post_call.args[0] - body = post_call.kwargs.get("json", {}) - assert body["query"] == "hello world" - assert body["top_k"] == 10 - assert body["mode"] == "hybrid" - - def test_query_handles_501_not_implemented(self, vault_env): - """query() handles 501 gracefully with clear error message.""" - import httpx as real_httpx - - from neurostack.cloud.sync import SyncError - - engine = _make_engine(vault_env) - - mock_resp = MagicMock() - mock_resp.status_code = 501 - mock_resp.raise_for_status.side_effect = real_httpx.HTTPStatusError( - "501 Not Implemented", - request=MagicMock(), - response=mock_resp, - ) - - mock_client = MagicMock() - mock_client.post.return_value = mock_resp - mock_client.__enter__ = MagicMock(return_value=mock_client) - mock_client.__exit__ = MagicMock(return_value=False) - - with patch("neurostack.cloud.sync.httpx.Client", return_value=mock_client): - with pytest.raises(SyncError, match="not yet available"): - engine.query("test query") - - -class TestSyncEngineAuth: - """Tests for Authorization header on all HTTP calls.""" - - def test_all_requests_include_bearer_auth(self, vault_env): - """All HTTP calls include Authorization: Bearer {api_key} header.""" - engine = _make_engine(vault_env) - - mock_upload_resp = MagicMock() - mock_upload_resp.status_code = 202 - mock_upload_resp.json.return_value = { - "job_id": "job-abc", - "status": "queued", - "message": "OK", - } - mock_upload_resp.raise_for_status = MagicMock() - - mock_status_resp = MagicMock() - mock_status_resp.status_code = 200 - mock_status_resp.json.return_value = { - "job_id": "job-abc", - "status": "complete", - "progress": 1.0, - } - mock_status_resp.raise_for_status = MagicMock() - - mock_client = MagicMock() - mock_client.post.return_value = mock_upload_resp - mock_client.get.return_value = mock_status_resp - mock_client.__enter__ = MagicMock(return_value=mock_client) - mock_client.__exit__ = MagicMock(return_value=False) - - with patch("neurostack.cloud.sync.httpx.Client", return_value=mock_client) as mock_cls: - engine.push() - - # Check that Client was created with auth headers - client_call = mock_cls.call_args - headers = client_call.kwargs.get("headers", {}) - assert headers.get("Authorization") == "Bearer ns_test_key_123" diff --git a/tests/test_cloud_sync_cli.py b/tests/test_cloud_sync_cli.py deleted file mode 100644 index 1735771..0000000 --- a/tests/test_cloud_sync_cli.py +++ /dev/null @@ -1,401 +0,0 @@ -# SPDX-License-Identifier: Apache-2.0 -# Copyright (c) 2024-2026 Raphael Southall -"""Tests for cloud sync CLI subcommands (push, pull, query).""" - -from __future__ import annotations - -import json -from argparse import Namespace -from unittest.mock import MagicMock, patch - -import pytest - -from neurostack.cloud.config import CloudConfig - -# --------------------------------------------------------------------------- -# Helpers -# --------------------------------------------------------------------------- - -def _make_args(**kwargs) -> Namespace: - """Build a minimal argparse Namespace for cmd_cloud.""" - defaults = { - "command": "cloud", - "cloud_command": None, - "json": False, - "depth": "auto", - "workspace": None, - } - defaults.update(kwargs) - return Namespace(**defaults) - - -def _authed_config(url: str = "https://api.test", key: str = "test-key") -> CloudConfig: - return CloudConfig(cloud_api_url=url, cloud_api_key=key) - - -def _empty_config() -> CloudConfig: - return CloudConfig(cloud_api_url="", cloud_api_key="") - - -def _mock_get_config(tmp_path): - """Return a mock Config with vault_root and db_dir set to tmp_path.""" - cfg = MagicMock() - cfg.vault_root = tmp_path - cfg.db_dir = tmp_path / "db" - return cfg - - -# --------------------------------------------------------------------------- -# Test 1: push with no credentials prints auth error and exits 1 -# --------------------------------------------------------------------------- - -class TestCloudPushNoAuth: - @patch("neurostack.cli.cloud.load_cloud_config") - @patch("neurostack.cli.cloud.get_config") - def test_push_no_credentials_exits_1(self, mock_cfg, mock_cloud_cfg, tmp_path, capsys): - """Push without credentials prints auth error and exits 1.""" - mock_cfg.return_value = _mock_get_config(tmp_path) - mock_cloud_cfg.return_value = _empty_config() - - from neurostack.cli.cloud import cmd_cloud_push - args = _make_args(cloud_command="push") - with pytest.raises(SystemExit, match="1"): - cmd_cloud_push(args) - - err = capsys.readouterr().err - assert "Not authenticated" in err - - -# --------------------------------------------------------------------------- -# Test 2: push with valid credentials calls VaultSyncEngine.push() -# --------------------------------------------------------------------------- - -class TestCloudPushSuccess: - @patch("neurostack.cloud.sync.VaultSyncEngine") - @patch("neurostack.cli.cloud.load_cloud_config") - @patch("neurostack.cli.cloud.get_config") - def test_push_success(self, mock_cfg, mock_cloud_cfg, mock_engine_cls, tmp_path, capsys): - """Push with valid credentials calls engine.push and prints result.""" - mock_cfg.return_value = _mock_get_config(tmp_path) - mock_cloud_cfg.return_value = _authed_config() - - mock_engine = MagicMock() - mock_engine.push.return_value = { - "status": "complete", - "message": "Pushed 5 files", - "note_count": 5, - } - mock_engine_cls.return_value = mock_engine - - from neurostack.cli.cloud import cmd_cloud_push - args = _make_args(cloud_command="push") - cmd_cloud_push(args) - - mock_engine.push.assert_called_once() - out = capsys.readouterr().out - assert "Pushed 5 files" in out - - -# --------------------------------------------------------------------------- -# Test 3: push --json outputs JSON result -# --------------------------------------------------------------------------- - -class TestCloudPushJson: - @patch("neurostack.cloud.sync.VaultSyncEngine") - @patch("neurostack.cli.cloud.load_cloud_config") - @patch("neurostack.cli.cloud.get_config") - def test_push_json(self, mock_cfg, mock_cloud_cfg, mock_engine_cls, tmp_path, capsys): - """Push with --json outputs JSON.""" - mock_cfg.return_value = _mock_get_config(tmp_path) - mock_cloud_cfg.return_value = _authed_config() - - result_dict = {"status": "complete", "message": "Pushed 5 files", "note_count": 5} - mock_engine = MagicMock() - mock_engine.push.return_value = result_dict - mock_engine_cls.return_value = mock_engine - - from neurostack.cli.cloud import cmd_cloud_push - args = _make_args(cloud_command="push", json=True) - cmd_cloud_push(args) - - out = capsys.readouterr().out - parsed = json.loads(out) - assert parsed["status"] == "complete" - assert parsed["note_count"] == 5 - - -# --------------------------------------------------------------------------- -# Test 4: pull with valid credentials calls engine.pull() and prints path -# --------------------------------------------------------------------------- - -class TestCloudPullSuccess: - @patch("neurostack.cloud.sync.VaultSyncEngine") - @patch("neurostack.cli.cloud.load_cloud_config") - @patch("neurostack.cli.cloud.get_config") - def test_pull_success(self, mock_cfg, mock_cloud_cfg, mock_engine_cls, tmp_path, capsys): - """Pull calls engine.pull and prints download path.""" - mock_cfg.return_value = _mock_get_config(tmp_path) - mock_cloud_cfg.return_value = _authed_config() - - # Create a fake DB file so stat() works - db_path = tmp_path / "db" / "neurostack.db" - db_path.parent.mkdir(parents=True, exist_ok=True) - db_path.write_bytes(b"x" * 1024 * 1024) # 1 MB - - mock_engine = MagicMock() - mock_engine.pull.return_value = db_path - mock_engine_cls.return_value = mock_engine - - from neurostack.cli.cloud import cmd_cloud_pull - args = _make_args(cloud_command="pull") - cmd_cloud_pull(args) - - mock_engine.pull.assert_called_once() - out = capsys.readouterr().out - assert "Downloaded" in out - assert "1.0 MB" in out - assert "Setup complete" in out - - -# --------------------------------------------------------------------------- -# Test 5: pull --json outputs JSON with db_path and size -# --------------------------------------------------------------------------- - -class TestCloudPullJson: - @patch("neurostack.cloud.sync.VaultSyncEngine") - @patch("neurostack.cli.cloud.load_cloud_config") - @patch("neurostack.cli.cloud.get_config") - def test_pull_json(self, mock_cfg, mock_cloud_cfg, mock_engine_cls, tmp_path, capsys): - """Pull with --json outputs JSON with db_path and size.""" - mock_cfg.return_value = _mock_get_config(tmp_path) - mock_cloud_cfg.return_value = _authed_config() - - db_path = tmp_path / "db" / "neurostack.db" - db_path.parent.mkdir(parents=True, exist_ok=True) - db_path.write_bytes(b"x" * 2048) - - mock_engine = MagicMock() - mock_engine.pull.return_value = db_path - mock_engine_cls.return_value = mock_engine - - from neurostack.cli.cloud import cmd_cloud_pull - args = _make_args(cloud_command="pull", json=True) - cmd_cloud_pull(args) - - out = capsys.readouterr().out - parsed = json.loads(out) - assert parsed["db_path"] == str(db_path) - assert parsed["size"] == 2048 - - -# --------------------------------------------------------------------------- -# Test 6: query calls engine.query() and prints formatted results -# --------------------------------------------------------------------------- - -class TestCloudQuerySuccess: - @patch("neurostack.cli.cloud.CloudClient") - @patch("neurostack.cli.cloud.load_cloud_config") - @patch("neurostack.cli.cloud.get_config") - def test_query_success(self, mock_cfg, mock_cloud_cfg, mock_client_cls, tmp_path, capsys): - """Query prints formatted results.""" - mock_cfg.return_value = _mock_get_config(tmp_path) - mock_cloud_cfg.return_value = _authed_config() - - mock_client = MagicMock() - mock_client.query.return_value = { - "depth_used": "auto", - "triples": [], - "summaries": [ - {"title": "Note A", "summary": "This is about testing"}, - {"title": "Note B", "summary": "Another result"}, - ], - "chunks": [], - } - mock_client_cls.return_value = mock_client - - from neurostack.cli.cloud import cmd_cloud_query - args = _make_args(cloud_command="query", query="test", top_k=10, mode="hybrid") - cmd_cloud_query(args) - - out = capsys.readouterr().out - assert "Note A" in out - assert "Note B" in out - - @patch("neurostack.cli.cloud.CloudClient") - @patch("neurostack.cli.cloud.load_cloud_config") - @patch("neurostack.cli.cloud.get_config") - def test_query_no_results(self, mock_cfg, mock_cloud_cfg, mock_client_cls, tmp_path, capsys): - """Query with no results prints 'No results found.'.""" - mock_cfg.return_value = _mock_get_config(tmp_path) - mock_cloud_cfg.return_value = _authed_config() - - mock_client = MagicMock() - mock_client.query.return_value = { - "depth_used": "auto", - "triples": [], - "summaries": [], - "chunks": [], - } - mock_client_cls.return_value = mock_client - - from neurostack.cli.cloud import cmd_cloud_query - args = _make_args(cloud_command="query", query="nonexistent", top_k=10, mode="hybrid") - cmd_cloud_query(args) - - out = capsys.readouterr().out - assert "No results found" in out - - -# --------------------------------------------------------------------------- -# Test 7: query --json outputs JSON results -# --------------------------------------------------------------------------- - -class TestCloudQueryJson: - @patch("neurostack.cli.cloud.CloudClient") - @patch("neurostack.cli.cloud.load_cloud_config") - @patch("neurostack.cli.cloud.get_config") - def test_query_json(self, mock_cfg, mock_cloud_cfg, mock_client_cls, tmp_path, capsys): - """Query with --json outputs JSON array.""" - mock_cfg.return_value = _mock_get_config(tmp_path) - mock_cloud_cfg.return_value = _authed_config() - - results = { - "depth_used": "auto", "triples": [], - "summaries": [{"title": "Note A"}], "chunks": [], - } - mock_client = MagicMock() - mock_client.query.return_value = results - mock_client_cls.return_value = mock_client - - from neurostack.cli.cloud import cmd_cloud_query - args = _make_args(cloud_command="query", query="test", top_k=10, mode="hybrid", json=True) - cmd_cloud_query(args) - - out = capsys.readouterr().out - parsed = json.loads(out) - assert parsed["summaries"][0]["title"] == "Note A" - - -# --------------------------------------------------------------------------- -# Test 8: query --top-k 5 --mode semantic passes args correctly -# --------------------------------------------------------------------------- - -class TestCloudQueryArgs: - @patch("neurostack.cli.cloud.CloudClient") - @patch("neurostack.cli.cloud.load_cloud_config") - @patch("neurostack.cli.cloud.get_config") - def test_query_custom_args(self, mock_cfg, mock_cloud_cfg, mock_client_cls, tmp_path): - """Query passes top_k and mode to engine correctly.""" - mock_cfg.return_value = _mock_get_config(tmp_path) - mock_cloud_cfg.return_value = _authed_config() - - mock_client = MagicMock() - mock_client.query.return_value = { - "depth_used": "auto", "triples": [], - "summaries": [], "chunks": [], - } - mock_client_cls.return_value = mock_client - - from neurostack.cli.cloud import cmd_cloud_query - args = _make_args(cloud_command="query", query="hello", top_k=5, mode="semantic") - cmd_cloud_query(args) - - -# --------------------------------------------------------------------------- -# Test 9: All commands exit 1 with SyncError message on failure -# --------------------------------------------------------------------------- - -class TestCloudSyncErrors: - @patch("neurostack.cloud.sync.VaultSyncEngine") - @patch("neurostack.cli.cloud.load_cloud_config") - @patch("neurostack.cli.cloud.get_config") - def test_push_sync_error(self, mock_cfg, mock_cloud_cfg, mock_engine_cls, tmp_path, capsys): - """Push exits 1 and prints error on SyncError.""" - from neurostack.cloud.sync import SyncError - - mock_cfg.return_value = _mock_get_config(tmp_path) - mock_cloud_cfg.return_value = _authed_config() - - mock_engine = MagicMock() - mock_engine.push.side_effect = SyncError("Upload failed: server error") - mock_engine_cls.return_value = mock_engine - - from neurostack.cli.cloud import cmd_cloud_push - args = _make_args(cloud_command="push") - with pytest.raises(SystemExit, match="1"): - cmd_cloud_push(args) - - err = capsys.readouterr().err - assert "Upload failed: server error" in err - - @patch("neurostack.cloud.sync.VaultSyncEngine") - @patch("neurostack.cli.cloud.load_cloud_config") - @patch("neurostack.cli.cloud.get_config") - def test_pull_sync_error(self, mock_cfg, mock_cloud_cfg, mock_engine_cls, tmp_path, capsys): - """Pull exits 1 and prints error on SyncError.""" - from neurostack.cloud.sync import SyncError - - mock_cfg.return_value = _mock_get_config(tmp_path) - mock_cloud_cfg.return_value = _authed_config() - - mock_engine = MagicMock() - mock_engine.pull.side_effect = SyncError("Download failed") - mock_engine_cls.return_value = mock_engine - - from neurostack.cli.cloud import cmd_cloud_pull - args = _make_args(cloud_command="pull") - with pytest.raises(SystemExit, match="1"): - cmd_cloud_pull(args) - - err = capsys.readouterr().err - assert "Download failed" in err - - @patch("neurostack.cli.cloud.CloudClient") - @patch("neurostack.cli.cloud.load_cloud_config") - @patch("neurostack.cli.cloud.get_config") - def test_query_sync_error(self, mock_cfg, mock_cloud_cfg, mock_client_cls, tmp_path, capsys): - """Query exits 1 and prints error on failure.""" - mock_cfg.return_value = _mock_get_config(tmp_path) - mock_cloud_cfg.return_value = _authed_config() - - mock_client = MagicMock() - mock_client.query.side_effect = Exception("Query API not available") - mock_client_cls.return_value = mock_client - - from neurostack.cli.cloud import cmd_cloud_query - args = _make_args(cloud_command="query", query="test", top_k=10, mode="hybrid") - with pytest.raises(SystemExit, match="1"): - cmd_cloud_query(args) - - err = capsys.readouterr().err - assert "Query API not available" in err - - @patch("neurostack.cli.cloud.load_cloud_config") - @patch("neurostack.cli.cloud.get_config") - def test_pull_no_credentials_exits_1(self, mock_cfg, mock_cloud_cfg, tmp_path, capsys): - """Pull without credentials prints auth error and exits 1.""" - mock_cfg.return_value = _mock_get_config(tmp_path) - mock_cloud_cfg.return_value = _empty_config() - - from neurostack.cli.cloud import cmd_cloud_pull - args = _make_args(cloud_command="pull") - with pytest.raises(SystemExit, match="1"): - cmd_cloud_pull(args) - - err = capsys.readouterr().err - assert "Not authenticated" in err - - @patch("neurostack.cli.cloud.load_cloud_config") - @patch("neurostack.cli.cloud.get_config") - def test_query_no_credentials_exits_1(self, mock_cfg, mock_cloud_cfg, tmp_path, capsys): - """Query without credentials prints auth error and exits 1.""" - mock_cfg.return_value = _mock_get_config(tmp_path) - mock_cloud_cfg.return_value = _empty_config() - - from neurostack.cli.cloud import cmd_cloud_query - args = _make_args(cloud_command="query", query="test", top_k=10, mode="hybrid") - with pytest.raises(SystemExit, match="1"): - cmd_cloud_query(args) - - err = capsys.readouterr().err - assert "Not authenticated" in err diff --git a/tests/test_cloud_tar_upload.py b/tests/test_cloud_tar_upload.py deleted file mode 100644 index 926f139..0000000 --- a/tests/test_cloud_tar_upload.py +++ /dev/null @@ -1,152 +0,0 @@ -# SPDX-License-Identifier: Apache-2.0 -# Copyright (c) 2024-2026 Raphael Southall -"""Tests for tar.gz upload format in the vault sync engine.""" - -from __future__ import annotations - -import hashlib -import io -import json -import tarfile -from dataclasses import dataclass, field -from pathlib import Path -from unittest.mock import MagicMock, patch - - -@dataclass -class _FakeDiff: - """Minimal stand-in for SyncDiff used by _build_tar_archive.""" - - added: list[str] = field(default_factory=list) - changed: list[str] = field(default_factory=list) - removed: list[str] = field(default_factory=list) - - @property - def upload_files(self) -> list[str]: - return self.added + self.changed - - @property - def has_changes(self) -> bool: - return bool(self.added or self.changed or self.removed) - - -def _make_engine(tmp_path: Path): - """Create a VaultSyncEngine pointed at *tmp_path*.""" - from neurostack.cloud.sync import VaultSyncEngine - - return VaultSyncEngine( - cloud_api_url="https://api.example.com", - cloud_api_key="sk-test", - vault_root=tmp_path, - db_dir=tmp_path / "db", - ) - - -class TestBuildTarArchive: - """Tests for VaultSyncEngine._build_tar_archive.""" - - def test_build_tar_archive_contains_manifest(self, tmp_path): - """_build_tar_archive creates a valid tar.gz with _manifest.json.""" - (tmp_path / "note.md").write_text("# Hello") - engine = _make_engine(tmp_path) - diff = _FakeDiff(added=["note.md"]) - - archive = engine._build_tar_archive(diff.upload_files, diff) - - tar = tarfile.open(fileobj=io.BytesIO(archive), mode="r:gz") - names = tar.getnames() - assert "_manifest.json" in names - tar.close() - - def test_build_tar_archive_contains_files(self, tmp_path): - """tar contains all upload files with correct content.""" - (tmp_path / "a.md").write_text("alpha") - (tmp_path / "sub").mkdir() - (tmp_path / "sub" / "b.md").write_text("beta") - engine = _make_engine(tmp_path) - diff = _FakeDiff(added=["a.md", "sub/b.md"]) - - archive = engine._build_tar_archive(diff.upload_files, diff) - - tar = tarfile.open(fileobj=io.BytesIO(archive), mode="r:gz") - assert "a.md" in tar.getnames() - assert "sub/b.md" in tar.getnames() - - a_content = tar.extractfile("a.md").read() - assert a_content == b"alpha" - - b_content = tar.extractfile("sub/b.md").read() - assert b_content == b"beta" - tar.close() - - def test_manifest_has_correct_format(self, tmp_path): - """_manifest.json has format_version=1, removed list, file_hashes with sha256 prefix.""" - (tmp_path / "note.md").write_text("content") - engine = _make_engine(tmp_path) - diff = _FakeDiff(added=["note.md"], removed=["old.md"]) - - archive = engine._build_tar_archive(diff.upload_files, diff) - - tar = tarfile.open(fileobj=io.BytesIO(archive), mode="r:gz") - manifest = json.loads(tar.extractfile("_manifest.json").read()) - tar.close() - - assert manifest["format_version"] == 1 - assert manifest["removed"] == ["old.md"] - assert "note.md" in manifest["file_hashes"] - - # Hash should have sha256: prefix - h = manifest["file_hashes"]["note.md"] - assert h.startswith("sha256:") - - # Verify the actual hash value - expected = "sha256:" + hashlib.sha256(b"content").hexdigest() - assert h == expected - - def test_push_sends_tar_format(self, tmp_path): - """push() sends Content-Type: application/gzip with tar.gz body.""" - (tmp_path / "note.md").write_text("# Test note") - engine = _make_engine(tmp_path) - - # Mock manifest load/save so diff shows the file as added - mock_manifest = MagicMock() - mock_manifest.entries = {} - - # Mock the HTTP response chain - mock_upload_resp = MagicMock() - mock_upload_resp.json.return_value = {"job_id": "job-123", "status": "queued"} - mock_upload_resp.raise_for_status = MagicMock() - - mock_status_resp = MagicMock() - mock_status_resp.json.return_value = {"status": "complete"} - mock_status_resp.raise_for_status = MagicMock() - - captured_kwargs = {} - - def fake_post(url, **kwargs): - captured_kwargs.update(kwargs) - return mock_upload_resp - - mock_client = MagicMock() - mock_client.__enter__ = MagicMock(return_value=mock_client) - mock_client.__exit__ = MagicMock(return_value=False) - mock_client.post = fake_post - mock_client.get = MagicMock(return_value=mock_status_resp) - - with ( - patch("neurostack.cloud.sync.Manifest.load", return_value=mock_manifest), - patch("neurostack.cloud.sync.Manifest.save"), - patch("httpx.Client", return_value=mock_client), - ): - engine.push() - - # Verify Content-Type header was set - assert "headers" in captured_kwargs - assert captured_kwargs["headers"]["Content-Type"] == "application/gzip" - assert captured_kwargs["headers"]["X-Upload-Format"] == "tar.gz" - - # Verify content is valid tar.gz - body = captured_kwargs["content"] - tar = tarfile.open(fileobj=io.BytesIO(body), mode="r:gz") - assert "_manifest.json" in tar.getnames() - tar.close() diff --git a/tests/test_cloud_timer.py b/tests/test_cloud_timer.py deleted file mode 100644 index 416a9ee..0000000 --- a/tests/test_cloud_timer.py +++ /dev/null @@ -1,91 +0,0 @@ -"""Tests for neurostack.cloud.timer — systemd user timer for periodic cloud sync.""" -from __future__ import annotations - -from unittest.mock import MagicMock, patch - -import pytest - -from neurostack.cloud.timer import ( - SERVICE_NAME, - install_timer, - timer_status, - uninstall_timer, -) - - -@pytest.fixture(autouse=True) -def _isolate(tmp_path): - """Redirect systemd dir to tmp_path and stub subprocess + path lookup.""" - with ( - patch("neurostack.cloud.timer._systemd_user_dir", return_value=tmp_path), - patch("neurostack.cloud.timer._find_neurostack_path", return_value="/usr/bin/neurostack"), - patch("neurostack.cloud.timer.subprocess") as mock_sub, - ): - mock_sub.run = MagicMock() - yield tmp_path, mock_sub - - -def test_install_timer_creates_service_file(_isolate): - tmp_path, _ = _isolate - result = install_timer(interval="10min") - service_path = tmp_path / f"{SERVICE_NAME}.service" - assert service_path.exists() - content = service_path.read_text() - assert "ExecStart=/usr/bin/neurostack cloud sync --quiet" in content - assert result["service_path"] == str(service_path) - - -def test_install_timer_creates_timer_file(_isolate): - tmp_path, _ = _isolate - result = install_timer(interval="10min") - timer_path = tmp_path / f"{SERVICE_NAME}.timer" - assert timer_path.exists() - content = timer_path.read_text() - assert "OnBootSec=5min" in content - assert result["timer_path"] == str(timer_path) - - -def test_timer_file_has_correct_interval(_isolate): - tmp_path, _ = _isolate - install_timer(interval="30min") - timer_path = tmp_path / f"{SERVICE_NAME}.timer" - content = timer_path.read_text() - assert "OnUnitActiveSec=30min" in content - - -def test_uninstall_timer_removes_files(_isolate): - tmp_path, _ = _isolate - install_timer(interval="15min") - service_path = tmp_path / f"{SERVICE_NAME}.service" - timer_path = tmp_path / f"{SERVICE_NAME}.timer" - assert service_path.exists() - assert timer_path.exists() - - result = uninstall_timer() - assert result["removed"] is True - assert not service_path.exists() - assert not timer_path.exists() - assert len(result["paths"]) == 2 - - -def test_timer_status_not_installed(_isolate): - status = timer_status() - assert status["installed"] is False - assert status["active"] is False - assert status["interval"] is None - assert status["next_run"] is None - - -def test_timer_status_installed(_isolate): - tmp_path, mock_sub = _isolate - install_timer(interval="20min") - - # Make is-active return non-zero (inactive but installed) - inactive_result = MagicMock() - inactive_result.returncode = 1 - mock_sub.run.return_value = inactive_result - - status = timer_status() - assert status["installed"] is True - assert status["interval"] == "20min" - assert status["active"] is False