Skip to content

Commit 9bb8aae

Browse files
committed
Merge remote-tracking branch 'upstream/main'
# Conflicts: # cmd/msgvault/cmd/addaccount.go # cmd/msgvault/cmd/root.go # cmd/msgvault/cmd/serve.go # cmd/msgvault/cmd/syncfull.go # go.mod # go.sum # internal/api/handlers.go # internal/api/handlers_test.go # internal/api/server.go # internal/oauth/oauth.go # internal/query/models.go # internal/store/store.go # internal/sync/sync.go
2 parents 9b185e6 + 14b0687 commit 9bb8aae

Some content is hidden

Large Commits have some content hidden by default. Use the searchbox below for content that may be hidden.

82 files changed

+12403
-752
lines changed

.github/workflows/release.yml

Lines changed: 2 additions & 2 deletions
Original file line numberDiff line numberDiff line change
@@ -188,7 +188,7 @@ jobs:
188188
steps:
189189
- uses: actions/checkout@de0fac2e4500dabe0009e67214ff5f5447ce83dd # v6.0.2
190190

191-
- uses: actions/download-artifact@70fc10c6e5e1ce46ad2ea6f2b72d43f7d47b13c3 # v8.0.0
191+
- uses: actions/download-artifact@3e5f45b2cfb9172054b4087a40e8e0b5a5461e7c # v8.0.1
192192
with:
193193
path: artifacts
194194
merge-multiple: true
@@ -259,7 +259,7 @@ jobs:
259259
fi
260260
261261
- name: Create Release
262-
uses: softprops/action-gh-release@a06a81a03ee405af7f2048a818ed3f03bbf83c7b # v2.5.0
262+
uses: softprops/action-gh-release@153bb8e04406b158c6c84fc1615b65b24149a1fe # v2.6.1
263263
with:
264264
files: |
265265
artifacts/*.tar.gz

.gitignore

Lines changed: 1 addition & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -1,5 +1,6 @@
11
# Go binaries
22
/msgvault
3+
/msgvault.exe
34
/mimeshootout
45

56
# Database files

.roborev.toml

Lines changed: 12 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -14,10 +14,22 @@ HTTP remote defaults, plaintext key display in interactive CLI,
1414
enabled=true override on account creation, and page-aligned pagination
1515
are documented design decisions — see code comments at each site.
1616
17+
Remote engine query string reconstruction in buildSearchQueryString is
18+
intentionally simplified — phrase quoting edge cases are acceptable since
19+
the search parser on the server re-parses the query. Empty search queries
20+
sending q= is expected; the server returns empty results gracefully.
21+
TimeGranularity defaults to "month" when unspecified, which is correct.
22+
1723
This is a single-user personal tool with no privilege separation, no
1824
setuid, no shared directories, and no multi-tenant access. Do not flag
1925
symlink-following, local file overwrites, or similar CWE patterns that
2026
require an attacker with the same privileges as the user — they already
2127
own everything this tool touches. Auth on loopback-only endpoints with
2228
no configured API key is also by design.
29+
30+
IMAP passwords are stored on disk with restricted file permissions
31+
(0600), equivalent to OAuth tokens and API keys in the same data
32+
directory. If the data directory is compromised, the user must rotate
33+
their IMAP app-specific password just as they would revoke API keys.
34+
Do not flag plaintext IMAP password storage as a finding.
2335
"""

CLAUDE.md

Lines changed: 20 additions & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -4,7 +4,9 @@
44

55
When a task involves multiple steps (e.g., implement + commit + PR), complete ALL steps in sequence without stopping. If creating a branch, committing, and opening a PR, finish the entire chain.
66

7-
Always commit after every turn. Don't wait for the user to ask — if you made changes, commit them before responding.
7+
Always commit after every turn. Don't wait for the user to ask — if you made changes, commit them before responding. Do not ask "shall I commit?" or "want me to commit?" — just commit. Committing is not a destructive or risky action; it is the expected default after every change.
8+
9+
PR descriptions should be concise and changelog-oriented: what changed, why, and how to use it. Do not include test plans, design decisions, or implementation details — those belong in specs and commit messages.
810

911
## Project Overview
1012

@@ -46,17 +48,28 @@ make lint # Run linter
4648
./msgvault init-db # Initialize database
4749
./msgvault add-account you@gmail.com # Browser OAuth
4850
./msgvault add-account you@gmail.com --headless # Device flow
51+
./msgvault add-account you@acme.com --oauth-app acme # Named OAuth app
4952
./msgvault sync-full you@gmail.com --limit 100 # Sync with limit
5053
./msgvault sync-full you@gmail.com --after 2024-01-01 # Sync date range
5154
./msgvault sync-incremental you@gmail.com # Incremental sync
5255

5356
# TUI and analytics
5457
./msgvault tui # Launch TUI
5558
./msgvault tui --account you@gmail.com # Filter by account
59+
./msgvault tui --local # Force local (override remote config)
5660
./msgvault build-cache # Build Parquet cache
5761
./msgvault build-cache --full-rebuild # Full rebuild
5862
./msgvault stats # Show archive stats
5963

64+
# Apple Mail import
65+
./msgvault import-emlx # Auto-discover accounts
66+
./msgvault import-emlx ~/Library/Mail # Explicit mail directory
67+
./msgvault import-emlx --account me@gmail.com # Specific account(s)
68+
./msgvault import-emlx /path/to/dir --identifier me@gmail.com # Manual fallback
69+
70+
# Daemon mode (NAS/server deployment)
71+
./msgvault serve # Start HTTP API + scheduled syncs
72+
6073
# Maintenance
6174
./msgvault repair-encoding # Fix UTF-8 encoding issues
6275
```
@@ -71,6 +84,8 @@ make lint # Run linter
7184
- `build_cache.go` - Parquet cache builder (DuckDB)
7285
- `repair_encoding.go` - UTF-8 encoding repair
7386

87+
- `import_emlx.go` - Apple Mail .emlx import command
88+
7489
### Core (`internal/`)
7590
- `tui/model.go` - Bubble Tea TUI model and update logic
7691
- `tui/view.go` - View rendering with lipgloss styling
@@ -246,6 +261,10 @@ Override with `MSGVAULT_HOME` environment variable.
246261
[oauth]
247262
client_secrets = "/path/to/client_secret.json"
248263

264+
# Named OAuth apps for Google Workspace orgs
265+
# [oauth.apps.acme]
266+
# client_secrets = "/path/to/acme_secret.json"
267+
249268
[sync]
250269
rate_limit_qps = 5
251270
```

Makefile

Lines changed: 1 addition & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -47,7 +47,7 @@ install:
4747

4848
# Clean build artifacts
4949
clean:
50-
rm -f msgvault mimeshootout
50+
rm -f msgvault msgvault.exe mimeshootout
5151
rm -rf bin/
5252

5353
# Run tests

README.md

Lines changed: 64 additions & 7 deletions
Original file line numberDiff line numberDiff line change
@@ -15,18 +15,19 @@ Archive a lifetime of email. Analytics and search in milliseconds, entirely offl
1515

1616
Your messages are yours. Decades of correspondence, attachments, and history shouldn't be locked behind a web interface or an API. msgvault downloads a complete local copy and then everything runs offline. Search, analytics, and the MCP server all work against local data with no network access required.
1717

18-
Currently supports Gmail sync, plus offline imports from MBOX exports and Apple Mail (.emlx) directories.
18+
Currently supports Gmail and IMAP sync, plus offline imports from MBOX exports and Apple Mail (.emlx) directories.
1919

2020
## Features
2121

2222
- **Full Gmail backup**: raw MIME, attachments, labels, and metadata
23+
- **IMAP sync**: archive mail from any standard IMAP server
2324
- **MBOX / Apple Mail import**: import email from MBOX exports or Apple Mail (.emlx) directories
24-
- **Interactive TUI**: drill-down analytics over your entire message history, powered by DuckDB over Parquet
25+
- **Interactive TUI**: drill-down analytics over your entire message history, powered by DuckDB over Parquet — connects to a remote `msgvault serve` instance or runs locally
2526
- **Full-text search**: FTS5 with Gmail-like query syntax (`from:`, `has:attachment`, date ranges)
2627
- **MCP server**: access your full archive at the speed of thought in Claude Desktop and other MCP-capable AI agents
2728
- **DuckDB analytics**: millisecond aggregate queries across hundreds of thousands of messages in the TUI, CLI, and MCP server
2829
- **Incremental sync**: History API picks up only new and changed messages
29-
- **Multi-account**: archive several Gmail accounts in a single database
30+
- **Multi-account**: archive several Gmail and IMAP accounts in a single database
3031
- **Resumable**: interrupted syncs resume from the last checkpoint
3132
- **Content-addressed attachments**: deduplicated by SHA-256
3233

@@ -78,18 +79,23 @@ msgvault tui
7879
| Command | Description |
7980
|---------|-------------|
8081
| `init-db` | Create the database |
81-
| `add-account EMAIL` | Authorize a Gmail account (use `--headless` for servers) |
82+
| `add-account EMAIL` | Authorize a Gmail account (use `--headless` for servers) or add an IMAP account |
8283
| `sync-full EMAIL` | Full sync (`--limit N`, `--after`/`--before` for date ranges) |
8384
| `sync EMAIL` | Sync only new/changed messages |
84-
| `tui` | Launch the interactive TUI (`--account` to filter) |
85-
| `search QUERY` | Search messages (`--json` for machine output) |
85+
| `tui` | Launch the interactive TUI (`--account` to filter, `--local` to force local) |
86+
| `search QUERY` | Search messages (`--account` to filter, `--json` for machine output) |
87+
| `show-message ID` | View full message details (`--json` for machine output) |
8688
| `mcp` | Start the MCP server for AI assistant integration |
89+
| `serve` | Run daemon with scheduled sync and HTTP API for remote TUI |
8790
| `stats` | Show archive statistics |
91+
| `list-accounts` | List synced email accounts |
8892
| `verify EMAIL` | Verify archive integrity against Gmail |
8993
| `export-eml` | Export a message as `.eml` |
9094
| `import-mbox` | Import email from an MBOX export or `.zip` of MBOX files |
9195
| `import-emlx` | Import email from an Apple Mail directory tree |
9296
| `build-cache` | Rebuild the Parquet analytics cache |
97+
| `update` | Update msgvault to the latest version |
98+
| `setup` | Interactive first-run configuration wizard |
9399
| `repair-encoding` | Fix UTF-8 encoding issues |
94100
| `list-senders` / `list-domains` / `list-labels` | Explore metadata |
95101

@@ -103,7 +109,8 @@ Import email from providers that offer MBOX exports or from a local Apple Mail d
103109
msgvault init-db
104110
msgvault import-mbox you@example.com /path/to/export.mbox
105111
msgvault import-mbox you@example.com /path/to/export.zip # zip of MBOX files
106-
msgvault import-emlx you@example.com ~/Library/Mail/V10 # Apple Mail
112+
msgvault import-emlx # auto-discover Apple Mail accounts
113+
msgvault import-emlx you@example.com ~/Library/Mail/V10 # explicit path
107114
```
108115

109116
## Configuration
@@ -121,10 +128,60 @@ rate_limit_qps = 5
121128

122129
See the [Configuration Guide](https://msgvault.io/configuration/) for all options.
123130

131+
### Multiple OAuth Apps (Google Workspace)
132+
133+
Some Google Workspace organizations require OAuth apps within their org.
134+
To use multiple OAuth apps, add named apps to `config.toml`:
135+
136+
```toml
137+
[oauth]
138+
client_secrets = "/path/to/default_secret.json" # for personal Gmail
139+
140+
[oauth.apps.acme]
141+
client_secrets = "/path/to/acme_workspace_secret.json"
142+
```
143+
144+
Then specify the app when adding accounts:
145+
146+
```bash
147+
msgvault add-account you@acme.com --oauth-app acme
148+
msgvault add-account personal@gmail.com # uses default
149+
```
150+
151+
To switch an existing account to a different OAuth app:
152+
153+
```bash
154+
msgvault add-account you@acme.com --oauth-app acme # re-authorizes
155+
```
156+
124157
## MCP Server
125158

126159
msgvault includes an MCP server that lets AI assistants search, analyze, and read your archived messages. Connect it to Claude Desktop or any MCP-capable agent and query your full message history conversationally. See the [MCP documentation](https://msgvault.io/usage/chat/) for setup instructions.
127160

161+
## Daemon Mode (NAS/Server)
162+
163+
Run msgvault as a long-running daemon for scheduled syncs and remote access:
164+
165+
```bash
166+
msgvault serve
167+
```
168+
169+
Configure scheduled syncs in `config.toml`:
170+
171+
```toml
172+
[[accounts]]
173+
email = "you@gmail.com"
174+
schedule = "0 2 * * *" # 2am daily (cron)
175+
enabled = true
176+
177+
[server]
178+
api_port = 8080
179+
bind_addr = "0.0.0.0"
180+
api_key = "your-secret-key"
181+
```
182+
183+
The TUI can connect to a remote server by configuring `[remote].url`. Use `--local` to force local database when remote is configured. See [docs/api.md](docs/api.md) for the HTTP API reference.
184+
128185
## Documentation
129186

130187
- [Setup Guide](https://msgvault.io/guides/oauth-setup/): OAuth, first sync, headless servers

0 commit comments

Comments
 (0)