Skip to content
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
6 changes: 3 additions & 3 deletions .env.example
Original file line number Diff line number Diff line change
Expand Up @@ -16,9 +16,9 @@ OLLAMA_PORT=11434
OLLAMA_MODEL=llama3.2:3b

# Centralized Sync (optional, for multi-machine knowledge sharing)
ODIN_SYNC_ENABLED=false
# ODIN_URL=http://your-server:8080
# ODIN_API_TOKEN=your-token
REMOTE_SYNC_ENABLED=false
# REMOTE_SYNC_URL=http://your-server:8080
# REMOTE_SYNC_TOKEN=your-token

# Cloud API Sync (optional)
# PREFRONTAL_API_URL=http://your-api:8080
Expand Down
2 changes: 1 addition & 1 deletion CLAUDE.md
Original file line number Diff line number Diff line change
Expand Up @@ -48,7 +48,7 @@ vendor/bin/pest tests/Feature/Commands/KnowledgeSearchCommandTest.php
| `QdrantService` | All vector DB operations (upsert, search, delete, collections) |
| `EmbeddingService` | Text-to-vector conversion |
| `KnowledgeCacheService` | Redis caching for sub-200ms queries |
| `OdinSyncService` | Background sync to centralized Odin server |
| `RemoteSyncService` | Background sync to centralized remote server |
| `WriteGateService` | Filters knowledge quality before persistence |
| `EntryMetadataService` | Staleness detection, confidence degradation |
| `CorrectionService` | Multi-tier correction propagation |
Expand Down
8 changes: 4 additions & 4 deletions MISSION.md
Original file line number Diff line number Diff line change
Expand Up @@ -16,7 +16,7 @@ Knowledge is a command-line tool that captures technical decisions, learnings, a

### 2. Offline-First, Sync-Later
- **Local Qdrant**: Full functionality without network
- **Background sync**: Queue writes, sync to Odin every N minutes
- **Background sync**: Queue writes, sync to remote server every N minutes
- **Read-optimized**: Instant local reads, async writes to central DB
- **Graceful degradation**: Always functional, network optional

Expand Down Expand Up @@ -82,10 +82,10 @@ Knowledge is a command-line tool that captures technical decisions, learnings, a
│ Background sync
┌─────────────────────────────────────────┐
Odin (Centralized)
Remote Server (Centralized) │
│ - Same stack as local │
│ - Team knowledge repository
│ - Exposed via Tailscale mesh
│ - Team/shared knowledge repository │
│ - Exposed via VPN, Tailscale, or LAN
└─────────────────────────────────────────┘
```

Expand Down
10 changes: 6 additions & 4 deletions README.md
Original file line number Diff line number Diff line change
@@ -1,5 +1,7 @@
# Knowledge CLI

[![Sentinel Gate](https://github.com/conduit-ui/knowledge/actions/workflows/gate.yml/badge.svg)](https://github.com/conduit-ui/knowledge/actions/workflows/gate.yml)

AI-powered knowledge base with semantic search, Qdrant vector storage, and Ollama intelligence.

## What It Does
Expand Down Expand Up @@ -34,7 +36,7 @@ Embedding Server (sentence-transformers)
Ollama (optional - async auto-tagging via background queue)
Odin Sync (background sync to centralized server)
Remote Sync (optional - background sync to centralized server)
```

No SQLite. No schema migrations. Pure vector storage. Per-project isolation via auto-detected git namespaces.
Expand Down Expand Up @@ -96,7 +98,7 @@ All commands support `--project=<name>` to target a specific project namespace a
| Command | Description |
|---------|-------------|
| `sync` | Bidirectional sync (--push / --pull) |
| `sync:odin` | Background sync to Odin central server |
| `sync:remote` | Background sync to centralized remote server |
| `sync:purge` | Purge sync queue |

### Code Intelligence
Expand Down Expand Up @@ -162,9 +164,9 @@ REDIS_PORT=6379
OLLAMA_HOST=http://localhost:11434
```

### Odin (Production)
### Remote Server (Production)

Uses `docker-compose.odin.yml` with Tailscale networking for centralized knowledge sync.
Uses `docker-compose.remote.yml` to bind services to a specific network interface (e.g. Tailscale, VPN, LAN) for centralized knowledge sync across multiple machines.

## Development

Expand Down
4 changes: 2 additions & 2 deletions ROADMAP.md
Original file line number Diff line number Diff line change
Expand Up @@ -8,8 +8,8 @@ Replaced SQLite entirely with Qdrant-only architecture. No schema migrations, no
### Redis Caching Layer
KnowledgeCacheService provides sub-200ms query responses through aggressive caching of embeddings, search results, and collection stats.

### Odin Background Sync
OdinSyncService syncs knowledge to centralized Odin server. Includes deletion propagation, sync purge, and bidirectional push/pull.
### Remote Background Sync
RemoteSyncService syncs knowledge to a centralized remote server. Includes deletion propagation, sync purge, and bidirectional push/pull. Useful for home labs and teams sharing a knowledge base.

### Entry Metadata & Staleness Detection
EntryMetadataService tracks entry freshness with confidence degradation over time. Superseded marking instead of destructive overwrites.
Expand Down
199 changes: 67 additions & 132 deletions app/Commands/IndexCodeCommand.php
Original file line number Diff line number Diff line change
Expand Up @@ -4,187 +4,122 @@

namespace App\Commands;

use App\Services\CodeIndexerService;
use App\Services\SymbolIndexService;
use LaravelZero\Framework\Commands\Command;

use function Laravel\Prompts\error;
use function Laravel\Prompts\info;
use function Laravel\Prompts\note;
use function Laravel\Prompts\progress;
use function Laravel\Prompts\spin;
use function Laravel\Prompts\table;
use function Laravel\Prompts\warning;

class IndexCodeCommand extends Command
{
protected $signature = 'index-code
{--path=* : Additional paths to index}
{--dry-run : Show what would be indexed without actually indexing}
{--stats : Show indexing statistics only}';
{path? : Path to index (defaults to current directory)}
{--incremental : Only re-index changed files}
{--list : List all indexed repositories}';

protected $description = 'Index code files for semantic search';
protected $description = 'Index code files using tree-sitter AST parsing';

/** @var array<string> */
private const DEFAULT_PATHS = [];

public function handle(CodeIndexerService $indexer): int
public function handle(SymbolIndexService $indexer): int
{
/** @var array<string> $additionalPaths */
$additionalPaths = (array) $this->option('path');
$dryRun = (bool) $this->option('dry-run');
$statsOnly = (bool) $this->option('stats');

$paths = array_unique(array_merge(self::DEFAULT_PATHS, $additionalPaths));

// Filter to existing paths
$validPaths = array_filter($paths, fn ($p): bool => is_dir($p));

if ($validPaths === []) {
error('No valid paths to index.');

return self::FAILURE;
if ((bool) $this->option('list')) {
return $this->listRepos($indexer);
}

info('Code Indexer');
note('Paths to index: '.implode(', ', array_map('basename', $validPaths)));

// Ensure collection exists
if (! $dryRun) {
$created = spin(
fn (): bool => $indexer->ensureCollection(),
'Ensuring code collection exists...'
);
$pathArg = $this->argument('path');
$path = is_string($pathArg) ? $pathArg : getcwd();
$incremental = (bool) $this->option('incremental');

if (! $created) {
error('Failed to create/verify code collection in Qdrant.');
if ($path === false || ! is_dir($path)) {
error("Invalid path: {$path}");

return self::FAILURE;
}
}

// Collect all files first
$files = [];
foreach ($indexer->findFiles($validPaths) as $file) {
$files[] = $file;
return self::FAILURE;
}

$totalFiles = count($files);
info("Found {$totalFiles} files to index.");
info('Symbol Indexer (tree-sitter AST)');
note('Path: '.$path);

if ($statsOnly || $dryRun) {
$this->showStats($files);
/** @var array{success: bool, repo?: string, file_count?: int, symbol_count?: int, languages?: array<string, int>, error?: string, incremental?: bool, changed?: int, new?: int, deleted?: int, warnings?: array<string>} $result */
$result = spin(
fn (): array => $indexer->indexFolder($path, $incremental),
$incremental ? 'Incremental indexing...' : 'Indexing with tree-sitter...'
);

if ($dryRun) {
warning('Dry run - no files were indexed.');
}
if (! $result['success']) {
error($result['error'] ?? 'Indexing failed.');

return self::SUCCESS;
return self::FAILURE;
}

if ($totalFiles === 0) {
warning('No files found to index.');
if (isset($result['message'])) {
info($result['message']);

return self::SUCCESS;
}

// Index files with progress bar
$indexed = 0;
$failed = 0;
$totalChunks = 0;
$errors = [];

$progress = progress(
label: 'Indexing files...',
steps: $totalFiles
);

$progress->start();

foreach ($files as $file) {
$result = $indexer->indexFile($file['path'], $file['repo']);
info('Indexing complete!');

if ($result['success']) {
$indexed++;
$totalChunks += $result['chunks'];
} else {
$failed++;
if (isset($result['error'])) {
$errors[$file['path']] = $result['error'];
}
}
$rows = [
['Repository', $result['repo'] ?? 'unknown'],
];

$progress->advance();
if (isset($result['incremental']) && $result['incremental']) {
$rows[] = ['Changed files', (string) ($result['changed'] ?? 0)];
$rows[] = ['New files', (string) ($result['new'] ?? 0)];
$rows[] = ['Deleted files', (string) ($result['deleted'] ?? 0)];
} else {
$rows[] = ['Files indexed', (string) ($result['file_count'] ?? 0)];
}

$progress->finish();
$rows[] = ['Symbols extracted', (string) ($result['symbol_count'] ?? 0)];

// Show results
info('Indexing complete!');
if (isset($result['languages'])) {
$langStr = implode(', ', array_map(
fn (string $lang, int $count): string => "{$lang}: {$count}",
array_keys($result['languages']),
array_values($result['languages'])
));
$rows[] = ['Languages', $langStr];
}

table(
['Metric', 'Value'],
[
['Files indexed', (string) $indexed],
['Files failed', (string) $failed],
['Total chunks', (string) $totalChunks],
]
);
table(['Metric', 'Value'], $rows);

if ($errors !== [] && count($errors) <= 10) {
warning('Errors:');
foreach ($errors as $path => $err) {
note(" {$path}: {$err}");
if (isset($result['warnings']) && $result['warnings'] !== []) {
foreach (array_slice($result['warnings'], 0, 5) as $warn) {
warning($warn);
}
} elseif ($errors !== []) {
warning(count($errors).' files had errors.');
}

return $failed > 0 && $indexed === 0 ? self::FAILURE : self::SUCCESS;
return self::SUCCESS;
}

/**
* Show statistics about files to be indexed.
*
* @param array<array{path: string, repo: string}> $files
*/
private function showStats(array $files): void
private function listRepos(SymbolIndexService $indexer): int
{
$byRepo = [];
$byLang = [];
$repos = $indexer->listRepos();

foreach ($files as $file) {
$repo = $file['repo'];
$ext = pathinfo($file['path'], PATHINFO_EXTENSION);
$lang = $this->extToLang($ext);
if ($repos === []) {
info('No indexed repositories found.');

$byRepo[$repo] = ($byRepo[$repo] ?? 0) + 1;
$byLang[$lang] = ($byLang[$lang] ?? 0) + 1;
return self::SUCCESS;
}

info('Files by repository:');
$repoRows = [];
foreach ($byRepo as $repo => $count) {
$repoRows[] = [$repo, (string) $count];
$rows = [];
foreach ($repos as $repo) {
$langs = implode(', ', array_keys($repo['languages']));
$rows[] = [
$repo['repo'],
(string) $repo['file_count'],
(string) $repo['symbol_count'],
$langs,
$repo['indexed_at'],
];
}
table(['Repository', 'Files'], $repoRows);

info('Files by language:');
$langRows = [];
foreach ($byLang as $lang => $count) {
$langRows[] = [$lang, (string) $count];
}
table(['Language', 'Files'], $langRows);
}
table(['Repository', 'Files', 'Symbols', 'Languages', 'Indexed At'], $rows);

private function extToLang(string $ext): string
{
return match (strtolower($ext)) {
'php' => 'PHP',
'py' => 'Python',
'js', 'jsx' => 'JavaScript',
'ts', 'tsx' => 'TypeScript',
'vue' => 'Vue',
default => 'Other',
};
return self::SUCCESS;
}
}
Loading
Loading