Skip to content
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
1 change: 1 addition & 0 deletions docs/src/SUMMARY.md
Original file line number Diff line number Diff line change
Expand Up @@ -17,6 +17,7 @@
- [invariants](./cmd-invariants.md)
- [query](./cmd-query.md)
- [batch](./cmd-batch.md)
- [cascade](./cmd-cascade.md)
- [Profiles](./profiles.md)
- [Input Formats](./formats.md)
- [The Engine](./engine.md)
Expand Down
142 changes: 142 additions & 0 deletions docs/src/cmd-cascade.md
Original file line number Diff line number Diff line change
@@ -0,0 +1,142 @@
# cascade

`cascade` detects temporal cause-effect chains in event data. Given a stream of timestamped events grouped by entity, it identifies sequences where one event type triggers another — and measures how reliably that pattern holds.

Where `anomalies` finds single-record outliers, `cascade` finds multi-record temporal patterns: event A happens to entity X, then event B follows within a window.

---

## Usage

```bash
vajra cascade <input> [flags]
```

**Arguments:**

| Argument | Description |
|---|---|
| `<input>` | Path to a JSON/NDJSON file, `-` for stdin, or an HTTP URL |

**Flags:**

| Flag | Description | Default |
|---|---|---|
| `--entity-field <path>` | JSONPath to the entity identifier (e.g., `'$.author'`) | required |
| `--time-field <path>` | JSONPath to the timestamp field (e.g., `'$.date'`) | required |
| `--event-field <path>` | JSONPath to the event type field (e.g., `'$.type'`) | required |
| `--response-values <vals>` | Comma-separated list of event values that count as responses (e.g., `fix,revert`) | required |
| `--format <fmt>` | Output format: `text`, `json`, `markdown`, `compact-ai` | `text` |
| `--input-format <fmt>` | Override auto-detected input format | auto |
| `--quiet` | Suppress progress output | off |

---

## What It Reports

### Cascade Rate

The fraction of trigger events that are followed by a response event from the same entity within the detection window. A high cascade rate means the cause-effect pattern is reliable.

### Self-Fix Rate

The fraction of cascades where the same entity that caused the trigger also produced the response. Measures whether entities clean up their own problems.

### Hot Entities

Entities that appear disproportionately in cascade chains. These are the nexus points — the authors, services, or components that most frequently participate in cause-and-effect sequences.

### Cascade Chains

The full chain detail: trigger event, response event, entity, timestamps, and time delta between cause and effect.

---

## Algorithm

O(n log n). Records are grouped by entity using a BTreeMap (ordered map), sorted by timestamp within each group, then scanned linearly to detect trigger-response pairs. The BTreeMap ensures deterministic iteration order regardless of input ordering.

---

## Example: Commit Cascade Analysis

```bash
vajra cascade commits.ndjson \
--entity-field '$.author' \
--time-field '$.date' \
--event-field '$.type' \
--response-values 'fix,revert'
```

```text
=== Cascade Report ===
Records: 1,247
Entities: 34
Trigger events: 312
Response events: 89

Cascade rate: 0.285 (89 of 312 triggers followed by a response)
Self-fix rate: 0.742 (66 of 89 responses by the same entity)

Hot entities:
alice 23 cascades (25.8%)
bob 14 cascades (15.7%)
charlie 9 cascades (10.1%)

Cascade chains (top 5 by frequency):
bug -> fix 62 occurrences, median delta: 2.3 days
bug -> revert 18 occurrences, median delta: 0.4 days
regression -> fix 9 occurrences, median delta: 4.1 days
```

---

## Example: JSON Output

```bash
vajra cascade commits.ndjson \
--entity-field '$.author' \
--time-field '$.date' \
--event-field '$.type' \
--response-values 'fix,revert' \
--format json
```

```json
{
"records": 1247,
"entities": 34,
"trigger_events": 312,
"response_events": 89,
"cascade_rate": 0.285,
"self_fix_rate": 0.742,
"hot_entities": [
{"entity": "alice", "cascades": 23, "fraction": 0.258},
{"entity": "bob", "cascades": 14, "fraction": 0.157},
{"entity": "charlie", "cascades": 9, "fraction": 0.101}
],
"chains": [
{"trigger": "bug", "response": "fix", "count": 62, "median_delta_days": 2.3},
{"trigger": "bug", "response": "revert", "count": 18, "median_delta_days": 0.4},
{"trigger": "regression", "response": "fix", "count": 9, "median_delta_days": 4.1}
]
}
```

---

## When to Use It

- **Incident response analysis.** Which errors lead to fixes, and how quickly? Which lead to reverts?
- **Developer workflow.** Who introduces bugs and who fixes them? Is there a self-fix pattern?
- **Service dependency.** Event A in service X triggers event B in service Y — cascade reveals the coupling.
- **Repository health.** Measure how reliably bugs get resolved and how long the resolution takes.

---

## Pairs Well With

- [`stats`](./cmd-stats.md) — statistical profile of the event fields before cascade analysis
- [`anomalies`](./cmd-anomalies.md) — unusual cascade chains (an entity that never self-fixes) are anomaly candidates
- [`invariants`](./cmd-invariants.md) — cascade patterns are temporal invariants; invariants discovers structural ones
- [`essence`](./cmd-essence.md) — cascade metrics feed into essence generation for project health assessments
32 changes: 32 additions & 0 deletions docs/src/cmd-drift.md
Original file line number Diff line number Diff line change
Expand Up @@ -28,6 +28,38 @@ vajra drift <baseline> <candidate> [flags]
| `--input-format <fmt>` | Override auto-detected input format | auto |
| `--redact` | Apply built-in redaction before output | off |
| `--quiet` | Suppress progress output | off |
| `--group-by <path>` | JSONPath for population-level comparison (e.g., `'$.author_type'`) | off |

---

## Population-Level Comparison

When `--group-by` is specified, `drift` partitions records by the field value and computes pairwise drift between all groups. Instead of comparing two documents, you compare two (or more) subpopulations within the same dataset.

```bash
vajra drift prs.ndjson --group-by '$.author_type'
```

```text
Drift Report (grouped by $.author_type)
Groups: bot (412 records), human (835 records)

Pairwise drift: bot vs human
Structural similarity: 0.91 (Jaccard)

Distribution shifts:
$.files_changed JSD: 0.42 (high)
bot: median 1.0, p95 3.0
human: median 4.0, p95 18.0

$.review_comments JSD: 0.38 (moderate)
bot: median 0.0, p95 1.0
human: median 2.0, p95 8.0

Overall severity: HIGH (significant distributional divergence)
```

This is useful for comparing behavioral subgroups — bot vs. human PRs, different teams, production vs. staging, before vs. after a policy change — without needing separate files.

---

Expand Down
37 changes: 37 additions & 0 deletions docs/src/cmd-stats.md
Original file line number Diff line number Diff line change
Expand Up @@ -27,6 +27,43 @@ vajra stats <input> [flags]
| `--streaming` | Force streaming mode (sketch-based approximations) | off |
| `--redact` | Apply built-in redaction before output | off |
| `--quiet` | Suppress progress output | off |
| `--window <period>` | Temporal windowing: `month`, `week`, or `day` | off |
| `--time-field <path>` | JSONPath to timestamp field (e.g., `'$.date'`). Auto-detected if omitted. | auto |

---

## Temporal Windowing

When `--window` is specified, `stats` partitions records by time period and computes per-window statistics. Cross-window trend lines are included in the output, showing how distributions shift over time.

The `--time-field` flag tells Vajra which field contains the timestamp. If omitted, Vajra auto-detects by scanning for fields with date/time patterns (ISO 8601, Unix timestamps, common date formats).

```bash
vajra stats commits.ndjson --window month --time-field '$.date'
```

```text
=== Statistical Summary (windowed: month) ===
Document: commits.ndjson (1,247 records, 8 paths)

--- Window: 2026-01 (312 records) ---
$.files_changed
Mean: 4.2 Median: 3.0 p95: 12.0

--- Window: 2026-02 (298 records) ---
$.files_changed
Mean: 5.1 Median: 4.0 p95: 15.0

--- Window: 2026-03 (337 records) ---
$.files_changed
Mean: 6.8 Median: 5.0 p95: 19.0

--- Cross-Window Trends ---
$.files_changed mean: 4.2 -> 5.1 -> 6.8 (upward, +62% over 3 months)
$.type "fix" share: 0.18 -> 0.24 -> 0.31 (increasing)
```

Windowing works with any multi-record input: NDJSON, CSV, multi-document YAML, or directories.

---

Expand Down
88 changes: 81 additions & 7 deletions docs/src/formats.md
Original file line number Diff line number Diff line change
Expand Up @@ -19,6 +19,9 @@ Vajra reads more than JSON. It reads anything that can be interpreted as structu
| Zstd | `.zst`, `.json.zst` | Zstd magic bytes | Decompressed transparently. Inner format auto-detected. |
| HTTP URL | `http://`, `https://` | URL scheme prefix | Fetched via blocking HTTP GET. Response body auto-detected. |
| Source Code | `.rs`, `.py`, `.js`, `.ts`, `.go`, `.java`, `.c`, `.cpp`, `.rb` | File extension matches known language | Parsed via tree-sitter into AST. Requires `vajra-source` feature. |
| Git Repository | (directory) | Directory contains `.git/` | Reads commit history directly. See flags below. |
| V8 CPU Profile | `.cpuprofile` | File extension | Parses V8 `.cpuprofile` JSON into analyzable structure. |
| strace Summary | — | Content contains `% time` header | Parses `strace -c` summary output into structured records. |
| Stdin | `-` | Explicit `-` argument | Content auto-detected from first bytes. |

---
Expand All @@ -27,9 +30,9 @@ Vajra reads more than JSON. It reads anything that can be interpreted as structu

When no `--input-format` is specified, Vajra detects the format in this order:

1. **Check the argument.** If it is `-`, read from stdin. If it starts with `http://` or `https://`, fetch via HTTP.
1. **Check the argument.** If it is `-`, read from stdin. If it starts with `http://` or `https://`, fetch via HTTP. If it is a directory containing `.git/`, treat as a git repository.

2. **Check the extension.** `.json` -> JSON. `.ndjson`/`.jsonl` -> NDJSON. `.yaml`/`.yml` -> YAML. `.csv` -> CSV. `.tsv` -> TSV. `.md` -> Markdown. `.pdf` -> PDF. `.rs`/`.py`/`.js`/`.go`/etc. -> Source Code (via tree-sitter).
2. **Check the extension.** `.json` -> JSON. `.ndjson`/`.jsonl` -> NDJSON. `.yaml`/`.yml` -> YAML. `.csv` -> CSV. `.tsv` -> TSV. `.md` -> Markdown. `.pdf` -> PDF. `.cpuprofile` -> V8 CPU Profile. `.rs`/`.py`/`.js`/`.go`/etc. -> Source Code (via tree-sitter).

3. **Check for compression.** If the extension is `.gz` or `.zst`, decompress and re-detect the inner format from the next extension (e.g., `.json.gz` -> decompress -> JSON).

Expand All @@ -39,6 +42,7 @@ When no `--input-format` is specified, Vajra detects the format in this order:
- Starts with `---` or matches `key: value` pattern -> YAML
- Consistent comma-separated columns -> CSV
- PDF magic bytes (`%PDF`) -> PDF
- Contains `% time` column header -> strace summary

5. **Fall back to JSON.** If nothing else matches, attempt JSON parsing.

Expand Down Expand Up @@ -80,7 +84,7 @@ Each line is an independent JSON document. Natural format for logs, event stream
vajra anomalies claims.ndjson
```

Every line becomes a separate document in the analysis. Commands like `anomalies` and `invariants` treat the lines as a population.
NDJSON records are aggregated into a single array for analysis. Commands like `stats`, `anomalies`, `invariants`, and `essence` compute across all records as a unified population.

Example input:

Expand Down Expand Up @@ -214,6 +218,78 @@ vajra inspect code.txt --input-format source --lang python # override format +

Source code analysis requires the `vajra-source` crate (included by default). The companion `vajra-domain-source` plugin adds recognizers for naming conventions (snake_case, camelCase, PascalCase) and code structure relationships.

#### Semantic Paths

The `--semantic-paths` flag maps tree-sitter node kinds to human-readable labels in the output. Instead of raw AST node names like `function_item` or `impl_item`, you see `function` and `implementation`.

```bash
vajra inspect main.rs --semantic-paths
```

Without `--semantic-paths`:

```text
$.program.function_item[0].identifier "process_record"
$.program.function_item[0].parameters.parameter[0] "record: &Record"
$.program.impl_item[0].identifier "Pipeline"
```

With `--semantic-paths`:

```text
$.program.function[0].name "process_record"
$.program.function[0].parameters.param[0] "record: &Record"
$.program.implementation[0].name "Pipeline"
```

Covers 9 languages: Rust, Python, JavaScript, TypeScript, Go, Java, C, C++, and Ruby.

### Git Repository

When the input is a directory containing a `.git/` subdirectory, Vajra reads the commit history directly — no export step required.

```bash
vajra stats ./my-repo
vajra cascade ./my-repo --entity-field '$.author' --time-field '$.date' --event-field '$.type' --response-values 'fix,revert'
```

Each commit becomes a JSON record with fields like `author`, `date`, `message`, `files_changed`, and `insertions`/`deletions`.

**Flags:**

| Flag | Description | Default |
|---|---|---|
| `--git-limit <N>` | Maximum number of commits to read | 500 |
| `--git-branch <branch>` | Branch to read from | current HEAD |

```bash
vajra stats ./my-repo --git-limit 1000 --git-branch main
```

Auto-detection is based on the presence of `.git/` in the input directory. To override, use `--input-format git`.

### V8 CPU Profile

Vajra parses `.cpuprofile` files produced by V8-based tools (Chrome DevTools, Node.js `--prof`). The profile's call tree is converted to a flat array of records with function name, source location, hit count, and self/total time.

```bash
vajra stats profile.cpuprofile
vajra anomalies profile.cpuprofile
```

Auto-detected by the `.cpuprofile` extension.

### strace Summary

Vajra parses the summary table produced by `strace -c`. Each syscall row becomes a record with fields for time percentage, seconds, calls, errors, and syscall name.

```bash
strace -c ls 2>&1 | vajra stats -
vajra stats strace_output.txt --input-format strace
```

Auto-detected when content contains the `% time` column header characteristic of `strace -c` output.

---

### Compressed Files (Gzip, Zstd)
Expand Down Expand Up @@ -257,13 +333,11 @@ zcat claims.json.gz | vajra inspect -

## Multi-Document Formats

NDJSON and multi-document YAML naturally contain multiple documents. When fed to single-document commands (`inspect`, `stats`, `fingerprint`), Vajra analyzes the first document. When fed to multi-document commands (`anomalies`, `invariants`, `batch`), all documents are analyzed as a population.

To explicitly analyze all documents from a multi-document format:
NDJSON and multi-document YAML naturally contain multiple documents. NDJSON records are now aggregated into a single array, so all commands — including `stats`, `anomalies`, `invariants`, and `essence` — compute across all records as a unified population.

```bash
vajra anomalies claims.ndjson # analyzes all lines as a batch
vajra stats claims.ndjson # analyzes the first line only
vajra stats claims.ndjson # computes stats across all records
```

---
Expand Down
Loading
Loading