diff --git a/BENCHMARKS.md b/BENCHMARKS.md new file mode 100644 index 000000000..6c1a6d331 --- /dev/null +++ b/BENCHMARKS.md @@ -0,0 +1,187 @@ +# graphify Benchmarks + +How graphify performs as conversational long-term memory and as a +code-intelligence layer, measured on an open harness with competing systems run +under identical conditions (same model, same budgets, same grader). + +Last updated: 2026-07-05. + +## Summary + +graphify's deterministic graph plus hybrid retrieval has the best retrieval +recall on LOCOMO of any system tested, the best LOCOMO QA accuracy per dollar, +ties for the best LongMemEval score, and builds its index with zero LLM credits. +Every system was run on the same harness with one shared model (Kimi K2.6), +identical budgets, and a judge blind-validated against a second independent judge +(90.6% agreement, Cohen's kappa 0.81). + +Highlights: +- LOCOMO retrieval recall@10 of 0.497, about 10x mem0 (0.048) and above BM25 (0.362). +- LOCOMO QA accuracy of 45.3%: +18 points over mem0, +14 over BM25, and within + 4.4 points of supermemory at about a tenth of supermemory's ingest cost. +- LongMemEval-S of 76%, tied for best with dense RAG. +- Zero LLM credits to build the graph, and about 11x cheaper memory ingest than + supermemory ($1.40 vs $15.67). + +## Results at a glance + +| Suite | Dataset (n) | Metric | graphify | Field | +|---|---|---|---|---| +| Memory | LOCOMO (300) | QA accuracy | 45.3% | supermemory 49.7% (11x ingest cost), bm25 31.3%, mem0 27.3% | +| Memory | LOCOMO (300) | recall@10 | 0.497 | bm25 0.362, mem0 0.048 | +| Memory | LongMemEval-S (50) | QA accuracy | 76% | dense RAG 76%, hybrid 74%, mem0 70% | +| Cost | LOCOMO ingest | USD | ~$1.40 | supermemory $15.67, mem0 $3.48 | +| Cost | graph build | LLM credits | $0 | n/a | + +## Harness + +graphify's own harness. Competing systems (mem0, supermemory) are run as +adapters inside it, so every system sees the same model, token budget, and +grader. + +``` +ingest -> index -> search -> answer -> grade +(build) (store) (retrieve) (Kimi K2.6) (key-fact coverage) +``` + +- Memory suite (`memory/`): graphify's graph retrieval vs dedicated memory + systems (mem0, supermemory) and classic baselines (BM25, dense RAG, + hybrid RRF). mem0 and supermemory run self-hosted as adapters, wired through + a proxy so their LLM calls also use Kimi K2.6. +- Code suite (`crosstool/`): a fixed coding agent (Claude Opus 4.8, at most 14 + turns, a grep/read/list floor plus one code-intelligence tool) answers graded + questions on ERPNext, a roughly 1M-LOC production repo + ([frappe/erpnext](https://github.com/frappe/erpnext)), with a temporal + sub-suite of 689 weekly AST checkpoints from 2011 to 2026. + +## Datasets + +- LOCOMO (`locomo10.json`, n=300): multi-session conversational QA. +- LongMemEval-S (n=50, English subset): long-horizon conversational memory. +- ERPNext: a large real-world Python codebase for code intelligence. + +LOCOMO and LongMemEval are the same academic datasets other memory systems +report on, so results are cross-referenceable. Datasets are not redistributed; +the harness documents the expected local layout. + +## Judge and grading + +Answers are graded by Kimi K2.6 against a gold set of atomic key facts a correct +answer must contain: + +``` +coverage = (covered + 0.5 * partial) / total +``` + +Every verdict cites a verbatim quote from the answer, so grades are auditable +rather than one opaque score. + +Judge validation: the judge was blind-validated against a second, independent +judge on a sampled set at 90.6% agreement, Cohen's kappa 0.81 (substantial +agreement). Most published memory benchmarks disclose no judge validation at +all; we publish ours so the grading itself can be audited. + +## Fairness rules + +- One model for every LLM role: Kimi K2.6 via Moonshot. +- One shared local embedder where the system allows it: BGE-m3 (1024-d, + multilingual). +- Identical token budgets. Every run writes a spend ledger and respects + `--max-spend`. +- Graphs build AST-only with no LLM (an unset API key produces zero credits); + embeddings use a local deterministic model. + +## Results: conversational memory + +### LOCOMO (n=300) + +Sorted by recall@10. + +| System | QA accuracy | recall@10 | Ingest cost | +|---|---|---|---| +| **graphify** (graph-expand) | **45.3%** | **0.497** | ~$1.40 | +| hybrid RRF | 43.3% | 0.493 | $0 (shared index) | +| graphify (SurrealDB engine) | 43.3% | 0.485 | $0 (shared index) | +| dense RAG | 41.3% | 0.439 | $0 (shared index) | +| BM25 | 31.3% | 0.362 | $0 (shared index) | +| supermemory | 49.7% | 0.149* | $15.67 | +| mem0 | 27.3% | 0.048 | $3.48 | + +Bold marks graphify's primary configuration, not the column maximum. Baselines +retrieve from the same harness-built index, so they incur no separate ingest +cost. + +`*` Retrieval-recall is embedder-confounded: supermemory's self-host locks in +its own 768-d English-only embedder rather than the shared BGE-m3. The +QA-accuracy axis (a shared Kimi reader and judge over each system's hits) is the +clean comparison. + +Reading: supermemory scores a few points higher on raw QA, but at about 11x the +ingest cost ($15.67 vs $1.40) and with about 3x worse retrieval recall. graphify +has the best retrieval recall on LOCOMO of any system tested, the best QA of the +systems on the shared embedder, and does it for about a tenth of supermemory's +cost. It retrieves the right memory about 10x more often than mem0 and answers ++18 points more accurately. A seed-only ablation (no graph expansion) still +scores 42.7% at $1.40 ingest, so most of the accuracy holds at the cheapest +setting. + +### LongMemEval-S (n=50) + +| System | QA accuracy | recall@10 | +|---|---|---| +| **graphify** (graph-expand) | **76%** | **0.844** | +| dense RAG | 76% | 0.848 | +| graphify (SurrealDB engine) | 74% | 0.833 | +| hybrid RRF | 74% | 0.822 | +| BM25 | 70% | 0.710 | +| mem0 | 70% | 0.344 | + +graphify ties dense RAG for the best QA accuracy (76%); dense RAG edges it on +recall (0.848 vs 0.844). Both retrieve far more than mem0 (recall 0.344). + +## Results: code intelligence + +On ERPNext (a roughly 1M-LOC production repo), giving a fixed coding agent one +graphify tool lifts key-fact coverage across the graded question set (n=6) from +70.8% (a grep and read baseline) to 82.0%, at about 140K tokens per query. +graphify pays for itself in accuracy against searching raw files, and avoids the +context-stuffing anti-pattern of packing the whole repo into every turn (which +costs roughly 20x the tokens for lower coverage). + +## Results: temporal (15 years of ERPNext) + +689 weekly AST checkpoints, 2011 to 2026, built deterministically with no LLM. + +| Checkpoint | Nodes | Edges | Files | +|---|---|---|---| +| 2011-06-08 | 3,069 | 2,900 | 1,032 | +| 2026-06-24 | 22,620 | 48,710 | 3,758 | + +The graph grows about 7x in nodes and 17x in edges across the span. As the +codebase grows, plain lexical retrieval finds less of the answer while graph and +semantic retrieval scale with it, and the AST extraction itself stays stable. + +## Cost and token economics + +- Graph construction costs zero LLM credits. graphify extracts with tree-sitter + (deterministic, about 40 languages) and a local embedder, so building the + index uses no API tokens. Most memory and semantic-retrieval systems pay a + per-document LLM ingest cost. +- Memory ingest is about 11x cheaper: graphify's LOCOMO ingest runs around + $1.40 against supermemory's $15.67. +- Every number here is backed by a per-run spend ledger in the harness output. + +## Reproducing + +Set `MOONSHOT_API_KEY`. Datasets are fetched to the local layout documented in +the harness. Each run respects `--max-spend` and writes a spend ledger. + +```bash +# Memory (LOCOMO). This invokes the SurrealDB-engine row (43.3%); the +# graph-expand headline (45.3%) is a separate adapter in the same harness. +python memory/runner.py --phase 3 --split locomo --n 300 \ + --adapters graphify_v1_surreal --cn natural --workers 6 --max-spend 15 + +# Code cross-tool (ERPNext) +python crosstool/run.py --repo erpnext --max-spend +``` diff --git a/CHANGELOG.md b/CHANGELOG.md index f01b4fe4c..267e86228 100644 --- a/CHANGELOG.md +++ b/CHANGELOG.md @@ -4,6 +4,48 @@ Full release notes with details on each version: [GitHub Releases](https://githu ## Unreleased +- Feat: Sigma.js + graphology fallback for large `graph.html` views, plus batched/reconciled community labeling at scale. vis-network's live client-side forceAtlas2 physics simulation is genuinely slow/laggy past a few hundred nodes regardless of hardware — verified in production on a 1083-community monorepo graph (apa, 2026-07-03). Step 6 of the skill now generates a separate WebGL Sigma.js render (`graph_sigma.html`) with the layout precomputed offline in Python (`nx.forceatlas2_layout`, no client-side physics) once the view exceeds ~300 nodes; the full recipe lives in the new `references/sigma-viz.md`, shipped to every split-platform skill variant. Step 5's community labeling also gains a batched-subagent flow for 100+ communities (sample each community's top-degree nodes, dispatch one `general-purpose` subagent per ~15-20-community batch in parallel) plus a required cross-batch label-reconciliation pass — a naive single-pass label run on the same graph left 338 of 1083 communities (31%) sharing a label with another before reconciliation. + The Sigma view also fixes a node-sizing bug (`autoRescale` must be disabled alongside `itemSizesReference: "positions"`, per sigma's own docs — otherwise sizes stay anchored to the pre-rescale coordinate frame and don't track zoom) and recalibrates node size against the layout's own median nearest-neighbor distance instead of a fixed constant, so communities stay legible instead of overlapping into a blob as node count grows. It now also surfaces each community's dominant content kind as an icon (code/document/paper/image/rationale/concept, via `@sigma/node-image`'s `NodePictogramProgram`) and dominant top-level module as color, plus a left-side panel to filter by entity kind and by relation type (grouped into calls/structure/imports/references/documentation-and-concepts/groups, covering both code and non-code relations) and a clickable module legend to isolate one part of the codebase at a time. +- Fix: `--update`-style section writes to `CLAUDE.md`/`AGENTS.md` no longer corrupt or drop content (#1688, thanks @bdfinst). `_replace_or_append_section` located its managed block by substring (`marker in content`) and `next(... if marker in line)`, so a heading that appeared as a substring of another line (or duplicate headings) matched the wrong offset and the rewrite could truncate the file. It now matches the section heading exactly (`line.strip() == marker`), appends when absent, and prefers the last exact match when several exist, so unrelated content is preserved. +- Fix: token estimation no longer crashes on files containing tiktoken special-token text like `<|endoftext|>` (#1685, thanks @Kyzcreig). `_TOKENIZER.encode(content)` raises `ValueError` by default when the text contains a special token, which aborted packing on docs/corpora that merely mention these strings. Both `encode` sites now pass `disallowed_special=()` so such text is tokenized as ordinary bytes. +- Fix: the Ollama backend no longer multiplies a hang by the retry count (#1686, thanks @Kyzcreig). A stalled local model would wedge for `timeout * (max_retries + 1)`, which with the default 6 retries turned one long stall into a very long one. Ollama now defaults to zero client-side retries (a local model that stalls will not un-stall on retry); set `GRAPHIFY_MAX_RETRIES` to opt back in. Other backends are unchanged. Note: the underlying stall is non-deterministic and driven by the model server, so this bounds the wait rather than eliminating the hang. +- Fix: a truncated or slightly malformed community-labeling reply no longer discards the whole batch (#1690, thanks @vdgbcrypto). `_parse_label_response` now salvages the complete `"id": "name"` pairs from a reply that failed a strict `json.loads` (e.g. a reply truncated mid-object), raising only when no pairs can be recovered. The per-batch token budget was also raised (`256 + 48*n`, was `64 + 24*n`) to give models that prepend a short preamble enough headroom to finish the JSON. The exact provider truncation in the report could not be reproduced without a live key; the parser and budget fixes address the mechanism. +- Fix: cluster-only mode now reports the real token cost of community labeling instead of a hardcoded zero (#1694, thanks @sub4biz). The labeling LLM calls were never accounted for, so `GRAPH_REPORT.md`'s "Token cost" line always read `0 input · 0 output` in cluster-only runs. `_call_llm` now accumulates per-response usage into an optional accumulator that is threaded through the labeling path and surfaced in the report. Backends that do not return usage (the Claude Code CLI) still contribute nothing, which is honest rather than estimated. + +## 0.9.7 (2026-07-06) + +- Fix: Java standard-library types are no longer emitted as `references` noise (#1603, thanks @NydiaChung). A `_JAVA_BUILTIN_TYPES` skip list now suppresses ubiquitous `java.lang`/`java.util`/`java.io`/`java.time`/`java.math`/`java.nio.file` type names (`String`, `List`, `Map`, `Optional`, `Integer`, `Exception`, ...) at the type-ref walker; they never resolve to a project node, so edges to them were pure noise (mirrors `_GO_PREDECLARED_TYPES`/`_PYTHON_ANNOTATION_NOISE`). Nested user-type generic arguments still resolve: `List` drops the `List` edge but keeps `Item`. +- Feat: added a `pascal` optional extra for AST-quality Pascal/Delphi extraction (#1616, thanks @vinicius-l-machado). `extract_pascal` already used tree-sitter-pascal when present (with a regex fallback), but the grammar was never declared in the package metadata, so the AST path never ran out of the box. `uv tool install "graphifyy[pascal]"` now opts into it (also included in `[all]`); tree-sitter-pascal ships prebuilt wheels for every platform, so no C toolchain is needed. On a mid-size Delphi codebase the AST path yields notably more accurate `calls`/`inherits` edges than the regex fallback. +- Feat: JS/TS rationale comments and ADR/RFC citations are now extracted (#1599, thanks @niltonmourafilho-arch). Python already turned `# NOTE:`/`# WHY:`/`# HACK:` comments and docstrings into `rationale` nodes, but JS/TS comments were discarded. `extract_js` now runs the same post-pass: `// NOTE:`-style and block-comment rationale become `rationale` nodes with `rationale_for` edges, and `ADR-NNNN` / `RFC NNNN` citations become `doc_ref` nodes with `cites` edges from the file, closing the code-to-design-doc gap in mixed corpora at zero LLM cost (pure line scan). Files with no such comments are unaffected. +- Fix: extensionless executables with a shebang (CLI entry points like `devctl`, `manage`) are now extracted (#1683, thanks @Stashub). `detect` already classified a `#!/usr/bin/env bash`/`python`/`node`/... file as code, but `_get_extractor` dispatched on the suffix alone and returned nothing, so the file was silently dropped from the graph and its doc-referenced symbols stayed dangling stubs. Extensionless files now resolve their extractor from the shebang interpreter (`_SHEBANG_DISPATCH`), mirroring detection. Only interpreters with a real extractor are mapped (python, bash-family, node, ruby, lua, php, julia); others (perl, fish, Rscript) stay skipped rather than mis-parsed by a wrong grammar. +- Feat: Ruby `include`/`extend`/`prepend ` now emits a `mixes_in` edge to the module (#1668, thanks @krishnateja7). Concerns/mixins are the composition mechanism in Rails, but they produced no edges, so the blast radius of editing a shared concern was invisible to `affected`. A constant-argument mixin inside a class or module body now resolves to the module node (reusing the #1634 candidate logic and the #1640 module nodes, under the single-owner guard) and emits `Class --mixes_in--> Module`, which `affected` already traverses. `extend self` and non-constant arguments are skipped; an ambiguous or undefined module produces no edge. +- Feat: `affected ` now reaches callers that bind to the class's method nodes (#1669, thanks @krishnateja7). Since #1634 binds `Service.call` precisely to the `def self.call` method node, a class-level `affected` query missed those callers because `method`/`contains` are (correctly) not general-traversal relations. The reverse walk now seeds from the root's own member nodes (one `method`/`contains` hop outward) so method-bound callers are reachable from the class, with no change to the general traversal (no forward noise) and the member nodes themselves are not reported as hits. +- Fix: capitalized/mixed-case file extensions are no longer silently skipped (#1671, thanks @raman118). `collect_files` and `_get_extractor` matched suffixes case-sensitively, so `App.PY`, `script.JS`, `Lib.Ts`, etc. fell through and were never extracted. Suffix matching now falls back to the lowercased form for both file discovery and extractor dispatch (including `.blade.php`); an unsupported extension like `.xyz` is still skipped. +- Fix: the virtual PostgreSQL `source_file` URI no longer gets backslash-mangled on Windows (#1672, thanks @raman118). `introspect_postgres` built the synthetic `postgresql://host/db` path with `Path`, which rewrites `/` to `\` on Windows; it now uses `PurePosixPath` so the URI stays forward-slashed on every platform. +- Fix: a deferred `import(...)` no longer manufactures a phantom file cycle (#1241, thanks @Synvoya). Dynamic imports are real dependencies but not static ones, so two files that reference each other via one static import plus one dynamic import were reported as a circular dependency. The dynamic-import edge stays in the graph (marked `deferred`) but is excluded from `find_import_cycles`. +- Fix: an extractable source file that produces zero nodes is no longer cached, and is surfaced with a warning (#1666, thanks @krishnateja7). Every supported file yields at least a file node, so a zero-node result is anomalous (a transient batch/parallel hiccup). Caching it made the empty byte-stable across runs and silently blinded `affected`/`explain` to and through the file. The cache write is now skipped for a zero-node result so a rerun self-heals, and `extract` warns when an accepted source file lands in the graph with no nodes. This addresses the persistence and the silent blindness; if the underlying zero-node extraction still reproduces on a specific corpus, the warning now makes it visible to report. +- Fix: the Windows skill variant now declares `name: graphify` instead of `name: graphify-windows` (#1635, thanks @ray8875). `graphify install --platform windows` writes the variant to `~/.claude/skills/graphify/SKILL.md`, but Claude Code requires the skill folder name to equal the frontmatter `name`, so the `-windows` suffix broke discovery/validation. The variant suffix is a packaging detail, not part of the skill's identity. +- Fix: the OpenCode plugin joins its reminder to the user's command with `;` instead of `&&` (#1646, thanks @gonaik). Windows PowerShell 5.1 rejects `&&` as a statement separator (`not a valid statement separator`), so the first bash command of every OpenCode session on Windows failed. `;` works in PowerShell 5.1, Bash, and POSIX shells. (Both the OpenCode and Kilo plugin templates are fixed.) +- Fix: the `GRAPH_REPORT.md` "Import Cycles" section is now emitted only when the graph contains code (#1657, thanks @Ns2384-star). On a documents-only corpus there are no imports, so the section was pure noise ("None detected") on every run; it is now conditioned on code nodes or import edges being present. (The same report also confirms the mojibake and stdout-encoding items in that issue are already addressed on the current branch: manifest.json and `GRAPH_REPORT.md` are written UTF-8, and the CLI reconfigures stdout/stderr to UTF-8 with `errors="replace"`.) +- Fix: a modified `.docx`/`.xlsx` now re-enters `--update` (#1649, thanks @Ns2384-star). `detect_incremental` tracks the converted markdown sidecar, and `convert_office_file` early-returned whenever the sidecar already existed — so an Office source edited after its first conversion never updated its sidecar and was reported "unchanged" forever, freezing the graph on a living docs corpus. The sidecar is now re-converted when the source is newer than it (which bumps the sidecar's mtime/content so the incremental hash check picks it up); an unchanged source still skips the rewrite so it never churns (#1226). +- Fix: files whose absolute path exceeds Windows' 260-char limit are now hashed (#1655, thanks @Ns2384-star). `_md5_file`/`save_manifest`/`count_words` used plain `open()`/`stat()`, which the Windows file APIs reject for long paths unless prefixed with the extended-length marker `\\?\` — so deeply-nested files (accented, deep folders) never hashed, their manifest entry never stabilized, and `detect_incremental` re-flagged them as changed on every run. Change-detection I/O now prefixes long absolute paths on win32 (mirroring the normalization `cache.py` already applied to cache keys). No-op on other platforms. +- Perf: word counts are cached against each file's stat signature (#1656, thanks @Ns2384-star). `detect()` counted words in every PDF/docx/text file to size the corpus, re-opening and re-parsing every binary on each run — minutes on a large docs corpus even when only a few files changed. Counts are now memoized in the existing content-hash stat index (keyed by size + mtime), so an unchanged file is parsed once and read from the index thereafter; incremental detection drops from O(corpus) parsing to O(changed). +- Fix: a JS/TS call with no local definition and no import no longer binds to a same-named export in an unrelated package (#1659, thanks @leonaburime-ucla). When a callee had exactly one same-named definition repo-wide, the cross-file resolver emitted a `calls` edge at INFERRED/0.8 even with no import path between the two files. On a monorepo this fabricated dependencies: a 14-package repo showed `platform` and `sidecar` depending on `registry-protocol` purely because it exported generically-named symbols (`*Schema`, etc.) that unresolved calls collapsed onto. JS/TS modules have no implicit cross-module scope, so a cross-file call is real only if the caller imported it — direct JS/TS cross-file `calls` attribution is now gated on import evidence and left unresolved otherwise. Other languages keep the single-candidate resolution (C/C++ headers, Ruby autoload, same-package implicit scope legitimately call across files without an explicit import), and the `indirect_call` path (already INFERRED and callable-gated) is unchanged. As part of the fix, caller→file mapping for import-evidence now uses the raw call's `source_file` string, so a path-resolution/symlink mismatch can no longer spuriously fail evidence and mislabel a real cross-file call. + +## 0.9.6 (2026-07-04) + +- Fix: Ruby plain modules and `Struct.new` / `Class.new` / `Data.define` constant assignments now get container nodes (#1640, thanks @krishnateja7). The extractor only created nodes for `class Foo`, so `module Foo` (utility/`module_function` modules), `Foo = Struct.new(...) do ... end`, `Foo = Class.new(StandardError)`, and `Result = Data.define(...)` produced no node at all — their methods hung off the file via `contains` with dot-less labels, and no edge could ever target them. `module` is now a container type (methods attach via `method` like a class, nested modules included), and a constant assignment whose RHS is one of those factories synthesizes a class node named after the constant, attaches block-defined methods to it, and emits an `inherits` edge for `Class.new(Super)`. Plain constant assignments (`MAX = 100`, `X = Foo.new`) are untouched. +- Fix: Ruby constant-receiver singleton calls now resolve cross-file (#1634, thanks @krishnateja7). `Service.call`, `Model.where`, `SomeJob.perform_async` — the dominant Rails idiom — emitted no `calls` edge, so with Zeitwerk autoloading (no `require`s) a Rails app had essentially no cross-file edges and `affected`/`path` came up empty. `resolve_ruby_member_calls` now handles a capitalized (constant) receiver with any callee: it binds to the class's singleton/instance method when one is owned (`def self.call`, which the extractor indexes), else to the class node itself so inherited/dynamic class methods (ActiveRecord `where`/`find_by`) still give correct blast-radius. Namespaced receivers (`Billing::Processor.call`) resolve by the bare class name. The single-owning-class god-node guard is kept throughout — an ambiguous receiver resolves to nothing rather than a wrong edge. +- Fix: Apex `interface X extends A, B` now emits an `extends` edge per parent (#1645, thanks @Synvoya). The interface regex captured the parent list in group 2, but the handler only read the interface name (group 1), so multiple-inheritance parents were dropped and only the `contains` edge survived. The interface branch now iterates the parent list and resolves each the same way the class branch already does. +- Fix: Kotlin interface delegation (`class Foo : Bar by baz`) now emits the `implements` edge (#1644, thanks @Synvoya). The `by` form wraps the delegated interface in an `explicit_delegation` node, so neither the `constructor_invocation` nor the bare `user_type` branch fired and the edge was silently dropped. The delegation-specifier loop now unwraps `explicit_delegation` to its `user_type` (generic-argument recovery still runs), so idiomatic Kotlin delegation shows up in the graph. +- Fix: a malformed semantic chunk no longer crashes `extract` and discards every successful chunk (#1631, thanks @ssazy). When an LLM returned a well-formed object whose `edges` (or `nodes`/`hyperedges`) array carried a stray non-dict entry — a nested list where an edge object belongs — the AST+semantic merge and the semantic-cache write both called `.get()` per entry and raised `AttributeError: 'list' object has no attribute 'get'`. On a 34-chunk run where 33 succeeded, that meant no `graph.json` was written and the cache write failed too, so a re-run re-extracted everything. `_parse_llm_json` now sanitizes each fragment at the single parse chokepoint (keeping only dict entries and coercing a non-list value to `[]`), so the cache writer, the adaptive-retry merge, and the CLI merge are all protected in one place. +- Fix: an unresolved bare npm import no longer aliases onto an unrelated same-named local file (#1638, thanks @EveX1). `import colors from "tailwindcss/colors"` in a `.tsx` file emitted an `imports_from` edge to the bare id `colors`, and build.py's pre-migration alias index (which registers every local file's bare stem) then remapped it onto an unrelated `backend/utils/colors.py` — a confident (`EXTRACTED`) cross-language phantom edge, and one per `.tsx` file sharing the import. In a real monorepo eight unrelated `.tsx` files all landed on a single Python module. Common package subpaths (`colors`, `utils`, `types`, `config`, `client`) collide this way constantly. The external-import fallback now namespaces its target with the `ref` prefix (the same J-4 convention used for tsconfig `extends`/`$ref` externals), so it can never collapse to a local file/symbol id; the ref-namespaced target has no node, so build drops it as an external reference — the correct outcome for a third-party import. +- Fix: `graph.json` node/edge ordering is now stable run-to-run for document/semantic corpora (#1632, thanks @umeshpsatwe). With a parallel LLM backend, `extract_corpus_parallel` merged chunk results in completion order, so which network call happened to return first reordered the nodes and edges even when the model returned identical content — churning `graph.json` between otherwise-identical runs. Chunks are now merged in deterministic submission order after the pool drains (matching the serial path); the progress callback still fires in completion order so long local runs aren't silent. Note: the semantic content the LLM extracts is itself nondeterministic run-to-run — this fix removes the pipeline's own ordering churn, not the model's variance. + +- Fix: `graphify export obsidian` no longer crashes in `to_canvas` on a dangling community member (#1236 follow-up, thanks @swells808). The original #1236 fix guarded `to_obsidian` but not `to_canvas`, so a community member id with no backing node in the graph still raised `KeyError` while writing `graph.canvas` — after the notes had exported, leaving a partial mirror. `to_canvas` now applies the same dangling-member filter (`m in G and m in node_filenames`) in both the box-sizing and card-layout loops. + +- Feat: TS/JS member calls on a local `new` binding or a type-annotated parameter now resolve (#1630, thanks @DanielC000). `const s = new Svc(); s.doThing()` and a call on a typed param — including inside a returned closure (`(svc: Svc) => () => svc.doThing()`) — now emit `calls` edges to the receiver type's method, so `affected` no longer silently under-reports. Extends the #1316 `this.field` resolver: the per-file type table now also learns local `new` bindings and bare-typed parameters, and `walk_calls` descends into inline/returned closures (attributing their calls to the enclosing function) instead of stopping at the arrow boundary. Resolution keeps the single-definition guard; an untyped or non-bare-typed (array/union/generic) receiver produces no edge. + - Fix: the `query` reference doc's inline vocab/fallback snippets now read and write files with `encoding="utf-8"` (#1619 A2, thanks @edtrackai). On Windows (default cp1252) the bare `read_text()`/`write_text()` calls crashed on exactly the cross-language corpora the doc demonstrates (e.g. Cyrillic labels like `обработчик`). Fixed across all generated skill variants. - Fix: `graphify update`/`watch` no longer leaves stale sources after a deletion or a destination-only rename (#1623 / #1622, thanks @oleksii-tumanov). When the last supported file was deleted, or a rename reported only its destination in `changed_paths`, the removed source's nodes lingered in `graph.json`. The rebuild now reconciles extractor-backed sources against the files still present (code and document sources, subdirectory roots, legacy markers, symlinks, hyperedges) while preserving semantic and out-of-scope records. diff --git a/README.md b/README.md index dbd8489a5..ea3c862e3 100644 --- a/README.md +++ b/README.md @@ -1,11 +1,19 @@

- Graphify + Graphify

- 🇺🇸 English | 🇨🇳 简体中文 | 🇯🇵 日本語 | 🇰🇷 한국어 | 🇩🇪 Deutsch | 🇫🇷 Français | 🇪🇸 Español | 🇮🇳 हिन्दी | 🇧🇷 Português | 🇷🇺 Русский | 🇸🇦 العربية | 🇮🇷 فارسی | 🇮🇹 Italiano | 🇵🇱 Polski | 🇳🇱 Nederlands | 🇹🇷 Türkçe | 🇺🇦 Українська | 🇻🇳 Tiếng Việt | 🇮🇩 Bahasa Indonesia | 🇸🇪 Svenska | 🇬🇷 Ελληνικά | 🇷🇴 Română | 🇨🇿 Čeština | 🇫🇮 Suomi | 🇩🇰 Dansk | 🇳🇴 Norsk | 🇭🇺 Magyar | 🇹🇭 ภาษาไทย | 🇺🇿 Oʻzbekcha | 🇹🇼 繁體中文 | 🇵🇭 Filipino + Graphify-Labs%2Fgraphify | Trendshift

+
+
Read this in other languages + +🇺🇸 English | 🇨🇳 简体中文 | 🇯🇵 日本語 | 🇰🇷 한국어 | 🇩🇪 Deutsch | 🇫🇷 Français | 🇪🇸 Español | 🇮🇳 हिन्दी | 🇧🇷 Português | 🇷🇺 Русский | 🇸🇦 العربية | 🇮🇷 فارسی | 🇮🇹 Italiano | 🇵🇱 Polski | 🇳🇱 Nederlands | 🇹🇷 Türkçe | 🇺🇦 Українська | 🇻🇳 Tiếng Việt | 🇮🇩 Bahasa Indonesia | 🇸🇪 Svenska | 🇬🇷 Ελληνικά | 🇷🇴 Română | 🇨🇿 Čeština | 🇫🇮 Suomi | 🇩🇰 Dansk | 🇳🇴 Norsk | 🇭🇺 Magyar | 🇹🇭 ภาษาไทย | 🇺🇿 Oʻzbekcha | 🇹🇼 繁體中文 | 🇵🇭 Filipino + +
+
+

YC S26 Discord @@ -18,21 +26,22 @@ X

+Type `/graphify` in your AI coding assistant and it maps your entire project (code, docs, PDFs, images, videos) into a **knowledge graph** you can **query instead of grepping** through files. +

- - Star History Chart - + graphify's interactive graph.html showing the FastAPI codebase as a force-directed knowledge graph with a legend of detected communities +

+

+ The FastAPI codebase mapped by graphify. Every node is a concept, colors are detected communities, and the whole thing is clickable in graph.html.

-Type `/graphify` in your AI coding assistant and it maps your entire project — code, docs, PDFs, images, videos — into a knowledge graph you can query instead of grepping through files. - -Works in Claude Code, Codex, OpenCode, Kilo Code, Cursor, Gemini CLI, GitHub Copilot CLI, VS Code Copilot Chat, Aider, Amp, OpenClaw, Factory Droid, Trae, Hermes, Kimi Code, Kiro, Pi, Devin CLI, and Google Antigravity. +**Works in:** Claude Code, Codex, OpenCode, Kilo Code, Cursor, Gemini CLI, GitHub Copilot CLI, VS Code Copilot Chat, Aider, Amp, OpenClaw, Factory Droid, Trae, Hermes, Kimi Code, Kiro, Pi, Devin CLI, and Google Antigravity. ``` /graphify . ``` -That's it. You get three files: +That's it. You get **three files**: ``` graphify-out/ @@ -41,12 +50,62 @@ graphify-out/ └── graph.json the full graph — query it anytime without re-reading your files ``` -For a readable architecture page with Mermaid call-flow diagrams, run: +--- -```bash -graphify export callflow-html +## What it does + +A knowledge graph is not a vector index: it is **deterministic**, **every edge is explained**, and it costs **zero tokens to build**. + +| Capability | What you get | +|---|---| +| **God nodes** | The most-connected concepts, so you see what everything flows through | +| **Communities** | The graph split into subsystems (Leiden), with LLM-free labels | +| **Cross-file links** | `calls` / `imports` / `inherits` / `mixes_in` resolved across ~40 languages via tree-sitter AST | +| **Query, path, explain** | Ask a question, trace the path between two things, or explain one concept, all against `graph.json` | +| **Rationale + doc refs** | `# NOTE:` / `# WHY:` comments and ADR/RFC citations become first-class nodes linked to the code | +| **Beyond code** | Docs, PDFs, images, and video/audio all map into the same graph | +| **Local-first** | AST plus a local embedder build the graph with zero LLM credits; your code never leaves your machine | + +--- + +## See it in action + +Once the graph is built you query it instead of reading files. Real output, graphify run on the FastAPI codebase shown above: + +```text +$ graphify explain "APIRouter" +Node: APIRouter + Source: routing.py L2210 + Community: 2 + Degree: 47 + +Connections (47): + --> RequestValidationError [uses] [INFERRED] + --> Dependant [uses] [INFERRED] + --> .get() [method] [EXTRACTED] + <-- __init__.py [imports] [EXTRACTED] + ... + +$ graphify path "FastAPI" "ModelField" +Shortest path (3 hops): + FastAPI --uses--> DefaultPlaceholder <--references-- get_request_handler() --references--> ModelField ``` +Every edge carries a **confidence tag** (`EXTRACTED` = explicit in the source, `INFERRED` = derived by resolution), so you can tell what was read directly from what was inferred. `graphify query ""` returns a scoped subgraph for a plain-language question, and `graphify path A B` traces how any two things connect. + +--- + +## Benchmarks + +| Benchmark | Metric | graphify | Field | +|---|---|---|---| +| LOCOMO (n=300) | recall@10 | **0.497** | mem0 0.048, supermemory 0.149 | +| LOCOMO (n=300) | QA accuracy | 45.3% | supermemory 49.7%, mem0 27.3% | +| LongMemEval-S (n=50) | QA accuracy | **76%** | tied with dense RAG | +| Graph build | LLM credits | **0** | per-token for most systems | + +Every system ran on the same harness with the same model and budgets, scored by a judge blind-validated against a second judge (90.6% agreement, Cohen's kappa 0.81). Full per-system tables, the code-intelligence result, and reproduction commands: **[BENCHMARKS.md](./BENCHMARKS.md)**. + --- ## Prerequisites @@ -124,7 +183,8 @@ for example `graphify claude install --project` or `graphify codex install --pro > **Git hooks and uv tool / pipx:** `graphify hook install` embeds the current interpreter path directly into the hook scripts at install time, so the post-commit hook fires correctly even in GUI git clients and CI runners where `~/.local/bin` is not on PATH. If you reinstall or upgrade graphify, re-run `graphify hook install` to refresh the embedded path. -### Pick your platform +
+Pick your platform (20+ assistants, click to expand) | Platform | Install command | |----------|----------------| @@ -152,15 +212,16 @@ for example `graphify claude install --project` or `graphify codex install --pro | Devin CLI | `graphify devin install` | | Google Antigravity | `graphify antigravity install` | -Codex users also need `multi_agent = true` under `[features]` in `~/.codex/config.toml` for parallel extraction. CodeBuddy uses the same Agent tool and PreToolUse hook mechanism as Claude Code. Factory Droid uses the `Task` tool for parallel subagent dispatch. OpenClaw and Aider use sequential extraction (parallel agent support is still early on those platforms). Trae uses the Agent tool for parallel subagent dispatch and does **not** support PreToolUse hooks — AGENTS.md is the always-on mechanism. +Codex users also need `multi_agent = true` under `[features]` in `~/.codex/config.toml` for parallel extraction. CodeBuddy uses the same Agent tool and PreToolUse hook mechanism as Claude Code. Factory Droid uses the `Task` tool for parallel subagent dispatch. OpenClaw and Aider use sequential extraction (parallel agent support is still early on those platforms). Trae uses the Agent tool for parallel subagent dispatch and does **not** support `PreToolUse` hooks, so AGENTS.md is the always-on mechanism. `--platform agents` (alias `--platform skills`) targets the generic cross-framework [Agent-Skills](https://github.com/anthropics/skills) locations: the spec's user-global `~/.agents/skills/` (read by `npx skills` and spec-compliant frameworks) for a global install, and `./.agents/skills/` for a project (`--project`) install. The bare `graphify install` stays single-platform (Claude Code) by design — use the named `agents` platform when you want the skill discoverable by any framework that reads `.agents/skills`. > Codex uses `$graphify` instead of `/graphify`. -### Optional extras +
-Install only what you need: +
+Optional extras (install only what you need) | Extra | What it adds | Install | |---|---|---| @@ -183,9 +244,12 @@ Install only what you need: | `postgres` | Live PostgreSQL introspection (`--postgres DSN`) | `uv tool install "graphifyy[postgres]"` | | `dm` | BYOND DreamMaker `.dm`/`.dme` AST extraction (may need a C compiler + `python3-dev` if no wheel matches your platform) | `uv tool install "graphifyy[dm]"` | | `terraform` | Terraform / HCL `.tf`/`.tfvars`/`.hcl` AST extraction | `uv tool install "graphifyy[terraform]"` | +| `pascal` | Pascal / Delphi `.pas`/`.dpr`/`.dpk`/`.inc` AST extraction (more accurate `calls`/`inherits` edges; falls back to a regex extractor when absent) | `uv tool install "graphifyy[pascal]"` | | `chinese` | Chinese query segmentation (jieba) | `uv tool install "graphifyy[chinese]"` | | `all` | Everything above | `uv tool install "graphifyy[all]"` | +
+ --- ## Make your assistant always use the graph @@ -217,19 +281,24 @@ Run this once in your project after building a graph: | Devin CLI | `graphify devin install` | | Google Antigravity | `graphify antigravity install` | -This writes a small config file that tells your assistant to consult the knowledge graph for codebase questions — preferring scoped queries like `graphify query ""` over reading the full report or grepping raw files. On platforms that support payload-bearing hooks (Claude Code, Gemini CLI), a hook fires automatically before search-style tool calls (and, on Claude Code, before reading source files one by one via the Read/Glob tools) and nudges your assistant toward the graph path. On the others (Codex, OpenCode, Cursor, etc.), the persistent instruction files (`AGENTS.md`, `.cursor/rules/`, etc.) provide the same query-first guidance. `GRAPH_REPORT.md` is still available for broad architecture review. +This writes a small config file that tells your assistant to consult the knowledge graph for codebase questions, preferring scoped queries like `graphify query ""` over reading the full report or grepping raw files. -**CodeBuddy** does the same two things as Claude Code: writes a `CODEBUDDY.md` section telling CodeBuddy to read `graphify-out/GRAPH_REPORT.md` before answering architecture questions, and installs **PreToolUse hooks** (`.codebuddy/settings.json`) that fire before Bash search commands and file reads, nudging toward `graphify query` instead. +- **Hook platforms** (Claude Code, Gemini CLI): a hook fires automatically before search-style tool calls (and, on Claude Code, before reading source files one by one via the Read/Glob tools) and nudges your assistant toward the graph path. +- **Instruction-file platforms** (Codex, OpenCode, Cursor, etc.): persistent instruction files (`AGENTS.md`, `.cursor/rules/`, etc.) provide the same query-first guidance. -**Codex** writes to `AGENTS.md` and also installs a **PreToolUse hook** in `.codex/hooks.json` that fires before every Bash tool call — same always-on mechanism as Claude Code. +`GRAPH_REPORT.md` is still available for broad architecture review. -To remove graphify from all platforms at once: `graphify uninstall` (add `--purge` to also delete `graphify-out/`). Or use the per-platform command (e.g. `graphify claude uninstall`). +**CodeBuddy** does the same two things as Claude Code: writes a `CODEBUDDY.md` section telling CodeBuddy to read `graphify-out/GRAPH_REPORT.md` before answering architecture questions, and installs `PreToolUse` hooks (`.codebuddy/settings.json`) that fire before Bash search commands and file reads, nudging toward `graphify query` instead. ---- +**Codex** writes to `AGENTS.md` and also installs a `PreToolUse` hook in `.codex/hooks.json` that fires before every Bash tool call, same always-on mechanism as Claude Code. -**Kilo Code** installs the Graphify skill to `~/.config/kilo/skills/graphify/SKILL.md` and a native `/graphify` command to `~/.config/kilo/command/graphify.md`. `graphify kilo install` also writes `AGENTS.md` plus a native **`tool.execute.before` plugin** (`.kilo/plugins/graphify.js` + `.kilo/kilo.json` or `.kilo/kilo.jsonc` registration) so Kilo gets the same always-on graph reminder behavior through native `.kilo` config. +**Kilo Code** installs the Graphify skill to `~/.config/kilo/skills/graphify/SKILL.md` and a native `/graphify` command to `~/.config/kilo/command/graphify.md`. `graphify kilo install` also writes `AGENTS.md` plus a native `tool.execute.before` plugin (`.kilo/plugins/graphify.js` + `.kilo/kilo.json` or `.kilo/kilo.jsonc` registration) so Kilo gets the same always-on graph reminder behavior through native `.kilo` config. -**Cursor** writes `.cursor/rules/graphify.mdc` with `alwaysApply: true` — Cursor includes it in every conversation automatically, no hook needed. +**Cursor** writes `.cursor/rules/graphify.mdc` with `alwaysApply: true`, so Cursor includes it in every conversation automatically, no hook needed. + +To remove graphify from all platforms at once: `graphify uninstall` (add `--purge` to also delete `graphify-out/`). Or use the per-platform command (e.g. `graphify claude uninstall`). + +--- ## What's in the report @@ -245,7 +314,7 @@ To remove graphify from all platforms at once: `graphify uninstall` (add `--purg | Type | Extensions | |------|-----------| -| Code (36 tree-sitter grammars) | `.py .ts .js .jsx .tsx .mjs .go .rs .java .c .cpp .h .hpp .cu .cuh .metal .rb .cs .kt .scala .php .swift .lua .luau .zig .ps1 .psm1 .ex .exs .m .mm .jl .vue .svelte .astro .groovy .gradle .dart .v .sv .svh .sql .f .f90 .f95 .f03 .f08 .pas .pp .dpr .dpk .lpr .inc .dfm .lfm .lpk .sh .bash .json .dm .dme .dmi .dmm .dmf .sln .slnx .csproj .fsproj .vbproj .xaml .razor .cshtml` (`.dm`/`.dme` requires `uv tool install graphifyy[dm]`; CUDA `.cu`/`.cuh` and Metal `.metal` reuse the C++ grammar) | +| Code (36 tree-sitter grammars) | `.py .ts .mts .cts .js .jsx .tsx .mjs .go .rs .java .c .cpp .cc .cxx .h .hpp .cu .cuh .metal .rb .cs .kt .kts .scala .php .swift .lua .luau .toc .zig .ps1 .psm1 .psd1 .ex .exs .m .mm .jl .vue .svelte .astro .groovy .gradle .dart .v .sv .svh .sql .f .f90 .f95 .f03 .f08 .pas .pp .dpr .dpk .lpr .inc .dfm .lfm .lpk .sh .bash .json .dm .dme .dmi .dmm .dmf .sln .slnx .csproj .fsproj .vbproj .xaml .razor .cshtml` (`.dm`/`.dme` requires `uv tool install graphifyy[dm]`; `.mts`/`.cts` reuse the TypeScript grammar, `.cc`/`.cxx` and CUDA `.cu`/`.cuh` and Metal `.metal` reuse the C++ grammar) | | Salesforce Apex | `.cls .trigger` (regex-based; classes, interfaces, enums, methods, triggers, SOQL/DML edges) | | Terraform / HCL | `.tf .tfvars .hcl` (requires `uv tool install graphifyy[terraform]`) | | MCP configs | `.mcp.json` `mcp.json` `mcp_servers.json` `claude_desktop_config.json` — extracts server nodes, package refs, env var requirements | @@ -258,7 +327,7 @@ To remove graphify from all platforms at once: `graphify uninstall` (add `--purg | Video / Audio | `.mp4 .mov .mp3 .wav` and more (requires `uv tool install graphifyy[video]`) | | YouTube / URLs | any video URL (requires `uv tool install graphifyy[video]`) | -Code is extracted locally with no API calls (AST via tree-sitter). Everything else goes through your AI assistant's model API. +Code is extracted **locally with no API calls** (AST via tree-sitter). Everything else goes through your AI assistant's model API. Google Drive for desktop `.gdoc`, `.gsheet`, and `.gslides` files are shortcut pointers, not document content. To include native Google Docs, Sheets, and Slides @@ -445,7 +514,7 @@ These are only needed for **headless / CI extraction** (`graphify extract`). Whe - **Video / audio** — transcribed locally with faster-whisper. Nothing leaves your machine. - **Docs, PDFs, images** — sent to your AI assistant for semantic extraction (via the `/graphify` skill, using whatever model your IDE session runs). Headless `graphify extract` requires `GEMINI_API_KEY` / `GOOGLE_API_KEY` (Gemini), `MOONSHOT_API_KEY` (Kimi), `ANTHROPIC_API_KEY` (Claude), `OPENAI_API_KEY` (OpenAI), `DEEPSEEK_API_KEY` (DeepSeek), a running Ollama instance (`OLLAMA_BASE_URL`), AWS credentials via the standard provider chain (Bedrock - no API key needed, uses IAM), or the `claude` CLI binary (Claude Code - no API key needed, uses your Claude subscription). The `--dedup-llm` flag uses the same key. - **Data residency** — `graphify extract` auto-detects which provider to use based on which API key is set (priority: Gemini → Kimi → Claude → OpenAI → DeepSeek → Azure → Bedrock → Ollama). For code with data-residency requirements, use `--backend ollama` (fully local) or pass an explicit `--backend` flag. Kimi (`MOONSHOT_API_KEY`) routes to Moonshot AI servers in China. -- No telemetry, no usage tracking, no analytics. +- **No telemetry**, no usage tracking, no analytics. - **Query logging** — every `graphify query`, `graphify path`, `graphify explain`, and MCP `query_graph` call is logged to `~/.cache/graphify-queries.log` in JSON Lines format (timestamp, question, corpus, nodes returned, duration). Full subgraph responses are **not** stored by default. Set `GRAPHIFY_QUERY_LOG_DISABLE=1` to opt out, or `GRAPHIFY_QUERY_LOG=/dev/null` to silence without disabling the code path. --- @@ -513,6 +582,14 @@ uv tool upgrade graphifyy graphify install # overwrites the skill file ``` +**Claude Code prompt cache invalidated after every `graphify extract`** +Graphify writes output files (`graph.json`, `graphify-out/`) into the workspace. If those paths aren't ignored, every write invalidates Claude Code's prompt cache, forcing a full re-upload at cache-write rates on the next turn. Add them to `.claudeignore`: +```text +# .claudeignore +graph.json +graphify-out/ +``` + --- ## Full command reference @@ -679,7 +756,7 @@ graphify label ./my-project --backend=openai --model gpt-4o # force a specific --- -## Built on graphify — Penpax +## Built on graphify: Penpax [**Penpax**](https://graphifylabs.ai) is the always-on layer built on top of graphify — it applies the same graph approach to your entire working life: meetings, browser history, emails, files, and code, updating continuously in the background. @@ -739,3 +816,11 @@ uv run pytest tests/ -q -k "python" # filter by name See [ARCHITECTURE.md](ARCHITECTURE.md) for module responsibilities and how to add a language. + +--- + +

+ + Star History Chart + +

diff --git a/docs/graph-hero.png b/docs/graph-hero.png new file mode 100644 index 000000000..f4968f6d1 Binary files /dev/null and b/docs/graph-hero.png differ diff --git a/docs/logo.png b/docs/logo.png new file mode 100644 index 000000000..95ad01b3b Binary files /dev/null and b/docs/logo.png differ diff --git a/graphify/__main__.py b/graphify/__main__.py index 59dcd70a5..3a8113587 100644 --- a/graphify/__main__.py +++ b/graphify/__main__.py @@ -658,26 +658,31 @@ def _canonical_platform(platform_name: str) -> str: def _replace_or_append_section(content: str, marker: str, new_section: str) -> str: """Idempotently update or append a graphify-owned section in shared files. - If ``marker`` is not in ``content``, append ``new_section`` to the end - (with a blank-line separator if there's existing content). - - If ``marker`` IS in ``content``, replace the existing section in place. - The section runs from the first line containing ``marker`` to the line - before the next H2 heading (``## `` at line start), or to EOF if no later - H2 exists. This lets older installs receive the updated copy without - users having to uninstall and reinstall — important for the issue #580 - fix where existing report-first text would otherwise silently linger. + If no line is exactly ``marker`` (the heading, at column 0), append + ``new_section`` to the end (with a blank-line separator if there's existing + content). + + If a real ``marker`` heading exists, replace the existing section in place. + The section runs from that heading to the line before the next H2 heading + (``## `` at line start), or to EOF if no later H2 exists. This lets older + installs receive the updated copy without users having to uninstall and + reinstall (issue #580). + + The heading is matched only when a line *is* exactly ``marker`` (after + stripping surrounding whitespace), never as a substring. Matching ``## + graphify`` inside a bullet or an inline reference used to anchor the replace + on that mention and delete every line from there to the next heading, + silently destroying hand-curated content (#1688). When several exact + headings exist, the last one is used, since graphify's section is appended. """ - if marker not in content: + lines = content.split("\n") + starts = [i for i, line in enumerate(lines) if line.strip() == marker] + if not starts: if content.strip(): return content.rstrip() + "\n\n" + new_section.lstrip() return new_section.lstrip() - lines = content.split("\n") - start = next((i for i, line in enumerate(lines) if marker in line), None) - if start is None: - return content.rstrip() + "\n\n" + new_section.lstrip() - + start = starts[-1] end = len(lines) for j in range(start + 1, len(lines)): if lines[j].startswith("## "): @@ -1325,8 +1330,12 @@ def _devin_rules_uninstall(project_dir: Path) -> None: if (!existsSync(join(directory, "graphify-out", "graph.json"))) return; if (input.tool === "bash") { + // Separate with ';' not '&&' — Windows PowerShell 5.1 rejects '&&' as a + // statement separator ("not a valid statement separator"), which broke + // the first bash command in every OpenCode session on Windows (#1646). + // ';' works in PowerShell 5.1, Bash, and POSIX shells alike. output.args.command = - 'echo "[graphify] Knowledge graph available. Read graphify-out/GRAPH_REPORT.md for god nodes and architecture context before searching files." && ' + + 'echo "[graphify] Knowledge graph available. Read graphify-out/GRAPH_REPORT.md for god nodes and architecture context before searching files." ; ' + output.args.command; reminded = true; } @@ -1502,8 +1511,10 @@ def _uninstall_kilo_plugin(project_dir: Path) -> None: if (!existsSync(join(directory, "graphify-out", "graph.json"))) return; if (input.tool === "bash") { + // ';' not '&&' — Windows PowerShell 5.1 rejects '&&' as a statement + // separator, breaking the first bash command of the session (#1646). output.args.command = - 'echo "[graphify] knowledge graph at graphify-out/. For focused questions, run graphify query with your question (scoped subgraph, usually much smaller than GRAPH_REPORT.md) instead of grepping raw files. Read GRAPH_REPORT.md only for broad architecture context." && ' + + 'echo "[graphify] knowledge graph at graphify-out/. For focused questions, run graphify query with your question (scoped subgraph, usually much smaller than GRAPH_REPORT.md) instead of grepping raw files. Read GRAPH_REPORT.md only for broad architecture context." ; ' + output.args.command; reminded = true; } @@ -3549,6 +3560,10 @@ def main() -> None: } except Exception: existing_labels = {} + # Accumulate token usage from the labeling LLM calls so cluster-only mode + # reports real cost instead of a hardcoded zero (#1694). Stays {0, 0} on + # the reuse / no-label paths, which make no LLM calls. + label_token_usage = {"input": 0, "output": 0} if labels_path.exists() and not force_relabel: # Reuse saved labels, but don't blindly trust them: the graph may have # been re-scoped/re-clustered since labeling, in which case a cid now @@ -3632,6 +3647,7 @@ def main() -> None: generated_labels, _ = generate_community_labels( G, label_communities_input, backend=label_backend, model=label_model, gods=gods, max_concurrency=label_max_concurrency, batch_size=label_batch_size, + usage_out=label_token_usage, ) # Only let the LLM OVERRIDE where it produced a real name — its no-backend # fallback returns "Community {cid}" placeholders, which must not clobber @@ -3642,7 +3658,7 @@ def main() -> None: }) stages.mark("label") questions = suggest_questions(G, communities, labels) - tokens = {"input": 0, "output": 0} + tokens = label_token_usage from graphify.export import _git_head as _gh _commit = _gh() from graphify.report import load_learning_for_report as _llfr diff --git a/graphify/affected.py b/graphify/affected.py index dbb532b42..deacc39c8 100644 --- a/graphify/affected.py +++ b/graphify/affected.py @@ -149,6 +149,27 @@ def affected_nodes( queue: deque[tuple[str, int]] = deque([(seed, 0)]) hits: list[AffectedHit] = [] + # #1669: seed the reverse walk with the root's own member nodes (one outward + # `method`/`contains` hop). A caller can bind to a class's method node rather + # than the class node itself (e.g. `Service.call` resolves to the `def + # self.call` node, #1634), so those callers are unreachable from the class + # otherwise. The member nodes are seeds only (not reported as hits), and + # `method`/`contains` stay out of the general relation-filtered walk, so this + # adds no forward noise anywhere else. + if hasattr(graph, "out_edges"): + member_edges = graph.out_edges(seed, data=True) + else: + member_edges = ( + (s, t, d) for s, t, d in graph.edges(data=True) if s == seed + ) + for _s, member, data in member_edges: + if str(data.get("relation", "")) not in ("method", "contains"): + continue + member = str(member) + if member not in seen: + seen.add(member) + queue.append((member, 0)) + while queue: current, current_depth = queue.popleft() if current_depth >= depth: diff --git a/graphify/analyze.py b/graphify/analyze.py index 4be1a401c..0d4bbe4c9 100644 --- a/graphify/analyze.py +++ b/graphify/analyze.py @@ -663,6 +663,11 @@ def _endpoint_source_file(node_id: str) -> str: if rel not in ("imports_from", "re_exports"): continue + # Deferred `import(...)` edges are real dependencies but do not form a + # hard file-level cycle, so they are excluded from cycle detection (#1241). + if data.get("deferred"): + continue + src_file_attr = data.get("source_file", "") if not isinstance(src_file_attr, str) or not src_file_attr: continue diff --git a/graphify/cache.py b/graphify/cache.py index f377889ff..bb3f9d593 100644 --- a/graphify/cache.py +++ b/graphify/cache.py @@ -180,6 +180,7 @@ def file_hash(path: Path, root: Path = Path(".")) -> str: st = p.stat() entry = _stat_index.get(abs_key) if (entry + and entry.get("hash") is not None # word-count-only entries carry no hash and entry.get("size") == st.st_size and entry.get("mtime_ns") == st.st_mtime_ns): return entry["hash"] @@ -199,12 +200,65 @@ def file_hash(path: Path, root: Path = Path(".")) -> str: digest = h.hexdigest() if st is not None: - _stat_index[abs_key] = {"size": st.st_size, "mtime_ns": st.st_mtime_ns, "hash": digest} + entry = _stat_index.get(abs_key) + if (entry is not None + and entry.get("size") == st.st_size + and entry.get("mtime_ns") == st.st_mtime_ns): + entry["hash"] = digest # preserve a co-located word_count + else: + _stat_index[abs_key] = {"size": st.st_size, "mtime_ns": st.st_mtime_ns, "hash": digest} _stat_index_dirty = True return digest +def cached_word_count(path: Path, root: Path, compute) -> int: + """Word count with the same (size, mtime_ns) stat-fastpath cache as + :func:`file_hash`, persisted in the shared stat index. + + ``detect()`` counts words in every PDF/docx/text file to size the corpus, + which re-opens and re-parses every binary on each run — minutes on a large + docs corpus even when only a handful of files changed (#1656). This caches + the count against the file's stat signature so an unchanged file is counted + once and read from the index thereafter. ``compute(path)`` produces the + count on a miss. A file that can't be stat'd (e.g. a Windows long path the + index normalization can't reach) simply recomputes and isn't cached — + correct, just not accelerated. + """ + global _stat_index_dirty + p = _normalize_path(Path(path)) + root = _normalize_path(Path(root)) + _ensure_stat_index(root) + abs_key = str(p.resolve()) + st: "os.stat_result | None" = None + try: + st = p.stat() + entry = _stat_index.get(abs_key) + if (entry + and entry.get("size") == st.st_size + and entry.get("mtime_ns") == st.st_mtime_ns + and "word_count" in entry): + return entry["word_count"] + except OSError: + pass + + wc = compute(Path(path)) + + if st is not None: + entry = _stat_index.get(abs_key) + if (entry + and entry.get("size") == st.st_size + and entry.get("mtime_ns") == st.st_mtime_ns): + entry["word_count"] = wc # augment the existing hash entry in place + else: + _stat_index[abs_key] = { + "size": st.st_size, "mtime_ns": st.st_mtime_ns, "word_count": wc, + } + _stat_index_dirty = True + + return wc + + def _relativize_source_files_in(payload: dict, root: Path) -> None: """Mutate ``payload`` to rewrite absolute ``source_file`` fields as forward-slash relative paths from ``root``. diff --git a/graphify/detect.py b/graphify/detect.py index 0e1c4ba30..080dab813 100644 --- a/graphify/detect.py +++ b/graphify/detect.py @@ -630,11 +630,19 @@ def convert_office_file(path: Path, out_dir: Path) -> Path | None: normalized_path = unicodedata.normalize("NFC", str(path.resolve())) name_hash = hashlib.sha256(normalized_path.encode()).hexdigest()[:8] out_path = out_dir / f"{path.stem}_{name_hash}.md" - # Once the hash is stable the sidecar name is deterministic; skip re-writing - # an existing sidecar so an unchanged source never churns its mtime (which - # would still flag it as changed in detect_incremental). - if out_path.exists(): - return out_path + # Skip re-writing only when the sidecar is present AND at least as new as the + # source. detect_incremental tracks the SIDECAR (not the Office source), so a + # sidecar that is never rewritten after the source changes leaves the doc + # reported "unchanged" forever and freezes the graph (#1649). Re-converting + # when the source is newer bumps the sidecar's mtime/content, which the + # incremental hash check then correctly picks up. An unchanged source keeps + # its (newer-or-equal) sidecar untouched so it never churns (#1226). + try: + if out_path.exists() and os.stat(_os_path(out_path)).st_mtime >= os.stat(_os_path(path)).st_mtime: + return out_path + except OSError: + if out_path.exists(): + return out_path out_path.write_text( f"\n\n{text}", encoding="utf-8", @@ -651,7 +659,8 @@ def count_words(path: Path) -> int: return len(docx_to_markdown(path).split()) if ext == ".xlsx": return len(xlsx_to_markdown(path).split()) - return len(path.read_text(encoding="utf-8", errors="ignore").split()) + with open(_os_path(path), encoding="utf-8", errors="ignore") as f: + return len(f.read().split()) except Exception: return 0 @@ -1029,6 +1038,12 @@ def detect(root: Path, *, follow_symlinks: bool | None = None, google_workspace: } total_words = 0 + def _wc(path: Path) -> int: + # Cache word counts against each file's stat signature so unchanged + # PDFs/docx aren't re-parsed on every run just to size the corpus (#1656). + from graphify import cache as _cache + return _cache.cached_word_count(path, root, count_words) + skipped_sensitive: list[str] = [] ignore_patterns = _load_graphifyignore(root) ignore_cache: dict[Path, bool] = {} # shared across all _is_ignored calls in this scan @@ -1133,7 +1148,7 @@ def detect(root: Path, *, follow_symlinks: bool | None = None, google_workspace: if _is_ignored(md_path, root, ignore_patterns, _cache=ignore_cache): continue files[ftype].append(str(md_path)) - total_words += count_words(md_path) + total_words += _wc(md_path) else: skipped_sensitive.append(str(p) + " [Google Workspace export produced no readable text]") continue @@ -1144,14 +1159,14 @@ def detect(root: Path, *, follow_symlinks: bool | None = None, google_workspace: if _is_ignored(md_path, root, ignore_patterns, _cache=ignore_cache): continue files[ftype].append(str(md_path)) - total_words += count_words(md_path) + total_words += _wc(md_path) else: # Conversion failed (library not installed) - skip with note skipped_sensitive.append(str(p) + " [office conversion failed - pip install graphifyy[office]]") continue files[ftype].append(str(p)) if ftype != FileType.VIDEO: - total_words += count_words(p) + total_words += _wc(p) for ftype in files: files[ftype].sort() @@ -1185,12 +1200,40 @@ def detect(root: Path, *, follow_symlinks: bool | None = None, google_workspace: } +def _os_path(path: Path) -> str: + r"""Return an OS path string safe for open()/stat() on Windows long paths. + + On win32, paths longer than the legacy MAX_PATH (260 chars) are rejected by + the plain file APIs unless prefixed with the extended-length marker ``\\?\`` + (which also requires a fully-qualified path). Without it, _md5_file / + save_manifest / count_words silently fail to hash deeply-nested files, so + their manifest entry never stabilizes and detect_incremental re-flags them + as changed on every run (#1655). cache._normalize_path strips this prefix + for stable KEYS; this adds it for I/O. Non-win32 and already-prefixed paths + pass through unchanged. + """ + import sys + if sys.platform != "win32": + return str(path) + s = str(path) + if s.startswith("\\\\?\\"): + return s + try: + s = os.path.abspath(s) # \\?\ requires a fully-qualified path + except Exception: + return str(path) + if s.startswith("\\\\"): + # UNC share \\server\share -> \\?\UNC\server\share + return "\\\\?\\UNC\\" + s[2:] + return "\\\\?\\" + s + + def _md5_file(path: Path) -> str: """MD5 of file contents streamed in 64KB chunks — for change detection only.""" import hashlib as _hl h = _hl.md5(usedforsecurity=False) try: - with path.open("rb") as f: + with open(_os_path(path), "rb") as f: for chunk in iter(lambda: f.read(65536), b""): h.update(chunk) except OSError: @@ -1202,7 +1245,7 @@ def _stat_and_hash(path_str: str) -> tuple[str, float, str] | None: """Stat + MD5 a single file; returns None on OSError (e.g. deleted mid-run).""" try: p = Path(path_str) - return path_str, p.stat().st_mtime, _md5_file(p) + return path_str, os.stat(_os_path(p)).st_mtime, _md5_file(p) except OSError: return None @@ -1407,7 +1450,7 @@ def detect_incremental( for f in file_list: stored = manifest.get(f) try: - current_mtime = Path(f).stat().st_mtime + current_mtime = os.stat(_os_path(Path(f))).st_mtime except Exception: current_mtime = 0 diff --git a/graphify/export.py b/graphify/export.py index 176b17909..36e4f9949 100644 --- a/graphify/export.py +++ b/graphify/export.py @@ -1278,7 +1278,10 @@ def safe_name(label: str) -> str: group_sizes: dict[int, tuple[int, int]] = {} group_cols: dict[int, int] = {} for cid in sorted_cids: - members = communities[cid] + # Skip dangling community members with no backing node / filename, so box + # sizing matches the cards actually laid out and `G.nodes[m]` never + # KeyErrors below — mirrors the to_obsidian guard (#1236). + members = [m for m in communities[cid] if m in G and m in node_filenames] n = len(members) inner_cols = max(1, math.ceil(math.sqrt(n))) w = max(600, 220 * inner_cols) @@ -1351,6 +1354,9 @@ def safe_name(label: str) -> str: # Node cards inside the group - laid out in the same ceil(sqrt(n))-column # grid the box was sized for (group_cols[cid]), so cards fill the box. inner_cols = group_cols[cid] + # Same dangling-member guard as the sizing loop and to_obsidian (#1236): + # a community id absent from G / node_filenames would KeyError the sort. + members = [m for m in members if m in G and m in node_filenames] sorted_members = sorted(members, key=lambda n: G.nodes[n].get("label", n)) for m_idx, node_id in enumerate(sorted_members): col = m_idx % inner_cols diff --git a/graphify/extract.py b/graphify/extract.py index 267cf3664..be7e99448 100644 --- a/graphify/extract.py +++ b/graphify/extract.py @@ -868,6 +868,69 @@ def _java_type_parameters_in_scope(node, source: bytes) -> frozenset[str]: return frozenset(names) +# java.lang (auto-imported) plus the ubiquitous java.util / java.io / java.time / +# java.util.{stream,function,concurrent} / java.math / java.nio.file types that +# appear as field, parameter, return, and generic-argument annotations. They never +# resolve to a project node, so emitting `references` edges to them is pure noise +# (mirrors _GO_PREDECLARED_TYPES / _PYTHON_ANNOTATION_NOISE). Suppressed at the +# type-ref walker so they are never created as nodes or emitted as edges. The +# boxed-scalar/`void` primitives are already dropped by grammar node type above; +# these are the class/interface names the grammar reports as identifiers. +_JAVA_BUILTIN_TYPES = frozenset({ + # java.lang — core + "Object", "String", "CharSequence", "StringBuilder", "StringBuffer", + "Number", "Byte", "Short", "Integer", "Long", "Float", "Double", + "Boolean", "Character", "Void", "Class", "Enum", "Record", "Math", + "System", "Thread", "Runnable", "Comparable", "Iterable", "Cloneable", + "AutoCloseable", "Appendable", "Readable", "Process", "ProcessBuilder", + "Runtime", "Package", "ThreadLocal", "InheritableThreadLocal", + # java.lang — throwables + "Throwable", "Exception", "RuntimeException", "Error", + "IllegalArgumentException", "IllegalStateException", "NullPointerException", + "IndexOutOfBoundsException", "ArrayIndexOutOfBoundsException", + "ClassCastException", "NumberFormatException", "ArithmeticException", + "UnsupportedOperationException", "InterruptedException", + "CloneNotSupportedException", "SecurityException", "StackOverflowError", + "OutOfMemoryError", "AssertionError", + # java.util — collections & core + "Collection", "List", "ArrayList", "LinkedList", "Vector", "Stack", + "Set", "HashSet", "LinkedHashSet", "TreeSet", "SortedSet", "NavigableSet", + "EnumSet", "Map", "HashMap", "LinkedHashMap", "TreeMap", "SortedMap", + "NavigableMap", "Hashtable", "EnumMap", "Properties", "Queue", "Deque", + "ArrayDeque", "PriorityQueue", "Iterator", "ListIterator", "Comparator", + "Optional", "OptionalInt", "OptionalLong", "OptionalDouble", "Collections", + "Arrays", "Objects", "Date", "Calendar", "Random", "UUID", "Scanner", + "StringJoiner", "StringTokenizer", "BitSet", "Spliterator", "Locale", + "NoSuchElementException", "ConcurrentModificationException", + # java.util.stream + "Stream", "IntStream", "LongStream", "DoubleStream", "Collector", + "Collectors", + # java.util.function + "Function", "BiFunction", "Consumer", "BiConsumer", "Supplier", + "Predicate", "BiPredicate", "UnaryOperator", "BinaryOperator", + "IntFunction", "ToIntFunction", "ToLongFunction", "ToDoubleFunction", + # java.util.concurrent + "Callable", "Future", "CompletableFuture", "CompletionStage", "Executor", + "ExecutorService", "Executors", "ScheduledExecutorService", "TimeUnit", + "ConcurrentHashMap", "ConcurrentMap", "CopyOnWriteArrayList", + "BlockingQueue", "CountDownLatch", "Semaphore", "CyclicBarrier", + "AtomicInteger", "AtomicLong", "AtomicBoolean", "AtomicReference", + # java.time + "Instant", "Duration", "Period", "LocalDate", "LocalTime", "LocalDateTime", + "ZonedDateTime", "OffsetDateTime", "ZoneId", "ZoneOffset", "DayOfWeek", + "Month", "Year", "Clock", "DateTimeFormatter", + # java.io / java.nio.file + "IOException", "UncheckedIOException", "FileNotFoundException", "File", + "InputStream", "OutputStream", "Reader", "Writer", "BufferedReader", + "BufferedWriter", "InputStreamReader", "OutputStreamWriter", "FileReader", + "FileWriter", "PrintStream", "PrintWriter", "ByteArrayInputStream", + "ByteArrayOutputStream", "Serializable", "Closeable", "Path", "Paths", + "Files", + # java.math + "BigDecimal", "BigInteger", +}) + + def _java_collect_type_refs( node, source: bytes, @@ -885,19 +948,23 @@ def _java_collect_type_refs( return if t == "type_identifier": name = _read_text(node, source) - if name and name not in skip: + if name and name not in skip and name not in _JAVA_BUILTIN_TYPES: out.append((name, "generic_arg" if generic else "type")) return if t == "scoped_type_identifier": text = _read_text(node, source).rsplit(".", 1)[-1] - if text: + if text and text not in _JAVA_BUILTIN_TYPES: out.append((text, "generic_arg" if generic else "type")) return if t == "generic_type": for c in node.children: if c.type in ("type_identifier", "scoped_type_identifier"): text = _read_text(c, source).rsplit(".", 1)[-1] - if text and (c.type == "scoped_type_identifier" or text not in skip): + if ( + text + and text not in _JAVA_BUILTIN_TYPES + and (c.type == "scoped_type_identifier" or text not in skip) + ): out.append((text, "generic_arg" if generic else "type")) break for c in node.children: @@ -1771,7 +1838,17 @@ def _resolve_js_import_target(raw: str, str_path: str) -> "tuple[str, Path | Non module_name = raw.split("/")[-1] if not module_name: return None - return _make_id(module_name), None + # Unresolved: relative/absolute, tsconfig-alias and workspace resolution have + # all run and failed, so this is an external package (or a dangling local + # path). Namespace the id with the "ref" prefix — the J-4 convention already + # used for tsconfig `extends`/`$ref` externals — so it can NEVER collapse to + # the same _make_id as a local file/symbol node. Without it, the bare + # last-segment id (e.g. "tailwindcss/colors" -> "colors") collides with any + # unrelated local file of that stem via build.py's pre-migration alias index, + # producing a confident (EXTRACTED) cross-language phantom imports_from edge + # (#1638). The ref-namespaced target has no node, so build drops it as an + # external reference — the correct outcome for a third-party import. + return _make_id("ref", raw), None def _import_js(node, source: bytes, file_nid: str, stem: str, edges: list, str_path: str, scope_stack: list[str] | None = None) -> None: @@ -1922,8 +1999,13 @@ def _dynamic_import_js(node, source: bytes, caller_nid: str, str_path: str, edge edges.append({ "source": caller_nid, "target": tgt_nid, + # A deferred `import(...)` is a real dependency, so keep it as an + # `imports_from` edge (visible in the graph) but mark it `deferred` + # so find_import_cycles does not treat it as a static import and + # report a phantom file cycle (#1241). "relation": "imports_from", "context": "import", + "deferred": True, "confidence": "EXTRACTED", "source_file": str_path, "source_location": f"L{node.start_point[0] + 1}", @@ -2340,6 +2422,55 @@ def _new_type(declarator) -> str | None: return table +def _ts_receiver_type_table(root, source: bytes, table: dict[str, str]) -> None: + """Add TS/JS receiver bindings to ``table`` (name -> TypeName), for member-call + resolution beyond the constructor-injected `this.field` case (#1630): + + * local ``const/let/var x = new Foo()`` -> ``x: Foo`` (Pattern A); + * a type-annotated parameter ``(svc: Svc)`` -> ``svc: Svc`` (Pattern B), so a + call on the param — including inside a returned closure — resolves. + + File-scoped, first-binding-wins (merged into the constructor-injection table, + which is populated first and therefore wins on a name clash). Only a bare + ``type_identifier`` (a single class/interface name) is recorded — an array, + union, generic, qualified, or predefined type is skipped (precision over + recall, matching the receiver-typed resolvers for Swift/C#/C++).""" + def _bare_type_ident(type_annotation): + # type_annotation -> ": T"; accept only a single type_identifier child. + idents = [c for c in type_annotation.children if c.type == "type_identifier"] + others = [c for c in type_annotation.children + if c.is_named and c.type not in ("type_identifier",)] + if len(idents) == 1 and not others: + return _read_text(idents[0], source) + return None + + stack = [root] + while stack: + n = stack.pop() + t = n.type + if t == "variable_declarator": + name_n = n.child_by_field_name("name") + value = n.child_by_field_name("value") + if (name_n is not None and name_n.type == "identifier" + and value is not None and value.type == "new_expression"): + ctor = value.child_by_field_name("constructor") + if ctor is not None and ctor.type in ("identifier", "type_identifier"): + name = _read_text(name_n, source) + tname = _read_text(ctor, source) + if name and tname and name not in table: + table[name] = tname + elif t == "required_parameter" or t == "optional_parameter": + pat = n.child_by_field_name("pattern") + ann = n.child_by_field_name("type") + if pat is not None and pat.type == "identifier" and ann is not None: + tname = _bare_type_ident(ann) + name = _read_text(pat, source) + if name and tname and name not in table: + table[name] = tname + for c in n.children: + stack.append(c) + + def _objc_local_var_types(body_node, source: bytes, table: dict[str, str]) -> None: """Collect ``var -> ClassName`` from ObjC local declarations (``Foo *f = ...;``) in a method body, for receiver typing in the cross-file message-send pass @@ -2934,7 +3065,12 @@ def _swift_extra_walk(node, source: bytes, file_nid: str, stem: str, str_path: s _RUBY_CONFIG = LanguageConfig( ts_module="tree_sitter_ruby", - class_types=frozenset({"class"}), + # `module Foo` is a container node just like `class Foo` in tree-sitter's + # Ruby grammar (name in a `constant` child, body in `body_statement`), so it + # gets a node and its methods attach via `method` (#1640). Without it, plain + # utility/`module_function` modules produced no node and their methods hung + # off the file via `contains` with dot-less labels. + class_types=frozenset({"class", "module"}), function_types=frozenset({"method", "singleton_method"}), import_types=frozenset(), call_types=frozenset({"call"}), @@ -3224,6 +3360,92 @@ def visit(n) -> None: return bindings +def _ruby_const_last_name(node, source: bytes) -> str: + """Last constant of a ``constant`` or ``scope_resolution`` (``A::B::C`` -> ``C``).""" + if node is None: + return "" + if node.type == "constant": + return _read_text(node, source) + if node.type == "scope_resolution": + consts = [c for c in node.children if c.type == "constant"] + if consts: + return _read_text(consts[-1], source) + return "" + + +# `Const = (...)` shapes that define a lightweight class named after the +# constant. tree-sitter parses each as an `assignment`, not a `class`, so the +# generic class branch never saw them (#1640). +_RUBY_CLASS_FACTORIES = frozenset({("Struct", "new"), ("Class", "new"), ("Data", "define")}) + + +def _ruby_extra_walk(node, source: bytes, file_nid: str, stem: str, str_path: str, + nodes: list, edges: list, seen_ids: set, function_bodies: list, + parent_class_nid: str | None, add_node, add_edge, walk, + callable_def_nids: set) -> bool: + """Ruby: a constant assignment whose RHS is ``Struct.new(...)``, + ``Class.new(Super)`` or ``Data.define(...)`` defines a class named after the + constant (#1640). Synthesize the class node, attach block-defined methods via + ``method`` (by recursing the block with the new node as parent), and emit an + ``inherits`` edge for ``Class.new(Super)``. Returns True if handled. + """ + if node.type != "assignment": + return False + left = node.child_by_field_name("left") + right = node.child_by_field_name("right") + if left is None or right is None or left.type != "constant" or right.type != "call": + return False + recv = right.child_by_field_name("receiver") + meth = right.child_by_field_name("method") + if recv is None or meth is None or recv.type != "constant": + return False + if (_read_text(recv, source), _read_text(meth, source)) not in _RUBY_CLASS_FACTORIES: + return False + + const_name = _read_text(left, source) + if not const_name: + return False + line = node.start_point[0] + 1 + class_nid = _make_id(stem, const_name) + add_node(class_nid, const_name, line) + callable_def_nids.add(class_nid) # a class is callable (its constructor) + # Mirror the generic class branch: containment always hangs off the file node. + add_edge(file_nid, class_nid, "contains", line) + + # `Class.new(Super)` — the first positional constant argument is the superclass. + if _read_text(recv, source) == "Class": + args = next((c for c in right.children if c.type == "argument_list"), None) + if args is not None: + for arg in args.children: + if arg.type in ("constant", "scope_resolution"): + base = _ruby_const_last_name(arg, source) + if base: + base_nid = _make_id(stem, base) + if base_nid not in seen_ids: + base_nid = _make_id(base) + if base_nid not in seen_ids: + nodes.append({ + "id": base_nid, "label": base, + "file_type": "code", "source_file": "", + "source_location": "", + }) + seen_ids.add(base_nid) + add_edge(class_nid, base_nid, "inherits", line) + break + + # Recurse the do/brace block so block-defined methods attach to the class. + # The block wraps its statements in a `body_statement` (like a class body); + # descend into it so the method handler sees parent_class_nid — otherwise the + # default recurse resets the parent to None and the method hangs off the file + # with a dot-less label. + block = next((c for c in right.children if c.type in ("do_block", "block")), None) + if block is not None: + body = next((c for c in block.children if c.type == "body_statement"), block) + for child in body.children: + walk(child, parent_class_nid=class_nid) + return True + + # ── Generic extractor ───────────────────────────────────────────────────────── def _extract_generic( @@ -3295,6 +3517,10 @@ def _extract_generic( # `let vm = VM()`) live outside function bodies, so the call-walk never # reaches them. Collect (owner_nid, call_node) here and walk them too. initializer_nodes: list[tuple[str, object]] = [] + # Ruby include/extend/prepend mixins collected during the node walk (#1668), + # merged into raw_calls after the call-walk populates it (raw_calls does not + # exist yet while walk() runs). Resolved cross-file by the Ruby resolver. + _ruby_mixin_calls: list[dict] = [] # #1356: per-file map of local name -> declared type (properties + params), # threaded out as `swift_type_table` so member calls (`vm.update()`) can be # resolved to the receiver's real definition in _resolve_swift_member_calls. @@ -3589,6 +3815,17 @@ def _php_emit_base(base_name: str, rel: str, at_line: int) -> None: if sub.type == "user_type": user_type_node = sub break + # `class Foo : Bar by baz` wraps the delegated + # interface `Bar` in an `explicit_delegation` + # node; grab its first `user_type` descendant so + # the implements edge (and generic-arg recovery) + # still fire. + if sub.type == "explicit_delegation": + for inner in sub.children: + if inner.type == "user_type": + user_type_node = inner + break + break if user_type_node is None: continue base = _kotlin_user_type_name(user_type_node, source) @@ -3653,6 +3890,36 @@ def _php_emit_base(base_name: str, rel: str, at_line: int) -> None: seen_ids.add(base_nid) add_edge(class_nid, base_nid, "inherits", line) + # `include`/`extend`/`prepend ` in the class/module body -> + # a `mixes_in` edge to the module (#1668). The module usually lives + # in another file, so defer resolution to the cross-file Ruby + # resolver (reusing the #1634 candidate logic and the #1640 module + # nodes as targets). Only bare/namespaced constant arguments count; + # `extend self`, `include some_var`, etc. are skipped. + _rb_body = _find_body(node, config) + if _rb_body is not None: + for _stmt in _rb_body.children: + if _stmt.type != "call" or _stmt.child_by_field_name("receiver") is not None: + continue + _m = _stmt.child_by_field_name("method") + if _m is None or _read_text(_m, source) not in ("include", "extend", "prepend"): + continue + _args = _stmt.child_by_field_name("arguments") + if _args is None: + continue + for _arg in _args.children: + if _arg.type not in ("constant", "scope_resolution"): + continue + _mod = _ruby_const_last_name(_arg, source) + if _mod: + _ruby_mixin_calls.append({ + "caller_nid": class_nid, + "callee": _mod, + "is_mixin": True, + "source_file": str_path, + "source_location": f"L{_stmt.start_point[0] + 1}", + }) + # C#-specific: inheritance / interface implementation via base_list if config.ts_module == "tree_sitter_c_sharp": csharp_type_params = _csharp_type_parameters_in_scope(node, source) @@ -4588,6 +4855,13 @@ def _emit_java_parent_type(type_node, rel: str, at_line: int) -> None: ensure_named_node): return + if config.ts_module == "tree_sitter_ruby": + if _ruby_extra_walk(node, source, file_nid, stem, str_path, + nodes, edges, seen_ids, function_bodies, + parent_class_nid, add_node, add_edge, walk, + callable_def_nids): + return + # Python's `@property` / `@staticmethod` / `@classmethod` wrap the # inner function_definition in a `decorated_definition` node. The # default recurse below clears parent_class_nid, which would cause the @@ -4775,8 +5049,23 @@ def _php_class_const_scope(n) -> str | None: return None return _read_text(scope, source) + _tracked_body_ids: set[int] = set() + _JS_CLOSURE_TYPES = ("arrow_function", "function_expression") + def walk_calls(node, caller_nid: str) -> None: if node.type in config.function_boundary_types: + # JS/TS: an inline/returned closure not separately tracked in + # function_bodies would otherwise drop its calls at this boundary. + # Descend into it with the enclosing caller so `return () => + # svc.doThing()` links to the caller (#1630). Tracked closures + # (const-assigned arrows) are walked with their own nid — skip to + # avoid double-counting. + if (config.ts_module in ("tree_sitter_javascript", "tree_sitter_typescript") + and node.type in _JS_CLOSURE_TYPES): + body = node.child_by_field_name("body") + if body is not None and id(body) not in _tracked_body_ids: + for child in node.children: + walk_calls(child, caller_nid) return if node.type in config.call_types: @@ -4957,6 +5246,11 @@ def walk_calls(node, caller_nid: str) -> None: is_member_call = True if recv.type in ("identifier", "constant"): member_receiver = _read_text(recv, source) + elif recv.type == "scope_resolution": + # Namespaced receiver `Billing::Processor.call` — capture the + # last constant so cross-file resolution can bind it by the + # bare class name (the god-node guard bails if ambiguous). + member_receiver = _ruby_const_last_name(recv, source) or None else: # Generic: get callee from call_function_field func_node = node.child_by_field_name(config.call_function_field) if config.call_function_field else None @@ -5282,6 +5576,14 @@ def walk_calls(node, caller_nid: str) -> None: for _caller_nid, body_node in function_bodies: _swift_local_var_types(body_node, source, type_table) + # JS/TS: bodies already walked with their own caller_nid (const-assigned + # arrows, methods). An INLINE/returned arrow or function-expression that is + # NOT separately tracked (e.g. `return () => svc.doThing()`) is otherwise + # skipped at the arrow boundary in walk_calls, losing its calls — so let + # walk_calls descend into such untracked closures with the enclosing caller + # (#1630 Pattern B). Guarding on the tracked set prevents double-walking. + _tracked_body_ids.update(id(b) for _, b in function_bodies) + for caller_nid, body_node in function_bodies: walk_calls(body_node, caller_nid) @@ -5375,6 +5677,10 @@ def _scan_js_module_dispatch(n) -> None: if src in valid_ids and (tgt in valid_ids or edge["relation"] in ("imports", "imports_from", "re_exports")): clean_edges.append(edge) + # Ruby mixins were collected during the node walk (before raw_calls existed); + # fold them in so the cross-file resolver sees them (#1668). + if _ruby_mixin_calls: + raw_calls.extend(_ruby_mixin_calls) result = {"nodes": nodes, "edges": clean_edges, "raw_calls": raw_calls} if callable_def_nids: # Mark function / method / class defs with a `_callable` attribute so the @@ -5388,6 +5694,13 @@ def _scan_js_module_dispatch(n) -> None: n["_callable"] = True if swift_extensions: result["swift_extensions"] = swift_extensions + # TS/JS: augment the constructor-injection type table with local `new` + # bindings and type-annotated parameters, so `const s = new Svc(); s.m()` and + # a call on a typed param (incl. inside a closure) resolve (#1630). The + # constructor-injection entries are populated during the walk above and win on + # a name clash (first-binding-wins in the helper). + if config.ts_module in ("tree_sitter_javascript", "tree_sitter_typescript"): + _ts_receiver_type_table(root, source, type_table) if type_table: if config.ts_module == "tree_sitter_swift": result["swift_type_table"] = {"path": str_path, "table": type_table} @@ -5553,7 +5866,114 @@ def extract_js(path: Path) -> dict: config = _TS_CONFIG else: config = _JS_CONFIG - return _extract_generic(path, config) + result = _extract_generic(path, config) + if "error" not in result: + _extract_js_rationale(path, result) + return result + + +# ── JS/TS rationale + doc-reference extraction ──────────────────────────────── +# +# Parity with _extract_python_rationale: Python files get rationale nodes from +# docstrings and `# NOTE:`-style comments, but JS/TS comments were discarded +# entirely. That silently drops two high-value signals in mixed corpora: +# 1. rationale comments (`// NOTE:`, `// WHY:`, ...) — same as Python; +# 2. architecture-decision references (`ADR-0011`, `RFC 793`) that teams +# conventionally cite in file/function headers. These are the natural +# join points between code and design docs in the same graph — without +# them, code<->ADR edges never form even when the code cites the ADR. + +_JS_RATIONALE_PREFIXES = ( + "// NOTE:", "// IMPORTANT:", "// HACK:", "// WHY:", "// RATIONALE:", + "// TODO:", "// FIXME:", + "* NOTE:", "* IMPORTANT:", "* HACK:", "* WHY:", "* RATIONALE:", + "* TODO:", "* FIXME:", +) + +# Doc-reference tokens worth first-classing as graph nodes. Deliberately +# conservative: ADR-NNNN (Architecture Decision Records, any zero padding) +# and RFC NNNN / RFC-NNNN. +_JS_DOC_REF_RE = re.compile(r"\b(ADR[- ]?\d{1,5}|RFC[- ]?\d{1,5})\b", re.IGNORECASE) + +# Only look for doc references inside comments, not string literals or code. +_JS_COMMENT_LINE_RE = re.compile(r"^\s*(//|/\*|\*)") + + +def _extract_js_rationale(path: Path, result: dict) -> None: + """Post-pass: extract rationale comments and doc references from JS/TS source. + Mutates result in-place by appending to result['nodes'] and result['edges']. + """ + try: + source_text = path.read_text(encoding="utf-8", errors="replace") + except Exception: + return + + stem = _file_stem(path) + str_path = str(path) + nodes = result["nodes"] + edges = result["edges"] + seen_ids = {n["id"] for n in nodes} + file_nid = _make_id(str(path)) + seen_doc_refs: set[str] = set() + + def _add_rationale(text: str, line: int) -> None: + label = text[:80].replace("\r\n", " ").replace("\r", " ").replace("\n", " ").strip() + rid = _make_id(stem, "rationale", str(line)) + if rid not in seen_ids: + seen_ids.add(rid) + nodes.append({ + "id": rid, + "label": label, + "file_type": "rationale", + "source_file": str_path, + "source_location": f"L{line}", + }) + edges.append({ + "source": rid, + "target": file_nid, + "relation": "rationale_for", + "confidence": "EXTRACTED", + "source_file": str_path, + "source_location": f"L{line}", + "weight": 1.0, + }) + + def _add_doc_ref(token: str, line: int) -> None: + # Normalize "adr 11" / "ADR-0011" spellings to a canonical "ADR-0011" + # style label so references to the same document collapse to one node. + kind, num = re.match(r"([A-Za-z]+)[- ]?(\d+)", token).groups() + kind = kind.upper() + label = f"{kind}-{num.zfill(4)}" if kind == "ADR" else f"{kind}-{num}" + if label in seen_doc_refs: + return + seen_doc_refs.add(label) + rid = _make_id("docref", label) + if rid not in seen_ids: + seen_ids.add(rid) + nodes.append({ + "id": rid, + "label": label, + "file_type": "doc_ref", + "source_file": str_path, + "source_location": f"L{line}", + }) + edges.append({ + "source": file_nid, + "target": rid, + "relation": "cites", + "confidence": "EXTRACTED", + "source_file": str_path, + "source_location": f"L{line}", + "weight": 1.0, + }) + + for lineno, line_text in enumerate(source_text.splitlines(), start=1): + stripped = line_text.strip() + if any(stripped.startswith(p) for p in _JS_RATIONALE_PREFIXES): + _add_rationale(stripped.lstrip("/* "), lineno) + if _JS_COMMENT_LINE_RE.match(line_text): + for m in _JS_DOC_REF_RE.finditer(stripped): + _add_doc_ref(m.group(1), lineno) def extract_svelte(path: Path) -> dict: @@ -6208,6 +6628,16 @@ def add_edge(src: str, tgt: str, relation: str, line: int, add_node(iface_nid, iface_name, lineno) add_edge(file_nid if current_class_nid is None else current_class_nid, iface_nid, "contains", lineno) + if im.group(2): + for parent in im.group(2).split(","): + parent = parent.strip() + if parent: + parent_nid = _make_id(stem, parent) + if parent_nid not in seen_ids: + parent_nid = _make_id(parent) + if parent_nid not in seen_ids: + add_node(parent_nid, parent, lineno) + add_edge(iface_nid, parent_nid, "extends", lineno, confidence="INFERRED") pending_annotations = [] continue @@ -15565,6 +15995,30 @@ def _body_of(block): } +# Extensionless executables (CLI entry points like `devctl` or `manage`) carry +# their language in the shebang, not the suffix. detect.classify_file already +# routes them to the CODE path via _shebang_interpreter; _get_extractor must +# honor the same signal or these files are classified as code and then silently +# dropped by extraction. Only interpreters with a real extractor are mapped — +# detect's wider set (perl, fish, tcsh, Rscript) stays unmapped and skipped. +_SHEBANG_DISPATCH: dict[str, Any] = { + "python": extract_python, + "python2": extract_python, + "python3": extract_python, + "bash": extract_bash, + "sh": extract_bash, + "dash": extract_bash, + "zsh": extract_bash, + "ksh": extract_bash, + "node": extract_js, + "nodejs": extract_js, + "ruby": extract_ruby, + "lua": extract_lua, + "php": extract_php, + "julia": extract_julia, +} + + # ObjC-only directives. They are illegal in C and C++, so finding one in a `.h` # file is a near-zero-false-positive signal that the header is Objective-C (and so # belongs to extract_objc, not extract_c). `@property` is deliberately excluded: it @@ -15626,7 +16080,7 @@ def _is_cpp_header(path: Path) -> bool: def _get_extractor(path: Path) -> Any | None: """Return the correct extractor function for a file, or None if unsupported.""" - if path.name.endswith(".blade.php"): + if path.name.lower().endswith(".blade.php"): return extract_blade # MCP config files (.mcp.json, claude_desktop_config.json, ...) are routed # by filename before generic .json dispatch so they get MCP-aware nodes @@ -15642,14 +16096,25 @@ def _get_extractor(path: Path) -> Any | None: # (the suffix map sends `.h` to extract_c, which can't read @interface etc.). # ObjC sniffing has priority over the C++ sniff: an Objective-C++ header can # contain both `@interface` and inline C++ (`::`), and it must parse as ObjC. - if path.suffix == ".h": + suffix = path.suffix + if suffix not in _DISPATCH and suffix.lower() in _DISPATCH: + suffix = suffix.lower() + if suffix == ".h": if _is_objc_header(path): return extract_objc # A C++ class header routed to extract_c loses the class entirely (the C # grammar has no class_specifier). Reroute to extract_cpp (#1547). if _is_cpp_header(path): return extract_cpp - return _DISPATCH.get(path.suffix) + # Extensionless files: resolve by shebang, mirroring detect.classify_file. + # Without this, detect labels e.g. `#!/usr/bin/env bash` CLIs as code but + # extraction returns no extractor and the file silently contributes nothing. + if not suffix: + from graphify.detect import _shebang_interpreter + interp = _shebang_interpreter(path) + if interp is not None: + return _SHEBANG_DISPATCH.get(interp) + return _DISPATCH.get(suffix) def _safe_extract_with_xaml_root(extractor, path: Path, root: Path) -> dict: @@ -15691,7 +16156,12 @@ def _extract_single_file(args: tuple) -> tuple[int, dict]: return idx, {"nodes": [], "edges": []} result = _safe_extract_with_xaml_root(extractor, path, cache_root) - if not bypass_cache and "error" not in result: + # Never cache a zero-node result for an extractable file. Every supported + # source produces at least a file node, so an empty node list is anomalous + # (e.g. a transient batch/parallel hiccup). Caching it makes the empty + # byte-stable across runs and silently blinds affected/explain to and + # through the file (#1666); skipping the write lets a rerun self-heal. + if not bypass_cache and "error" not in result and result.get("nodes"): save_cached(path, result, cache_root) return idx, result @@ -15815,7 +16285,8 @@ def _extract_sequential( continue bypass_cache = path.suffix in _JS_CACHE_BYPASS_SUFFIXES result = _safe_extract_with_xaml_root(extractor, path, effective_root) - if not bypass_cache and "error" not in result: + # See _extract_single_file: don't cache an anomalous zero-node result (#1666). + if not bypass_cache and "error" not in result and result.get("nodes"): save_cached(path, result, effective_root) per_file[idx] = result if total_files >= _PROGRESS_INTERVAL: @@ -15911,6 +16382,27 @@ def extract( if per_file[i] is None: per_file[i] = {"nodes": [], "edges": []} + # #1666: surface any source file an extractor accepted but that produced zero + # nodes (not even a file node). Such a file is silently absent from the graph, + # so affected/explain are blind to and through it with no other signal. + _empty_sources: list[str] = [] + for i, _p in enumerate(paths): + _res = per_file[i] or {} + if _res.get("nodes") or _res.get("error"): + continue + if _get_extractor(_p) is not None: + _empty_sources.append(str(_p)) + if _empty_sources: + _shown = ", ".join(Path(x).name for x in _empty_sources[:5]) + _more = f" (+{len(_empty_sources) - 5} more)" if len(_empty_sources) > 5 else "" + print( + f" warning: {len(_empty_sources)} source file(s) produced zero nodes and " + f"are absent from the graph: {_shown}{_more}. A re-run will retry them " + f"(empties are no longer cached); if it persists, please report the " + f"file(s) (#1666).", + file=sys.stderr, flush=True, + ) + all_nodes: list[dict] = [] all_edges: list[dict] = [] all_raw_calls: list[dict] = [] @@ -16131,9 +16623,20 @@ def extract( elif e.get("relation") == "imports_from": file_to_module_imports.setdefault(e["source"], set()).add(e["target"]) - # Map each node back to its containing file_id so we can ask + # Map each node back to its containing file node id so we can ask # "did the caller's file import the callee's file?" - # Use relativized paths to match how file node IDs were remapped above (#502). + # A node and its file node share the exact same ``source_file`` string, and a + # file node is the one whose label is the basename (``add_node(file_nid, + # path.name)``). Resolving file membership by that shared string is robust + # against the path-resolution/symlink mismatch that makes + # ``relative_to(root.resolve())`` throw and fall back to a non-matching + # absolute-derived id — which would spuriously fail import evidence and (with + # the #1659 JS/TS gate below) drop a legitimately-imported call. + sf_to_file_nid: dict[str, str] = {} + for n in all_nodes: + sf = n.get("source_file") + if sf and n.get("label") == Path(str(sf)).name: + sf_to_file_nid.setdefault(str(sf), n["id"]) nid_to_file_nid: dict[str, str] = {} # nid -> raw source_file string, for the ambiguous-name tie-breakers below # (test/non-test classification + path proximity). Kept separate from the @@ -16144,6 +16647,12 @@ def extract( if not sf: continue nid_to_source_file[n["id"]] = str(sf) + fnid = sf_to_file_nid.get(str(sf)) + if fnid is not None: + nid_to_file_nid[n["id"]] = fnid + continue + # Fallback (no file node found for this source_file): derive it the old + # way from the relativized path. sf_path = Path(sf) try: sf_rel = sf_path.relative_to(root) if sf_path.is_absolute() else sf_path @@ -16159,6 +16668,10 @@ def extract( (e["source"], e["target"]) for e in all_edges if e.get("relation") in ("calls", "indirect_call") } + # JS/TS/JSX modules have no implicit cross-module scope: a call into another + # file is real ONLY if the caller imported it. So a cross-file call from one + # of these files with no import evidence is gated below (#1659). + _JS_TS_CALL_SUFFIXES = (".ts", ".tsx", ".mts", ".cts", ".js", ".jsx", ".mjs", ".cjs") for rc in all_raw_calls: callee = rc.get("callee", "") if not callee: @@ -16169,6 +16682,12 @@ def extract( # and collides with any top-level function named "log" in the corpus. if rc.get("is_member_call"): continue + # Skip Ruby include/extend/prepend mixin markers: they carry a module + # name as `callee` but are not calls — the Ruby resolver turns them into + # `mixes_in` edges. Letting the shared pass emit a `calls` edge here would + # both mislabel the relation and block the mixes_in emit as a dup (#1668). + if rc.get("is_mixin"): + continue # Exact-case match first (case is semantic). Fold only when the CALLING # file's language is case-insensitive, and only against the folded index of # case-insensitive-language definitions — so a Python `Path()` call can never @@ -16179,7 +16698,15 @@ def extract( if not candidates: continue caller = rc["caller_nid"] - caller_file_nid = nid_to_file_nid.get(caller) + # Resolve the caller's file via the raw_call's own source_file string, + # which is stable regardless of any caller_nid remap. An indirect + # callback's caller_nid is the file node, whose id may have been + # relativized after the raw_call was recorded, so a caller_nid lookup can + # miss and (with the #1659 gate) drop a legitimately-imported callback. + caller_file_nid = ( + sf_to_file_nid.get(str(rc.get("source_file", ""))) + or nid_to_file_nid.get(caller) + ) imported_symbols = file_to_symbol_imports.get(caller_file_nid, set()) imported_modules = file_to_module_imports.get(caller_file_nid, set()) @@ -16256,6 +16783,19 @@ def _has_import_evidence(candidate_id: str) -> bool: "weight": 1.0, }) continue + # #1659: a JS/TS DIRECT call with no import evidence is almost always an + # unrelated same-named export in a package that was never imported — a + # phantom cross-package edge (a 14-package monorepo had `platform` and + # `sidecar` shown as depending on `registry-protocol` purely because it + # exported generically-named symbols). JS/TS modules have no implicit + # cross-module scope, so leave it unresolved rather than binding by name + # alone. Other languages keep the #1553 single-candidate resolution: + # C/C++ headers, Ruby autoload, and same-package implicit scope + # legitimately call across files without an explicit import. Scoped to + # direct calls: the indirect_call path above is already conservative + # (INFERRED, callable-target-gated) and independent of import evidence. + if not has_import_evidence and str(rc.get("source_file", "")).endswith(_JS_TS_CALL_SUFFIXES): + continue if tgt != caller and (caller, tgt) not in existing_pairs: existing_pairs.add((caller, tgt)) # Promote to EXTRACTED when there's a direct import edge from the @@ -16359,7 +16899,8 @@ def _ignored(p: Path) -> bool: ] for fname in filenames: p = dp / fname - if p.suffix in _EXTENSIONS and not _ignored(p) and _resolves_under_root(p, containment_root): + suffix = p.suffix + if (suffix in _EXTENSIONS or suffix.lower() in _EXTENSIONS) and not _ignored(p) and _resolves_under_root(p, containment_root): results.append(p) return sorted(results) # Walk with symlink following + cycle detection @@ -16379,7 +16920,8 @@ def _ignored(p: Path) -> bool: ] for fname in filenames: p = dp / fname - if p.suffix in _EXTENSIONS and not _ignored(p) and _resolves_under_root(p, containment_root): + suffix = p.suffix + if (suffix in _EXTENSIONS or suffix.lower() in _EXTENSIONS) and not _ignored(p) and _resolves_under_root(p, containment_root): results.append(p) return sorted(results) diff --git a/graphify/llm.py b/graphify/llm.py index 44e98a8ca..a0c073afa 100644 --- a/graphify/llm.py +++ b/graphify/llm.py @@ -759,6 +759,30 @@ def _bedrock_content(user_message: str, refs: list[_ImageRef]) -> list[dict]: _LLM_JSON_MAX_BYTES = 10 * 1024 * 1024 # 10 MB hard cap before json.loads (F-016) +def _sanitize_fragment(parsed: dict) -> dict: + """Force ``nodes``/``edges``/``hyperedges`` to lists of dicts, in place. + + A model can return a well-formed top-level object whose ``edges`` (or + ``nodes``/``hyperedges``) array contains a stray non-dict entry — most often + a nested list where an edge object belongs, or the whole value being a bare + array/scalar instead of a list. Those entries slip past JSON parsing but + blow up every downstream consumer that calls ``.get()`` per entry + (semantic-cache write and the AST+semantic merge both did — #1631, crashing + with ``'list' object has no attribute 'get'`` and discarding all successful + chunks). Sanitizing here, at the single parse chokepoint, protects the cache + writer, the adaptive-retry merge, and the CLI merge in one place. + """ + for key in ("nodes", "edges", "hyperedges"): + value = parsed.get(key) + if value is None: + continue + if not isinstance(value, list): + parsed[key] = [] + continue + parsed[key] = [entry for entry in value if isinstance(entry, dict)] + return parsed + + def _parse_llm_json(raw: str) -> dict: """Strip optional markdown fences and parse JSON. Returns empty fragment on failure. @@ -792,7 +816,7 @@ def _parse_llm_json(raw: str) -> dict: try: parsed = json.loads(stripped) if isinstance(parsed, dict): - return parsed + return _sanitize_fragment(parsed) # Top-level array/scalar (common LLM output) is not a usable graph # fragment; fall through to the next strategy rather than returning a # non-dict that callers will try to subscript (e.g. result["input_tokens"]). @@ -827,7 +851,7 @@ def _parse_llm_json(raw: str) -> dict: try: parsed = json.loads(stripped[start : i + 1]) if isinstance(parsed, dict): - return parsed + return _sanitize_fragment(parsed) break except json.JSONDecodeError: break @@ -938,8 +962,18 @@ def _call_openai_compat( # default. Honour GRAPHIFY_API_TIMEOUT (seconds) for explicit override; # default to 600s, which is long enough for a 31B model on a 16k chunk # but still bounds runaway connections (issue #792 addendum). + # The SDK's transient-error retries (default 6) exist for cloud rate limits + # (429). A local Ollama server does not rate-limit, and if it wedges it will + # not recover by retrying, so 6 retries turn a 180s --api-timeout into a + # ~21min block (7 attempts x 180s) with no progress (#1686). Default ollama + # to 0 SDK retries so --api-timeout is the hard wall-clock bound and a hung + # request fails fast into the chunk-level retry/skip. An explicit + # GRAPHIFY_MAX_RETRIES still wins for users who want it. + _retries = _resolve_max_retries() + if backend == "ollama" and not os.environ.get("GRAPHIFY_MAX_RETRIES", "").strip(): + _retries = 0 client = OpenAI(api_key=api_key, base_url=base_url, timeout=_resolve_api_timeout(), - max_retries=_resolve_max_retries()) + max_retries=_retries) kwargs: dict = { "model": model, "messages": [ @@ -1483,7 +1517,7 @@ def _estimate_file_tokens(unit: "Path | FileSlice") -> int: content = read_slice_text(unit)[:_FILE_CHAR_CAP] except OSError: return 0 - return len(_TOKENIZER.encode(content)) + (_PER_FILE_OVERHEAD_CHARS // _CHARS_PER_TOKEN) + return len(_TOKENIZER.encode(content, disallowed_special=())) + (_PER_FILE_OVERHEAD_CHARS // _CHARS_PER_TOKEN) path = unit # Raster images are not read as text; a vision model bills them at a roughly @@ -1502,7 +1536,7 @@ def _estimate_file_tokens(unit: "Path | FileSlice") -> int: content = path.read_text(encoding="utf-8", errors="replace")[:_FILE_CHAR_CAP] except OSError: return 0 - return len(_TOKENIZER.encode(content)) + (_PER_FILE_OVERHEAD_CHARS // _CHARS_PER_TOKEN) + return len(_TOKENIZER.encode(content, disallowed_special=())) + (_PER_FILE_OVERHEAD_CHARS // _CHARS_PER_TOKEN) def _pack_chunks_by_tokens( @@ -1862,6 +1896,14 @@ def _run_one(idx: int, chunk: list[Path]) -> tuple[int, dict | None, Exception | if callable(on_chunk_done): on_chunk_done(idx, total, result) else: + # Merge in deterministic submission order, NOT completion order. Merging + # as chunks finish makes the node/edge ordering in the returned corpus + # (and therefore graph.json) depend on which network call happened to + # return first — so identical input churned run-to-run (#1632). Collect + # results keyed by chunk index and merge in sorted order after the pool + # drains; this matches the serial path's order. The progress callback + # still fires in completion order so long local runs aren't silent. + results_by_idx: dict[int, dict] = {} with ThreadPoolExecutor(max_workers=workers) as pool: futures = [pool.submit(_run_one, idx, chunk) for idx, chunk in enumerate(chunks)] for future in as_completed(futures): @@ -1874,9 +1916,11 @@ def _run_one(idx: int, chunk: list[Path]) -> tuple[int, dict | None, Exception | merged["failed_chunks"] += 1 continue assert result is not None - _merge_into(merged, result) + results_by_idx[idx] = result if callable(on_chunk_done): on_chunk_done(idx, total, result) + for idx in sorted(results_by_idx): + _merge_into(merged, results_by_idx[idx]) # Loud failure summary — surface chunk failures at end so they're never # buried mid-log. Exit 0 preserved for caller compatibility; the @@ -1905,9 +1949,15 @@ def _call_llm( backend: str, max_tokens: int = 200, model: str | None = None, + usage_out: dict | None = None, ) -> str: """Send a plain-text prompt to `backend` and return the model's text reply. + When ``usage_out`` is provided it is accumulated in place with ``input`` and + ``output`` token counts from the response, so callers (community labeling) + can total the cost of otherwise-uninstrumented LLM calls (#1694). Existing + callers that omit it are unaffected. + Used by lightweight callers (e.g. `graphify.dedup` LLM tiebreaker) that don't need the full extraction prompt or JSON-shaped output. Mirrors the backend dispatch logic of `extract_files_direct` but skips the @@ -1931,6 +1981,11 @@ def _call_llm( ) mdl = model or _default_model_for_backend(backend) + def _rec(inp, out) -> None: + if usage_out is not None: + usage_out["input"] = usage_out.get("input", 0) + int(inp or 0) + usage_out["output"] = usage_out.get("output", 0) + int(out or 0) + if backend == "claude": try: import anthropic @@ -1942,6 +1997,9 @@ def _call_llm( max_tokens=max_tokens, messages=[{"role": "user", "content": prompt}], ) + u = getattr(resp, "usage", None) + if u is not None: + _rec(getattr(u, "input_tokens", 0), getattr(u, "output_tokens", 0)) return resp.content[0].text if resp.content else "" if backend == "claude-cli": @@ -1975,6 +2033,14 @@ def _call_llm( if proc.returncode != 0: raise RuntimeError(f"claude -p exited {proc.returncode}: {proc.stderr.strip()[:500]}") envelope = _claude_cli_envelope(proc.stdout) + cli_usage = envelope.get("usage") or {} + if cli_usage: + _rec( + (cli_usage.get("input_tokens", 0) or 0) + + (cli_usage.get("cache_read_input_tokens", 0) or 0) + + (cli_usage.get("cache_creation_input_tokens", 0) or 0), + cli_usage.get("output_tokens", 0), + ) return envelope.get("result", "") @@ -1992,6 +2058,9 @@ def _call_llm( messages=[{"role": "user", "content": [{"text": prompt}]}], inferenceConfig=_bedrock_inference_config(max_tokens, mdl), ) + bu = resp.get("usage") or {} + if bu: + _rec(bu.get("inputTokens", 0), bu.get("outputTokens", 0)) return resp.get("output", {}).get("message", {}).get("content", [{}])[0].get("text", "") if backend == "azure": @@ -2012,6 +2081,9 @@ def _call_llm( resp = azure_client.chat.completions.create(**azure_kwargs) if not resp.choices or resp.choices[0].message is None: raise ValueError("Azure OpenAI returned empty or filtered response") + au = getattr(resp, "usage", None) + if au is not None: + _rec(getattr(au, "prompt_tokens", 0), getattr(au, "completion_tokens", 0)) return resp.choices[0].message.content or "" # OpenAI-compatible (kimi, openai, gemini, ollama) @@ -2044,6 +2116,9 @@ def _call_llm( resp = client.chat.completions.create(**kwargs) if not resp.choices or resp.choices[0].message is None: raise ValueError("LLM returned empty or filtered response") + ou = getattr(resp, "usage", None) + if ou is not None: + _rec(getattr(ou, "prompt_tokens", 0), getattr(ou, "completion_tokens", 0)) return resp.choices[0].message.content or "" @@ -2213,9 +2288,25 @@ def _parse_label_response(text: str, labeled_cids: list[int]) -> dict[int, str]: start, end = cleaned.find("{"), cleaned.rfind("}") if start != -1 and end > start: cleaned = cleaned[start:end + 1] - data = json.loads(cleaned) - if not isinstance(data, dict): - raise ValueError("label response is not a JSON object") + data: dict | None = None + try: + parsed = json.loads(cleaned) + if isinstance(parsed, dict): + data = parsed + except (json.JSONDecodeError, ValueError): + data = None + if data is None: + # Salvage: pull the complete "": "" pairs directly. A model + # can truncate its reply mid-object (a stingy token budget or a preamble + # eating the completion), which used to hard-fail the whole batch with + # e.g. `Expecting value: line 1 column 6` on a `{"0":` fragment (#1690). + # Recovering the pairs that DID arrive labels those communities instead + # of dropping the entire batch to placeholders. + pairs = re.findall(r'"?(-?\d+)"?\s*:\s*"([^"\\]*(?:\\.[^"\\]*)*)"', cleaned) + if pairs: + data = {k: v for k, v in pairs} + else: + raise ValueError(f"label response is not parseable JSON: {text[:120]!r}") out: dict[int, str] = {} for cid in labeled_cids: name = data.get(str(cid)) @@ -2234,6 +2325,7 @@ def _label_batch_with_retry( model: str | None, depth: int = 0, max_depth: int = 3, + usage_out: dict | None = None, ) -> dict[int, str]: """Label a batch of communities, splitting in half and retrying on parse failure. @@ -2257,10 +2349,18 @@ def _label_batch_with_retry( "Respond ONLY with a JSON object mapping the community id (as a string) to " "its name - no prose, no markdown fences.\n\n" + "\n".join(batch_lines) ) - max_tokens = _resolve_max_tokens(min(64 + 24 * len(batch_cids), 8192)) + # Budget generously: a 2-5 word name is ~10 tokens, but models (notably + # gemini) often prepend a short preamble or reasoning that eats the + # completion and truncates the JSON mid-object, which used to fail the whole + # batch (#1690). The old 64 + 24*n floor left no headroom. + max_tokens = _resolve_max_tokens(min(256 + 48 * len(batch_cids), 8192)) call_kwargs: dict = {"backend": backend, "max_tokens": max_tokens} if model is not None: call_kwargs["model"] = model + # Only forward usage_out when the caller wants accounting, so existing + # callers (and their test doubles) see the unchanged _call_llm signature. + if usage_out is not None: + call_kwargs["usage_out"] = usage_out try: text = _call_llm(prompt, **call_kwargs) @@ -2281,10 +2381,12 @@ def _label_batch_with_retry( left = _label_batch_with_retry( batch_cids[:mid], batch_lines[:mid], backend=backend, model=model, depth=depth + 1, max_depth=max_depth, + usage_out=usage_out, ) right = _label_batch_with_retry( batch_cids[mid:], batch_lines[mid:], backend=backend, model=model, depth=depth + 1, max_depth=max_depth, + usage_out=usage_out, ) return left | right @@ -2300,6 +2402,7 @@ def label_communities( top_k: int = _LABEL_TOP_K, batch_size: int = _LABEL_BATCH_SIZE, max_concurrency: int = 4, + usage_out: dict | None = None, ) -> dict[int, str]: """Return a complete ``{cid: name}`` map using ``backend`` for naming. @@ -2342,19 +2445,30 @@ def label_communities( def _run_batch(batch_idx: int): start = batch_idx * batch_size end = min(start + batch_size, len(labeled_cids)) + # Accumulate token usage into a per-batch dict so concurrent workers + # never race on the shared accumulator; it is merged on the main thread + # in _merge (#1694). + batch_usage: dict = {} if usage_out is not None else None + batch_kwargs = {"usage_out": batch_usage} if usage_out is not None else {} try: parsed = _label_batch_with_retry( labeled_cids[start:end], lines[start:end], backend=backend, model=model, + **batch_kwargs, ) - return batch_idx, parsed, None + return batch_idx, parsed, None, batch_usage except Exception as exc: # noqa: BLE001 - reported per-batch; surfaced below - return batch_idx, None, exc + return batch_idx, None, exc, batch_usage written = 0 errors: dict[int, Exception] = {} - def _merge(batch_idx: int, parsed, exc) -> None: + def _merge(batch_idx: int, parsed, exc, batch_usage=None) -> None: nonlocal written + # Count tokens even for a failed batch: the LLM call was billed whether + # or not the reply parsed. + if usage_out is not None and batch_usage: + usage_out["input"] = usage_out.get("input", 0) + batch_usage.get("input", 0) + usage_out["output"] = usage_out.get("output", 0) + batch_usage.get("output", 0) if exc is not None: errors[batch_idx] = exc start = batch_idx * batch_size @@ -2396,6 +2510,7 @@ def generate_community_labels( quiet: bool = False, max_concurrency: int = 4, batch_size: int = _LABEL_BATCH_SIZE, + usage_out: dict | None = None, ) -> tuple[dict[int, str], str]: """CLI entry point: resolve a backend, name communities, and degrade to ``Community N`` placeholders on any failure (no backend, API error, malformed @@ -2418,6 +2533,7 @@ def generate_community_labels( labels = label_communities( G, communities, backend=backend, model=model, gods=gods, max_concurrency=max_concurrency, batch_size=batch_size, + usage_out=usage_out, ) return labels, "llm" except Exception as exc: diff --git a/graphify/pg_introspect.py b/graphify/pg_introspect.py index 9182ffeae..24567fa18 100644 --- a/graphify/pg_introspect.py +++ b/graphify/pg_introspect.py @@ -1,5 +1,5 @@ from __future__ import annotations -from pathlib import Path +from pathlib import Path, PurePosixPath from graphify.extract import extract_sql @@ -135,7 +135,7 @@ def introspect_postgres(dsn: str | None = None) -> dict: info = psycopg.conninfo.conninfo_to_dict(dsn or "") host = info.get("host", "localhost") dbname = info.get("dbname", "db") - virtual_path = Path(f"postgresql://{host}/{dbname}") + virtual_path = PurePosixPath(f"postgresql://{host}/{dbname}") # Pass virtual path and in-memory DDL content to extract_sql result = extract_sql(virtual_path, content=ddl_string) diff --git a/graphify/report.py b/graphify/report.py index 1a1d9a365..5bac06d05 100644 --- a/graphify/report.py +++ b/graphify/report.py @@ -176,20 +176,30 @@ def generate( else: lines.append("- None detected - all connections are within the same source files.") - # Circular imports surfaced from file-level dependency graph. - from .analyze import find_import_cycles - cycles = find_import_cycles(G) - lines += ["", "## Import Cycles"] - if cycles: - for c in cycles: - cycle = c.get("cycle", []) - length = c.get("length", len(cycle)) - if not cycle: - continue - cycle_path = " -> ".join(cycle + [cycle[0]]) - lines.append(f"- {length}-file cycle: `{cycle_path}`") - else: - lines.append("- None detected.") + # Circular imports surfaced from file-level dependency graph. Only meaningful + # for code — a documents-only corpus has no imports, so the section is pure + # noise there ("None detected" on every run). Emit it only when the graph + # actually contains code (#1657). + _has_code = any( + d.get("file_type") == "code" for _, d in G.nodes(data=True) + ) or any( + d.get("relation") in ("imports", "imports_from") + for *_e, d in G.edges(data=True) + ) + if _has_code: + from .analyze import find_import_cycles + cycles = find_import_cycles(G) + lines += ["", "## Import Cycles"] + if cycles: + for c in cycles: + cycle = c.get("cycle", []) + length = c.get("length", len(cycle)) + if not cycle: + continue + cycle_path = " -> ".join(cycle + [cycle[0]]) + lines.append(f"- {length}-file cycle: `{cycle_path}`") + else: + lines.append("- None detected.") hyperedges = G.graph.get("hyperedges", []) if hyperedges: diff --git a/graphify/ruby_resolution.py b/graphify/ruby_resolution.py index d0e41dd33..e344175e1 100644 --- a/graphify/ruby_resolution.py +++ b/graphify/ruby_resolution.py @@ -28,6 +28,13 @@ def _key(label: str) -> str: return re.sub(r"[^a-zA-Z0-9]+", "", str(label)).lower() +# A Ruby class/module container node is labelled with a bare constant +# (``Processor``, ``TaxCalculator``); methods end in ``()`` and files in ``.rb``. +# Lets us register method-less containers (a ``Class.new(StandardError)`` error +# class, an empty module) that have no `method` edge to be found by. +_BARE_CONST_RE = re.compile(r"^[A-Z][A-Za-z0-9_]*$") + + def _ruby_raw_calls(per_file: list[dict]) -> list[dict]: calls: list[dict] = [] for result in per_file: @@ -69,6 +76,15 @@ def resolve_ruby_member_calls( tnode = node_by_id.get(tgt) if tnode is not None: method_index[(str(src), _key(tnode.get("label", "")))] = str(tgt) + # Also register class/module container nodes that own no `method` edge — a + # method-less `Class.new(StandardError)` or an empty module — so a constant + # receiver still resolves to a real node (#1640/#1634). External base stubs + # carry an empty source_file, so the `.rb` filter keeps them out. + for n in all_nodes: + nid = n.get("id") + sf = str(n.get("source_file", "")) + if nid and sf.endswith(".rb") and _BARE_CONST_RE.match(str(n.get("label", ""))): + class_def_nids.setdefault(_key(n.get("label", "")), []).append(str(nid)) for k in list(class_def_nids): class_def_nids[k] = sorted(set(class_def_nids[k])) @@ -78,7 +94,8 @@ def _unique_class(name: str) -> str | None: nids = class_def_nids.get(_key(name), []) return nids[0] if len(nids) == 1 else None - def _emit(caller: str, target: str, rc: dict[str, Any]) -> None: + def _emit(caller: str, target: str, rc: dict[str, Any], + relation: str = "calls", context: str = "call") -> None: if not caller or not target or caller == target: return if (caller, target) in existing_pairs: @@ -87,8 +104,8 @@ def _emit(caller: str, target: str, rc: dict[str, Any]) -> None: all_edges.append({ "source": caller, "target": target, - "relation": "calls", - "context": "call", + "relation": relation, + "context": context, "confidence": "EXTRACTED", "confidence_score": 1.0, "source_file": rc.get("source_file", ""), @@ -96,6 +113,21 @@ def _emit(caller: str, target: str, rc: dict[str, Any]) -> None: "weight": 1.0, }) + # `include`/`extend`/`prepend ` mixins (#1668): resolve the module by + # its constant name to the single owning module/class node and emit a + # `mixes_in` edge, under the same single-definition god-node guard. An + # ambiguous or unresolved constant produces no edge. + for rc in _ruby_raw_calls(per_file): + if not rc.get("is_mixin"): + continue + caller = str(rc.get("caller_nid", "")) + module_name = rc.get("callee") + if not caller or not module_name: + continue + target = _unique_class(str(module_name)) + if target is not None: + _emit(caller, target, rc, relation="mixes_in", context="mixin") + for rc in _ruby_raw_calls(per_file): if not rc.get("is_member_call"): continue @@ -104,12 +136,24 @@ def _emit(caller: str, target: str, rc: dict[str, Any]) -> None: if not caller or not callee: continue - # `Processor.new` -> instantiation edge to the class. + # Constant receiver: `Processor.new` (instantiation) or `Service.call` / + # `Model.where` (singleton / class method). The bare method name would + # collide with unrelated same-named methods, so we resolve by the + # receiver's class under the single-owning-class god-node guard. receiver = rc.get("receiver") - if callee == "new" and receiver and str(receiver)[:1].isupper(): + if receiver and str(receiver)[:1].isupper(): class_nid = _unique_class(str(receiver)) if class_nid is not None: - _emit(caller, class_nid, rc) + if callee == "new": + _emit(caller, class_nid, rc) + else: + # Emit to the singleton/instance method the class owns + # (`def self.call`, which the extractor indexes); otherwise + # to the class node itself, so inherited/dynamic class methods + # like ActiveRecord `where`/`find_by` still give correct + # blast-radius. An ambiguous receiver bails to nothing. + method_nid = method_index.get((class_nid, _key(str(callee)))) + _emit(caller, method_nid or class_nid, rc) continue # `p.run` where p's type is known -> edge to that class's method. diff --git a/graphify/skill-agents.md b/graphify/skill-agents.md index 2e19931fa..0d02e4a31 100644 --- a/graphify/skill-agents.md +++ b/graphify/skill-agents.md @@ -474,8 +474,24 @@ Substitute `IS_DIRECTED` and `INPUT_PATH` as in Step 4. If a `GRAPH HEALTH WARNI Read `graphify-out/.graphify_analysis.json`. For each community key, look at its node labels and write a 2-5 word plain-language name (e.g. "Attention Mechanism", "Training Pipeline", "Data Loading"). +**At scale (100+ communities), do NOT hand-label one at a time and do NOT fall back to a naive path-prefix heuristic (e.g. title-casing the common directory prefix).** Both break down past a handful of communities — the heuristic in particular produces duplicated-segment junk ("Backend Backend", "Frontend Frontend") and word-soup names that look plausible until you actually read them. Verified in production on a 1000+ community monorepo graph (apa, 2026-07-03): a heuristic pass needed a full redo. + +What works instead, for communities with >=20 nodes (the ones that actually surface in God Nodes / navigation): +1. For each such community, sample its ~15 highest-degree (most connected) member nodes — not the full member list, communities can have 300+ nodes. +2. Batch communities into groups of ~15-20 per subagent (write each batch's samples to a small JSON file under `graphify-out/`). +3. Dispatch one `general-purpose` subagent per batch (all in the same message, in parallel — same pattern as Step 3 Part B), each told to read its batch file and write a JSON `{community_id: name}` map based on genuine reasoning about the sampled node labels/paths, not just repeating directory names. +4. Merge all batch results. + +For the long tail of small communities (<20 nodes) where subagent cost isn't justified, a path-prefix heuristic is an acceptable lighter-weight fallback — but dedupe repeated words/segments, strip trailing file extensions, and spot-check a random sample of ~15 outputs across different sizes before trusting it. If duplicated-word or generic ("Community N") names appear in the sample, fix the heuristic before applying it to the rest. + Then regenerate the report and save the labels for the visualizer: +**Reconcile the FULL label set for cross-batch duplicates before regenerating the report — this is a required final pass, not optional polish.** Subagent batches only see their own slice and the small-community heuristic naturally collapses siblings from the same directory (e.g. many different test-file clusters under the same folder), so duplicates are expected even after the steps above — verified in production: 338 of 1083 communities (31%) shared a label with at least one other community before reconciliation (apa monorepo, 2026-07-03). After all labels are assembled: +1. Group community IDs by their current label. +2. For every group with >1 member, pick a distinguishing keyword from each member's OWN node labels (word-frequency count, excluding stopwords and words already in the shared base label) and append it, e.g. `"Agent Gateway MCP Client" -> "Agent Gateway MCP Client — Server"` / `"— Additional"`. +3. If a collision still remains after that (rare — e.g. multiple communities extracted from the very same source image), append a numeric suffix `" (1)"`, `" (2)"` as a last resort. +4. Re-check for zero duplicates before moving on. + ```bash $(cat graphify-out/.graphify_python) -c " import sys, json @@ -531,6 +547,8 @@ graphify export html # auto-aggregates to community view if graph > 5000 nodes # or: graphify export html --no-viz ``` +**If the resulting view (aggregated community graph, or the raw graph for small corpora) has more than ~300 nodes, ALSO generate a Sigma.js + graphology version and present that as the primary interactive file instead of vis-network's `graph.html`.** vis-network runs a live single-threaded JS forceAtlas2 physics simulation on load (200 iterations before it disables itself) — verified in production this is genuinely slow/laggy past a few hundred nodes (apa monorepo, 1083-community aggregated view, 2026-07-03), independent of hardware. Sigma.js (WebGL rendering) with the layout **precomputed offline in Python** (`nx.forceatlas2_layout`, no client-side physics at all) fixes this at the root rather than swapping one JS physics engine for another. See `references/sigma-viz.md` for the full recipe (meta-graph construction, offline layout, self-contained HTML template loading sigma/graphology from esm.sh). Below the 300-node threshold, vis-network's `graph.html` is fine as-is — the extra build step isn't worth it. + ### Steps 6b-8 - Wiki, Neo4j, FalkorDB, SVG, GraphML, MCP, benchmark (only on their flags) These run only when their flag is present (`--wiki`, `--neo4j`/`--neo4j-push`, `--falkordb`/`--falkordb-push`, `--svg`, `--graphml`, `--mcp`) or, for the token-reduction benchmark, when `total_words` exceeds 5,000. A default run with no export flags skips all of them. See `references/exports.md` for each one. Run any `--wiki` export before Step 9 cleanup so `.graphify_labels.json` is still available. diff --git a/graphify/skill-amp.md b/graphify/skill-amp.md index 2e19931fa..0d02e4a31 100644 --- a/graphify/skill-amp.md +++ b/graphify/skill-amp.md @@ -474,8 +474,24 @@ Substitute `IS_DIRECTED` and `INPUT_PATH` as in Step 4. If a `GRAPH HEALTH WARNI Read `graphify-out/.graphify_analysis.json`. For each community key, look at its node labels and write a 2-5 word plain-language name (e.g. "Attention Mechanism", "Training Pipeline", "Data Loading"). +**At scale (100+ communities), do NOT hand-label one at a time and do NOT fall back to a naive path-prefix heuristic (e.g. title-casing the common directory prefix).** Both break down past a handful of communities — the heuristic in particular produces duplicated-segment junk ("Backend Backend", "Frontend Frontend") and word-soup names that look plausible until you actually read them. Verified in production on a 1000+ community monorepo graph (apa, 2026-07-03): a heuristic pass needed a full redo. + +What works instead, for communities with >=20 nodes (the ones that actually surface in God Nodes / navigation): +1. For each such community, sample its ~15 highest-degree (most connected) member nodes — not the full member list, communities can have 300+ nodes. +2. Batch communities into groups of ~15-20 per subagent (write each batch's samples to a small JSON file under `graphify-out/`). +3. Dispatch one `general-purpose` subagent per batch (all in the same message, in parallel — same pattern as Step 3 Part B), each told to read its batch file and write a JSON `{community_id: name}` map based on genuine reasoning about the sampled node labels/paths, not just repeating directory names. +4. Merge all batch results. + +For the long tail of small communities (<20 nodes) where subagent cost isn't justified, a path-prefix heuristic is an acceptable lighter-weight fallback — but dedupe repeated words/segments, strip trailing file extensions, and spot-check a random sample of ~15 outputs across different sizes before trusting it. If duplicated-word or generic ("Community N") names appear in the sample, fix the heuristic before applying it to the rest. + Then regenerate the report and save the labels for the visualizer: +**Reconcile the FULL label set for cross-batch duplicates before regenerating the report — this is a required final pass, not optional polish.** Subagent batches only see their own slice and the small-community heuristic naturally collapses siblings from the same directory (e.g. many different test-file clusters under the same folder), so duplicates are expected even after the steps above — verified in production: 338 of 1083 communities (31%) shared a label with at least one other community before reconciliation (apa monorepo, 2026-07-03). After all labels are assembled: +1. Group community IDs by their current label. +2. For every group with >1 member, pick a distinguishing keyword from each member's OWN node labels (word-frequency count, excluding stopwords and words already in the shared base label) and append it, e.g. `"Agent Gateway MCP Client" -> "Agent Gateway MCP Client — Server"` / `"— Additional"`. +3. If a collision still remains after that (rare — e.g. multiple communities extracted from the very same source image), append a numeric suffix `" (1)"`, `" (2)"` as a last resort. +4. Re-check for zero duplicates before moving on. + ```bash $(cat graphify-out/.graphify_python) -c " import sys, json @@ -531,6 +547,8 @@ graphify export html # auto-aggregates to community view if graph > 5000 nodes # or: graphify export html --no-viz ``` +**If the resulting view (aggregated community graph, or the raw graph for small corpora) has more than ~300 nodes, ALSO generate a Sigma.js + graphology version and present that as the primary interactive file instead of vis-network's `graph.html`.** vis-network runs a live single-threaded JS forceAtlas2 physics simulation on load (200 iterations before it disables itself) — verified in production this is genuinely slow/laggy past a few hundred nodes (apa monorepo, 1083-community aggregated view, 2026-07-03), independent of hardware. Sigma.js (WebGL rendering) with the layout **precomputed offline in Python** (`nx.forceatlas2_layout`, no client-side physics at all) fixes this at the root rather than swapping one JS physics engine for another. See `references/sigma-viz.md` for the full recipe (meta-graph construction, offline layout, self-contained HTML template loading sigma/graphology from esm.sh). Below the 300-node threshold, vis-network's `graph.html` is fine as-is — the extra build step isn't worth it. + ### Steps 6b-8 - Wiki, Neo4j, FalkorDB, SVG, GraphML, MCP, benchmark (only on their flags) These run only when their flag is present (`--wiki`, `--neo4j`/`--neo4j-push`, `--falkordb`/`--falkordb-push`, `--svg`, `--graphml`, `--mcp`) or, for the token-reduction benchmark, when `total_words` exceeds 5,000. A default run with no export flags skips all of them. See `references/exports.md` for each one. Run any `--wiki` export before Step 9 cleanup so `.graphify_labels.json` is still available. diff --git a/graphify/skill-claw.md b/graphify/skill-claw.md index b3542438d..5c337677e 100644 --- a/graphify/skill-claw.md +++ b/graphify/skill-claw.md @@ -477,8 +477,24 @@ Substitute `IS_DIRECTED` and `INPUT_PATH` as in Step 4. If a `GRAPH HEALTH WARNI Read `graphify-out/.graphify_analysis.json`. For each community key, look at its node labels and write a 2-5 word plain-language name (e.g. "Attention Mechanism", "Training Pipeline", "Data Loading"). +**At scale (100+ communities), do NOT hand-label one at a time and do NOT fall back to a naive path-prefix heuristic (e.g. title-casing the common directory prefix).** Both break down past a handful of communities — the heuristic in particular produces duplicated-segment junk ("Backend Backend", "Frontend Frontend") and word-soup names that look plausible until you actually read them. Verified in production on a 1000+ community monorepo graph (apa, 2026-07-03): a heuristic pass needed a full redo. + +What works instead, for communities with >=20 nodes (the ones that actually surface in God Nodes / navigation): +1. For each such community, sample its ~15 highest-degree (most connected) member nodes — not the full member list, communities can have 300+ nodes. +2. Batch communities into groups of ~15-20 per subagent (write each batch's samples to a small JSON file under `graphify-out/`). +3. Dispatch one `general-purpose` subagent per batch (all in the same message, in parallel — same pattern as Step 3 Part B), each told to read its batch file and write a JSON `{community_id: name}` map based on genuine reasoning about the sampled node labels/paths, not just repeating directory names. +4. Merge all batch results. + +For the long tail of small communities (<20 nodes) where subagent cost isn't justified, a path-prefix heuristic is an acceptable lighter-weight fallback — but dedupe repeated words/segments, strip trailing file extensions, and spot-check a random sample of ~15 outputs across different sizes before trusting it. If duplicated-word or generic ("Community N") names appear in the sample, fix the heuristic before applying it to the rest. + Then regenerate the report and save the labels for the visualizer: +**Reconcile the FULL label set for cross-batch duplicates before regenerating the report — this is a required final pass, not optional polish.** Subagent batches only see their own slice and the small-community heuristic naturally collapses siblings from the same directory (e.g. many different test-file clusters under the same folder), so duplicates are expected even after the steps above — verified in production: 338 of 1083 communities (31%) shared a label with at least one other community before reconciliation (apa monorepo, 2026-07-03). After all labels are assembled: +1. Group community IDs by their current label. +2. For every group with >1 member, pick a distinguishing keyword from each member's OWN node labels (word-frequency count, excluding stopwords and words already in the shared base label) and append it, e.g. `"Agent Gateway MCP Client" -> "Agent Gateway MCP Client — Server"` / `"— Additional"`. +3. If a collision still remains after that (rare — e.g. multiple communities extracted from the very same source image), append a numeric suffix `" (1)"`, `" (2)"` as a last resort. +4. Re-check for zero duplicates before moving on. + ```bash $(cat graphify-out/.graphify_python) -c " import sys, json @@ -534,6 +550,8 @@ graphify export html # auto-aggregates to community view if graph > 5000 nodes # or: graphify export html --no-viz ``` +**If the resulting view (aggregated community graph, or the raw graph for small corpora) has more than ~300 nodes, ALSO generate a Sigma.js + graphology version and present that as the primary interactive file instead of vis-network's `graph.html`.** vis-network runs a live single-threaded JS forceAtlas2 physics simulation on load (200 iterations before it disables itself) — verified in production this is genuinely slow/laggy past a few hundred nodes (apa monorepo, 1083-community aggregated view, 2026-07-03), independent of hardware. Sigma.js (WebGL rendering) with the layout **precomputed offline in Python** (`nx.forceatlas2_layout`, no client-side physics at all) fixes this at the root rather than swapping one JS physics engine for another. See `references/sigma-viz.md` for the full recipe (meta-graph construction, offline layout, self-contained HTML template loading sigma/graphology from esm.sh). Below the 300-node threshold, vis-network's `graph.html` is fine as-is — the extra build step isn't worth it. + ### Steps 6b-8 - Wiki, Neo4j, FalkorDB, SVG, GraphML, MCP, benchmark (only on their flags) These run only when their flag is present (`--wiki`, `--neo4j`/`--neo4j-push`, `--falkordb`/`--falkordb-push`, `--svg`, `--graphml`, `--mcp`) or, for the token-reduction benchmark, when `total_words` exceeds 5,000. A default run with no export flags skips all of them. See `references/exports.md` for each one. Run any `--wiki` export before Step 9 cleanup so `.graphify_labels.json` is still available. diff --git a/graphify/skill-codex.md b/graphify/skill-codex.md index e6e411ea0..f28ef4cd3 100644 --- a/graphify/skill-codex.md +++ b/graphify/skill-codex.md @@ -474,8 +474,24 @@ Substitute `IS_DIRECTED` and `INPUT_PATH` as in Step 4. If a `GRAPH HEALTH WARNI Read `graphify-out/.graphify_analysis.json`. For each community key, look at its node labels and write a 2-5 word plain-language name (e.g. "Attention Mechanism", "Training Pipeline", "Data Loading"). +**At scale (100+ communities), do NOT hand-label one at a time and do NOT fall back to a naive path-prefix heuristic (e.g. title-casing the common directory prefix).** Both break down past a handful of communities — the heuristic in particular produces duplicated-segment junk ("Backend Backend", "Frontend Frontend") and word-soup names that look plausible until you actually read them. Verified in production on a 1000+ community monorepo graph (apa, 2026-07-03): a heuristic pass needed a full redo. + +What works instead, for communities with >=20 nodes (the ones that actually surface in God Nodes / navigation): +1. For each such community, sample its ~15 highest-degree (most connected) member nodes — not the full member list, communities can have 300+ nodes. +2. Batch communities into groups of ~15-20 per subagent (write each batch's samples to a small JSON file under `graphify-out/`). +3. Dispatch one `general-purpose` subagent per batch (all in the same message, in parallel — same pattern as Step 3 Part B), each told to read its batch file and write a JSON `{community_id: name}` map based on genuine reasoning about the sampled node labels/paths, not just repeating directory names. +4. Merge all batch results. + +For the long tail of small communities (<20 nodes) where subagent cost isn't justified, a path-prefix heuristic is an acceptable lighter-weight fallback — but dedupe repeated words/segments, strip trailing file extensions, and spot-check a random sample of ~15 outputs across different sizes before trusting it. If duplicated-word or generic ("Community N") names appear in the sample, fix the heuristic before applying it to the rest. + Then regenerate the report and save the labels for the visualizer: +**Reconcile the FULL label set for cross-batch duplicates before regenerating the report — this is a required final pass, not optional polish.** Subagent batches only see their own slice and the small-community heuristic naturally collapses siblings from the same directory (e.g. many different test-file clusters under the same folder), so duplicates are expected even after the steps above — verified in production: 338 of 1083 communities (31%) shared a label with at least one other community before reconciliation (apa monorepo, 2026-07-03). After all labels are assembled: +1. Group community IDs by their current label. +2. For every group with >1 member, pick a distinguishing keyword from each member's OWN node labels (word-frequency count, excluding stopwords and words already in the shared base label) and append it, e.g. `"Agent Gateway MCP Client" -> "Agent Gateway MCP Client — Server"` / `"— Additional"`. +3. If a collision still remains after that (rare — e.g. multiple communities extracted from the very same source image), append a numeric suffix `" (1)"`, `" (2)"` as a last resort. +4. Re-check for zero duplicates before moving on. + ```bash $(cat graphify-out/.graphify_python) -c " import sys, json @@ -531,6 +547,8 @@ graphify export html # auto-aggregates to community view if graph > 5000 nodes # or: graphify export html --no-viz ``` +**If the resulting view (aggregated community graph, or the raw graph for small corpora) has more than ~300 nodes, ALSO generate a Sigma.js + graphology version and present that as the primary interactive file instead of vis-network's `graph.html`.** vis-network runs a live single-threaded JS forceAtlas2 physics simulation on load (200 iterations before it disables itself) — verified in production this is genuinely slow/laggy past a few hundred nodes (apa monorepo, 1083-community aggregated view, 2026-07-03), independent of hardware. Sigma.js (WebGL rendering) with the layout **precomputed offline in Python** (`nx.forceatlas2_layout`, no client-side physics at all) fixes this at the root rather than swapping one JS physics engine for another. See `references/sigma-viz.md` for the full recipe (meta-graph construction, offline layout, self-contained HTML template loading sigma/graphology from esm.sh). Below the 300-node threshold, vis-network's `graph.html` is fine as-is — the extra build step isn't worth it. + ### Steps 6b-8 - Wiki, Neo4j, FalkorDB, SVG, GraphML, MCP, benchmark (only on their flags) These run only when their flag is present (`--wiki`, `--neo4j`/`--neo4j-push`, `--falkordb`/`--falkordb-push`, `--svg`, `--graphml`, `--mcp`) or, for the token-reduction benchmark, when `total_words` exceeds 5,000. A default run with no export flags skips all of them. See `references/exports.md` for each one. Run any `--wiki` export before Step 9 cleanup so `.graphify_labels.json` is still available. diff --git a/graphify/skill-copilot.md b/graphify/skill-copilot.md index b3542438d..5c337677e 100644 --- a/graphify/skill-copilot.md +++ b/graphify/skill-copilot.md @@ -477,8 +477,24 @@ Substitute `IS_DIRECTED` and `INPUT_PATH` as in Step 4. If a `GRAPH HEALTH WARNI Read `graphify-out/.graphify_analysis.json`. For each community key, look at its node labels and write a 2-5 word plain-language name (e.g. "Attention Mechanism", "Training Pipeline", "Data Loading"). +**At scale (100+ communities), do NOT hand-label one at a time and do NOT fall back to a naive path-prefix heuristic (e.g. title-casing the common directory prefix).** Both break down past a handful of communities — the heuristic in particular produces duplicated-segment junk ("Backend Backend", "Frontend Frontend") and word-soup names that look plausible until you actually read them. Verified in production on a 1000+ community monorepo graph (apa, 2026-07-03): a heuristic pass needed a full redo. + +What works instead, for communities with >=20 nodes (the ones that actually surface in God Nodes / navigation): +1. For each such community, sample its ~15 highest-degree (most connected) member nodes — not the full member list, communities can have 300+ nodes. +2. Batch communities into groups of ~15-20 per subagent (write each batch's samples to a small JSON file under `graphify-out/`). +3. Dispatch one `general-purpose` subagent per batch (all in the same message, in parallel — same pattern as Step 3 Part B), each told to read its batch file and write a JSON `{community_id: name}` map based on genuine reasoning about the sampled node labels/paths, not just repeating directory names. +4. Merge all batch results. + +For the long tail of small communities (<20 nodes) where subagent cost isn't justified, a path-prefix heuristic is an acceptable lighter-weight fallback — but dedupe repeated words/segments, strip trailing file extensions, and spot-check a random sample of ~15 outputs across different sizes before trusting it. If duplicated-word or generic ("Community N") names appear in the sample, fix the heuristic before applying it to the rest. + Then regenerate the report and save the labels for the visualizer: +**Reconcile the FULL label set for cross-batch duplicates before regenerating the report — this is a required final pass, not optional polish.** Subagent batches only see their own slice and the small-community heuristic naturally collapses siblings from the same directory (e.g. many different test-file clusters under the same folder), so duplicates are expected even after the steps above — verified in production: 338 of 1083 communities (31%) shared a label with at least one other community before reconciliation (apa monorepo, 2026-07-03). After all labels are assembled: +1. Group community IDs by their current label. +2. For every group with >1 member, pick a distinguishing keyword from each member's OWN node labels (word-frequency count, excluding stopwords and words already in the shared base label) and append it, e.g. `"Agent Gateway MCP Client" -> "Agent Gateway MCP Client — Server"` / `"— Additional"`. +3. If a collision still remains after that (rare — e.g. multiple communities extracted from the very same source image), append a numeric suffix `" (1)"`, `" (2)"` as a last resort. +4. Re-check for zero duplicates before moving on. + ```bash $(cat graphify-out/.graphify_python) -c " import sys, json @@ -534,6 +550,8 @@ graphify export html # auto-aggregates to community view if graph > 5000 nodes # or: graphify export html --no-viz ``` +**If the resulting view (aggregated community graph, or the raw graph for small corpora) has more than ~300 nodes, ALSO generate a Sigma.js + graphology version and present that as the primary interactive file instead of vis-network's `graph.html`.** vis-network runs a live single-threaded JS forceAtlas2 physics simulation on load (200 iterations before it disables itself) — verified in production this is genuinely slow/laggy past a few hundred nodes (apa monorepo, 1083-community aggregated view, 2026-07-03), independent of hardware. Sigma.js (WebGL rendering) with the layout **precomputed offline in Python** (`nx.forceatlas2_layout`, no client-side physics at all) fixes this at the root rather than swapping one JS physics engine for another. See `references/sigma-viz.md` for the full recipe (meta-graph construction, offline layout, self-contained HTML template loading sigma/graphology from esm.sh). Below the 300-node threshold, vis-network's `graph.html` is fine as-is — the extra build step isn't worth it. + ### Steps 6b-8 - Wiki, Neo4j, FalkorDB, SVG, GraphML, MCP, benchmark (only on their flags) These run only when their flag is present (`--wiki`, `--neo4j`/`--neo4j-push`, `--falkordb`/`--falkordb-push`, `--svg`, `--graphml`, `--mcp`) or, for the token-reduction benchmark, when `total_words` exceeds 5,000. A default run with no export flags skips all of them. See `references/exports.md` for each one. Run any `--wiki` export before Step 9 cleanup so `.graphify_labels.json` is still available. diff --git a/graphify/skill-droid.md b/graphify/skill-droid.md index d1147b903..afa602ccc 100644 --- a/graphify/skill-droid.md +++ b/graphify/skill-droid.md @@ -474,8 +474,24 @@ Substitute `IS_DIRECTED` and `INPUT_PATH` as in Step 4. If a `GRAPH HEALTH WARNI Read `graphify-out/.graphify_analysis.json`. For each community key, look at its node labels and write a 2-5 word plain-language name (e.g. "Attention Mechanism", "Training Pipeline", "Data Loading"). +**At scale (100+ communities), do NOT hand-label one at a time and do NOT fall back to a naive path-prefix heuristic (e.g. title-casing the common directory prefix).** Both break down past a handful of communities — the heuristic in particular produces duplicated-segment junk ("Backend Backend", "Frontend Frontend") and word-soup names that look plausible until you actually read them. Verified in production on a 1000+ community monorepo graph (apa, 2026-07-03): a heuristic pass needed a full redo. + +What works instead, for communities with >=20 nodes (the ones that actually surface in God Nodes / navigation): +1. For each such community, sample its ~15 highest-degree (most connected) member nodes — not the full member list, communities can have 300+ nodes. +2. Batch communities into groups of ~15-20 per subagent (write each batch's samples to a small JSON file under `graphify-out/`). +3. Dispatch one `general-purpose` subagent per batch (all in the same message, in parallel — same pattern as Step 3 Part B), each told to read its batch file and write a JSON `{community_id: name}` map based on genuine reasoning about the sampled node labels/paths, not just repeating directory names. +4. Merge all batch results. + +For the long tail of small communities (<20 nodes) where subagent cost isn't justified, a path-prefix heuristic is an acceptable lighter-weight fallback — but dedupe repeated words/segments, strip trailing file extensions, and spot-check a random sample of ~15 outputs across different sizes before trusting it. If duplicated-word or generic ("Community N") names appear in the sample, fix the heuristic before applying it to the rest. + Then regenerate the report and save the labels for the visualizer: +**Reconcile the FULL label set for cross-batch duplicates before regenerating the report — this is a required final pass, not optional polish.** Subagent batches only see their own slice and the small-community heuristic naturally collapses siblings from the same directory (e.g. many different test-file clusters under the same folder), so duplicates are expected even after the steps above — verified in production: 338 of 1083 communities (31%) shared a label with at least one other community before reconciliation (apa monorepo, 2026-07-03). After all labels are assembled: +1. Group community IDs by their current label. +2. For every group with >1 member, pick a distinguishing keyword from each member's OWN node labels (word-frequency count, excluding stopwords and words already in the shared base label) and append it, e.g. `"Agent Gateway MCP Client" -> "Agent Gateway MCP Client — Server"` / `"— Additional"`. +3. If a collision still remains after that (rare — e.g. multiple communities extracted from the very same source image), append a numeric suffix `" (1)"`, `" (2)"` as a last resort. +4. Re-check for zero duplicates before moving on. + ```bash $(cat graphify-out/.graphify_python) -c " import sys, json @@ -531,6 +547,8 @@ graphify export html # auto-aggregates to community view if graph > 5000 nodes # or: graphify export html --no-viz ``` +**If the resulting view (aggregated community graph, or the raw graph for small corpora) has more than ~300 nodes, ALSO generate a Sigma.js + graphology version and present that as the primary interactive file instead of vis-network's `graph.html`.** vis-network runs a live single-threaded JS forceAtlas2 physics simulation on load (200 iterations before it disables itself) — verified in production this is genuinely slow/laggy past a few hundred nodes (apa monorepo, 1083-community aggregated view, 2026-07-03), independent of hardware. Sigma.js (WebGL rendering) with the layout **precomputed offline in Python** (`nx.forceatlas2_layout`, no client-side physics at all) fixes this at the root rather than swapping one JS physics engine for another. See `references/sigma-viz.md` for the full recipe (meta-graph construction, offline layout, self-contained HTML template loading sigma/graphology from esm.sh). Below the 300-node threshold, vis-network's `graph.html` is fine as-is — the extra build step isn't worth it. + ### Steps 6b-8 - Wiki, Neo4j, FalkorDB, SVG, GraphML, MCP, benchmark (only on their flags) These run only when their flag is present (`--wiki`, `--neo4j`/`--neo4j-push`, `--falkordb`/`--falkordb-push`, `--svg`, `--graphml`, `--mcp`) or, for the token-reduction benchmark, when `total_words` exceeds 5,000. A default run with no export flags skips all of them. See `references/exports.md` for each one. Run any `--wiki` export before Step 9 cleanup so `.graphify_labels.json` is still available. diff --git a/graphify/skill-kilo.md b/graphify/skill-kilo.md index c45578165..b4892b39c 100644 --- a/graphify/skill-kilo.md +++ b/graphify/skill-kilo.md @@ -477,8 +477,24 @@ Substitute `IS_DIRECTED` and `INPUT_PATH` as in Step 4. If a `GRAPH HEALTH WARNI Read `graphify-out/.graphify_analysis.json`. For each community key, look at its node labels and write a 2-5 word plain-language name (e.g. "Attention Mechanism", "Training Pipeline", "Data Loading"). +**At scale (100+ communities), do NOT hand-label one at a time and do NOT fall back to a naive path-prefix heuristic (e.g. title-casing the common directory prefix).** Both break down past a handful of communities — the heuristic in particular produces duplicated-segment junk ("Backend Backend", "Frontend Frontend") and word-soup names that look plausible until you actually read them. Verified in production on a 1000+ community monorepo graph (apa, 2026-07-03): a heuristic pass needed a full redo. + +What works instead, for communities with >=20 nodes (the ones that actually surface in God Nodes / navigation): +1. For each such community, sample its ~15 highest-degree (most connected) member nodes — not the full member list, communities can have 300+ nodes. +2. Batch communities into groups of ~15-20 per subagent (write each batch's samples to a small JSON file under `graphify-out/`). +3. Dispatch one `general-purpose` subagent per batch (all in the same message, in parallel — same pattern as Step 3 Part B), each told to read its batch file and write a JSON `{community_id: name}` map based on genuine reasoning about the sampled node labels/paths, not just repeating directory names. +4. Merge all batch results. + +For the long tail of small communities (<20 nodes) where subagent cost isn't justified, a path-prefix heuristic is an acceptable lighter-weight fallback — but dedupe repeated words/segments, strip trailing file extensions, and spot-check a random sample of ~15 outputs across different sizes before trusting it. If duplicated-word or generic ("Community N") names appear in the sample, fix the heuristic before applying it to the rest. + Then regenerate the report and save the labels for the visualizer: +**Reconcile the FULL label set for cross-batch duplicates before regenerating the report — this is a required final pass, not optional polish.** Subagent batches only see their own slice and the small-community heuristic naturally collapses siblings from the same directory (e.g. many different test-file clusters under the same folder), so duplicates are expected even after the steps above — verified in production: 338 of 1083 communities (31%) shared a label with at least one other community before reconciliation (apa monorepo, 2026-07-03). After all labels are assembled: +1. Group community IDs by their current label. +2. For every group with >1 member, pick a distinguishing keyword from each member's OWN node labels (word-frequency count, excluding stopwords and words already in the shared base label) and append it, e.g. `"Agent Gateway MCP Client" -> "Agent Gateway MCP Client — Server"` / `"— Additional"`. +3. If a collision still remains after that (rare — e.g. multiple communities extracted from the very same source image), append a numeric suffix `" (1)"`, `" (2)"` as a last resort. +4. Re-check for zero duplicates before moving on. + ```bash $(cat graphify-out/.graphify_python) -c " import sys, json @@ -534,6 +550,8 @@ graphify export html # auto-aggregates to community view if graph > 5000 nodes # or: graphify export html --no-viz ``` +**If the resulting view (aggregated community graph, or the raw graph for small corpora) has more than ~300 nodes, ALSO generate a Sigma.js + graphology version and present that as the primary interactive file instead of vis-network's `graph.html`.** vis-network runs a live single-threaded JS forceAtlas2 physics simulation on load (200 iterations before it disables itself) — verified in production this is genuinely slow/laggy past a few hundred nodes (apa monorepo, 1083-community aggregated view, 2026-07-03), independent of hardware. Sigma.js (WebGL rendering) with the layout **precomputed offline in Python** (`nx.forceatlas2_layout`, no client-side physics at all) fixes this at the root rather than swapping one JS physics engine for another. See `references/sigma-viz.md` for the full recipe (meta-graph construction, offline layout, self-contained HTML template loading sigma/graphology from esm.sh). Below the 300-node threshold, vis-network's `graph.html` is fine as-is — the extra build step isn't worth it. + ### Steps 6b-8 - Wiki, Neo4j, FalkorDB, SVG, GraphML, MCP, benchmark (only on their flags) These run only when their flag is present (`--wiki`, `--neo4j`/`--neo4j-push`, `--falkordb`/`--falkordb-push`, `--svg`, `--graphml`, `--mcp`) or, for the token-reduction benchmark, when `total_words` exceeds 5,000. A default run with no export flags skips all of them. See `references/exports.md` for each one. Run any `--wiki` export before Step 9 cleanup so `.graphify_labels.json` is still available. diff --git a/graphify/skill-kiro.md b/graphify/skill-kiro.md index b3542438d..5c337677e 100644 --- a/graphify/skill-kiro.md +++ b/graphify/skill-kiro.md @@ -477,8 +477,24 @@ Substitute `IS_DIRECTED` and `INPUT_PATH` as in Step 4. If a `GRAPH HEALTH WARNI Read `graphify-out/.graphify_analysis.json`. For each community key, look at its node labels and write a 2-5 word plain-language name (e.g. "Attention Mechanism", "Training Pipeline", "Data Loading"). +**At scale (100+ communities), do NOT hand-label one at a time and do NOT fall back to a naive path-prefix heuristic (e.g. title-casing the common directory prefix).** Both break down past a handful of communities — the heuristic in particular produces duplicated-segment junk ("Backend Backend", "Frontend Frontend") and word-soup names that look plausible until you actually read them. Verified in production on a 1000+ community monorepo graph (apa, 2026-07-03): a heuristic pass needed a full redo. + +What works instead, for communities with >=20 nodes (the ones that actually surface in God Nodes / navigation): +1. For each such community, sample its ~15 highest-degree (most connected) member nodes — not the full member list, communities can have 300+ nodes. +2. Batch communities into groups of ~15-20 per subagent (write each batch's samples to a small JSON file under `graphify-out/`). +3. Dispatch one `general-purpose` subagent per batch (all in the same message, in parallel — same pattern as Step 3 Part B), each told to read its batch file and write a JSON `{community_id: name}` map based on genuine reasoning about the sampled node labels/paths, not just repeating directory names. +4. Merge all batch results. + +For the long tail of small communities (<20 nodes) where subagent cost isn't justified, a path-prefix heuristic is an acceptable lighter-weight fallback — but dedupe repeated words/segments, strip trailing file extensions, and spot-check a random sample of ~15 outputs across different sizes before trusting it. If duplicated-word or generic ("Community N") names appear in the sample, fix the heuristic before applying it to the rest. + Then regenerate the report and save the labels for the visualizer: +**Reconcile the FULL label set for cross-batch duplicates before regenerating the report — this is a required final pass, not optional polish.** Subagent batches only see their own slice and the small-community heuristic naturally collapses siblings from the same directory (e.g. many different test-file clusters under the same folder), so duplicates are expected even after the steps above — verified in production: 338 of 1083 communities (31%) shared a label with at least one other community before reconciliation (apa monorepo, 2026-07-03). After all labels are assembled: +1. Group community IDs by their current label. +2. For every group with >1 member, pick a distinguishing keyword from each member's OWN node labels (word-frequency count, excluding stopwords and words already in the shared base label) and append it, e.g. `"Agent Gateway MCP Client" -> "Agent Gateway MCP Client — Server"` / `"— Additional"`. +3. If a collision still remains after that (rare — e.g. multiple communities extracted from the very same source image), append a numeric suffix `" (1)"`, `" (2)"` as a last resort. +4. Re-check for zero duplicates before moving on. + ```bash $(cat graphify-out/.graphify_python) -c " import sys, json @@ -534,6 +550,8 @@ graphify export html # auto-aggregates to community view if graph > 5000 nodes # or: graphify export html --no-viz ``` +**If the resulting view (aggregated community graph, or the raw graph for small corpora) has more than ~300 nodes, ALSO generate a Sigma.js + graphology version and present that as the primary interactive file instead of vis-network's `graph.html`.** vis-network runs a live single-threaded JS forceAtlas2 physics simulation on load (200 iterations before it disables itself) — verified in production this is genuinely slow/laggy past a few hundred nodes (apa monorepo, 1083-community aggregated view, 2026-07-03), independent of hardware. Sigma.js (WebGL rendering) with the layout **precomputed offline in Python** (`nx.forceatlas2_layout`, no client-side physics at all) fixes this at the root rather than swapping one JS physics engine for another. See `references/sigma-viz.md` for the full recipe (meta-graph construction, offline layout, self-contained HTML template loading sigma/graphology from esm.sh). Below the 300-node threshold, vis-network's `graph.html` is fine as-is — the extra build step isn't worth it. + ### Steps 6b-8 - Wiki, Neo4j, FalkorDB, SVG, GraphML, MCP, benchmark (only on their flags) These run only when their flag is present (`--wiki`, `--neo4j`/`--neo4j-push`, `--falkordb`/`--falkordb-push`, `--svg`, `--graphml`, `--mcp`) or, for the token-reduction benchmark, when `total_words` exceeds 5,000. A default run with no export flags skips all of them. See `references/exports.md` for each one. Run any `--wiki` export before Step 9 cleanup so `.graphify_labels.json` is still available. diff --git a/graphify/skill-opencode.md b/graphify/skill-opencode.md index 684e0e223..e923319b3 100644 --- a/graphify/skill-opencode.md +++ b/graphify/skill-opencode.md @@ -469,8 +469,24 @@ Substitute `IS_DIRECTED` and `INPUT_PATH` as in Step 4. If a `GRAPH HEALTH WARNI Read `graphify-out/.graphify_analysis.json`. For each community key, look at its node labels and write a 2-5 word plain-language name (e.g. "Attention Mechanism", "Training Pipeline", "Data Loading"). +**At scale (100+ communities), do NOT hand-label one at a time and do NOT fall back to a naive path-prefix heuristic (e.g. title-casing the common directory prefix).** Both break down past a handful of communities — the heuristic in particular produces duplicated-segment junk ("Backend Backend", "Frontend Frontend") and word-soup names that look plausible until you actually read them. Verified in production on a 1000+ community monorepo graph (apa, 2026-07-03): a heuristic pass needed a full redo. + +What works instead, for communities with >=20 nodes (the ones that actually surface in God Nodes / navigation): +1. For each such community, sample its ~15 highest-degree (most connected) member nodes — not the full member list, communities can have 300+ nodes. +2. Batch communities into groups of ~15-20 per subagent (write each batch's samples to a small JSON file under `graphify-out/`). +3. Dispatch one `general-purpose` subagent per batch (all in the same message, in parallel — same pattern as Step 3 Part B), each told to read its batch file and write a JSON `{community_id: name}` map based on genuine reasoning about the sampled node labels/paths, not just repeating directory names. +4. Merge all batch results. + +For the long tail of small communities (<20 nodes) where subagent cost isn't justified, a path-prefix heuristic is an acceptable lighter-weight fallback — but dedupe repeated words/segments, strip trailing file extensions, and spot-check a random sample of ~15 outputs across different sizes before trusting it. If duplicated-word or generic ("Community N") names appear in the sample, fix the heuristic before applying it to the rest. + Then regenerate the report and save the labels for the visualizer: +**Reconcile the FULL label set for cross-batch duplicates before regenerating the report — this is a required final pass, not optional polish.** Subagent batches only see their own slice and the small-community heuristic naturally collapses siblings from the same directory (e.g. many different test-file clusters under the same folder), so duplicates are expected even after the steps above — verified in production: 338 of 1083 communities (31%) shared a label with at least one other community before reconciliation (apa monorepo, 2026-07-03). After all labels are assembled: +1. Group community IDs by their current label. +2. For every group with >1 member, pick a distinguishing keyword from each member's OWN node labels (word-frequency count, excluding stopwords and words already in the shared base label) and append it, e.g. `"Agent Gateway MCP Client" -> "Agent Gateway MCP Client — Server"` / `"— Additional"`. +3. If a collision still remains after that (rare — e.g. multiple communities extracted from the very same source image), append a numeric suffix `" (1)"`, `" (2)"` as a last resort. +4. Re-check for zero duplicates before moving on. + ```bash $(cat graphify-out/.graphify_python) -c " import sys, json @@ -526,6 +542,8 @@ graphify export html # auto-aggregates to community view if graph > 5000 nodes # or: graphify export html --no-viz ``` +**If the resulting view (aggregated community graph, or the raw graph for small corpora) has more than ~300 nodes, ALSO generate a Sigma.js + graphology version and present that as the primary interactive file instead of vis-network's `graph.html`.** vis-network runs a live single-threaded JS forceAtlas2 physics simulation on load (200 iterations before it disables itself) — verified in production this is genuinely slow/laggy past a few hundred nodes (apa monorepo, 1083-community aggregated view, 2026-07-03), independent of hardware. Sigma.js (WebGL rendering) with the layout **precomputed offline in Python** (`nx.forceatlas2_layout`, no client-side physics at all) fixes this at the root rather than swapping one JS physics engine for another. See `references/sigma-viz.md` for the full recipe (meta-graph construction, offline layout, self-contained HTML template loading sigma/graphology from esm.sh). Below the 300-node threshold, vis-network's `graph.html` is fine as-is — the extra build step isn't worth it. + ### Steps 6b-8 - Wiki, Neo4j, FalkorDB, SVG, GraphML, MCP, benchmark (only on their flags) These run only when their flag is present (`--wiki`, `--neo4j`/`--neo4j-push`, `--falkordb`/`--falkordb-push`, `--svg`, `--graphml`, `--mcp`) or, for the token-reduction benchmark, when `total_words` exceeds 5,000. A default run with no export flags skips all of them. See `references/exports.md` for each one. Run any `--wiki` export before Step 9 cleanup so `.graphify_labels.json` is still available. diff --git a/graphify/skill-pi.md b/graphify/skill-pi.md index b3542438d..5c337677e 100644 --- a/graphify/skill-pi.md +++ b/graphify/skill-pi.md @@ -477,8 +477,24 @@ Substitute `IS_DIRECTED` and `INPUT_PATH` as in Step 4. If a `GRAPH HEALTH WARNI Read `graphify-out/.graphify_analysis.json`. For each community key, look at its node labels and write a 2-5 word plain-language name (e.g. "Attention Mechanism", "Training Pipeline", "Data Loading"). +**At scale (100+ communities), do NOT hand-label one at a time and do NOT fall back to a naive path-prefix heuristic (e.g. title-casing the common directory prefix).** Both break down past a handful of communities — the heuristic in particular produces duplicated-segment junk ("Backend Backend", "Frontend Frontend") and word-soup names that look plausible until you actually read them. Verified in production on a 1000+ community monorepo graph (apa, 2026-07-03): a heuristic pass needed a full redo. + +What works instead, for communities with >=20 nodes (the ones that actually surface in God Nodes / navigation): +1. For each such community, sample its ~15 highest-degree (most connected) member nodes — not the full member list, communities can have 300+ nodes. +2. Batch communities into groups of ~15-20 per subagent (write each batch's samples to a small JSON file under `graphify-out/`). +3. Dispatch one `general-purpose` subagent per batch (all in the same message, in parallel — same pattern as Step 3 Part B), each told to read its batch file and write a JSON `{community_id: name}` map based on genuine reasoning about the sampled node labels/paths, not just repeating directory names. +4. Merge all batch results. + +For the long tail of small communities (<20 nodes) where subagent cost isn't justified, a path-prefix heuristic is an acceptable lighter-weight fallback — but dedupe repeated words/segments, strip trailing file extensions, and spot-check a random sample of ~15 outputs across different sizes before trusting it. If duplicated-word or generic ("Community N") names appear in the sample, fix the heuristic before applying it to the rest. + Then regenerate the report and save the labels for the visualizer: +**Reconcile the FULL label set for cross-batch duplicates before regenerating the report — this is a required final pass, not optional polish.** Subagent batches only see their own slice and the small-community heuristic naturally collapses siblings from the same directory (e.g. many different test-file clusters under the same folder), so duplicates are expected even after the steps above — verified in production: 338 of 1083 communities (31%) shared a label with at least one other community before reconciliation (apa monorepo, 2026-07-03). After all labels are assembled: +1. Group community IDs by their current label. +2. For every group with >1 member, pick a distinguishing keyword from each member's OWN node labels (word-frequency count, excluding stopwords and words already in the shared base label) and append it, e.g. `"Agent Gateway MCP Client" -> "Agent Gateway MCP Client — Server"` / `"— Additional"`. +3. If a collision still remains after that (rare — e.g. multiple communities extracted from the very same source image), append a numeric suffix `" (1)"`, `" (2)"` as a last resort. +4. Re-check for zero duplicates before moving on. + ```bash $(cat graphify-out/.graphify_python) -c " import sys, json @@ -534,6 +550,8 @@ graphify export html # auto-aggregates to community view if graph > 5000 nodes # or: graphify export html --no-viz ``` +**If the resulting view (aggregated community graph, or the raw graph for small corpora) has more than ~300 nodes, ALSO generate a Sigma.js + graphology version and present that as the primary interactive file instead of vis-network's `graph.html`.** vis-network runs a live single-threaded JS forceAtlas2 physics simulation on load (200 iterations before it disables itself) — verified in production this is genuinely slow/laggy past a few hundred nodes (apa monorepo, 1083-community aggregated view, 2026-07-03), independent of hardware. Sigma.js (WebGL rendering) with the layout **precomputed offline in Python** (`nx.forceatlas2_layout`, no client-side physics at all) fixes this at the root rather than swapping one JS physics engine for another. See `references/sigma-viz.md` for the full recipe (meta-graph construction, offline layout, self-contained HTML template loading sigma/graphology from esm.sh). Below the 300-node threshold, vis-network's `graph.html` is fine as-is — the extra build step isn't worth it. + ### Steps 6b-8 - Wiki, Neo4j, FalkorDB, SVG, GraphML, MCP, benchmark (only on their flags) These run only when their flag is present (`--wiki`, `--neo4j`/`--neo4j-push`, `--falkordb`/`--falkordb-push`, `--svg`, `--graphml`, `--mcp`) or, for the token-reduction benchmark, when `total_words` exceeds 5,000. A default run with no export flags skips all of them. See `references/exports.md` for each one. Run any `--wiki` export before Step 9 cleanup so `.graphify_labels.json` is still available. diff --git a/graphify/skill-trae.md b/graphify/skill-trae.md index ac16b925d..72fbd9937 100644 --- a/graphify/skill-trae.md +++ b/graphify/skill-trae.md @@ -475,8 +475,24 @@ Substitute `IS_DIRECTED` and `INPUT_PATH` as in Step 4. If a `GRAPH HEALTH WARNI Read `graphify-out/.graphify_analysis.json`. For each community key, look at its node labels and write a 2-5 word plain-language name (e.g. "Attention Mechanism", "Training Pipeline", "Data Loading"). +**At scale (100+ communities), do NOT hand-label one at a time and do NOT fall back to a naive path-prefix heuristic (e.g. title-casing the common directory prefix).** Both break down past a handful of communities — the heuristic in particular produces duplicated-segment junk ("Backend Backend", "Frontend Frontend") and word-soup names that look plausible until you actually read them. Verified in production on a 1000+ community monorepo graph (apa, 2026-07-03): a heuristic pass needed a full redo. + +What works instead, for communities with >=20 nodes (the ones that actually surface in God Nodes / navigation): +1. For each such community, sample its ~15 highest-degree (most connected) member nodes — not the full member list, communities can have 300+ nodes. +2. Batch communities into groups of ~15-20 per subagent (write each batch's samples to a small JSON file under `graphify-out/`). +3. Dispatch one `general-purpose` subagent per batch (all in the same message, in parallel — same pattern as Step 3 Part B), each told to read its batch file and write a JSON `{community_id: name}` map based on genuine reasoning about the sampled node labels/paths, not just repeating directory names. +4. Merge all batch results. + +For the long tail of small communities (<20 nodes) where subagent cost isn't justified, a path-prefix heuristic is an acceptable lighter-weight fallback — but dedupe repeated words/segments, strip trailing file extensions, and spot-check a random sample of ~15 outputs across different sizes before trusting it. If duplicated-word or generic ("Community N") names appear in the sample, fix the heuristic before applying it to the rest. + Then regenerate the report and save the labels for the visualizer: +**Reconcile the FULL label set for cross-batch duplicates before regenerating the report — this is a required final pass, not optional polish.** Subagent batches only see their own slice and the small-community heuristic naturally collapses siblings from the same directory (e.g. many different test-file clusters under the same folder), so duplicates are expected even after the steps above — verified in production: 338 of 1083 communities (31%) shared a label with at least one other community before reconciliation (apa monorepo, 2026-07-03). After all labels are assembled: +1. Group community IDs by their current label. +2. For every group with >1 member, pick a distinguishing keyword from each member's OWN node labels (word-frequency count, excluding stopwords and words already in the shared base label) and append it, e.g. `"Agent Gateway MCP Client" -> "Agent Gateway MCP Client — Server"` / `"— Additional"`. +3. If a collision still remains after that (rare — e.g. multiple communities extracted from the very same source image), append a numeric suffix `" (1)"`, `" (2)"` as a last resort. +4. Re-check for zero duplicates before moving on. + ```bash $(cat graphify-out/.graphify_python) -c " import sys, json @@ -532,6 +548,8 @@ graphify export html # auto-aggregates to community view if graph > 5000 nodes # or: graphify export html --no-viz ``` +**If the resulting view (aggregated community graph, or the raw graph for small corpora) has more than ~300 nodes, ALSO generate a Sigma.js + graphology version and present that as the primary interactive file instead of vis-network's `graph.html`.** vis-network runs a live single-threaded JS forceAtlas2 physics simulation on load (200 iterations before it disables itself) — verified in production this is genuinely slow/laggy past a few hundred nodes (apa monorepo, 1083-community aggregated view, 2026-07-03), independent of hardware. Sigma.js (WebGL rendering) with the layout **precomputed offline in Python** (`nx.forceatlas2_layout`, no client-side physics at all) fixes this at the root rather than swapping one JS physics engine for another. See `references/sigma-viz.md` for the full recipe (meta-graph construction, offline layout, self-contained HTML template loading sigma/graphology from esm.sh). Below the 300-node threshold, vis-network's `graph.html` is fine as-is — the extra build step isn't worth it. + ### Steps 6b-8 - Wiki, Neo4j, FalkorDB, SVG, GraphML, MCP, benchmark (only on their flags) These run only when their flag is present (`--wiki`, `--neo4j`/`--neo4j-push`, `--falkordb`/`--falkordb-push`, `--svg`, `--graphml`, `--mcp`) or, for the token-reduction benchmark, when `total_words` exceeds 5,000. A default run with no export flags skips all of them. See `references/exports.md` for each one. Run any `--wiki` export before Step 9 cleanup so `.graphify_labels.json` is still available. diff --git a/graphify/skill-vscode.md b/graphify/skill-vscode.md index 31a3352f7..c9592c649 100644 --- a/graphify/skill-vscode.md +++ b/graphify/skill-vscode.md @@ -473,8 +473,24 @@ Substitute `IS_DIRECTED` and `INPUT_PATH` as in Step 4. If a `GRAPH HEALTH WARNI Read `graphify-out/.graphify_analysis.json`. For each community key, look at its node labels and write a 2-5 word plain-language name (e.g. "Attention Mechanism", "Training Pipeline", "Data Loading"). +**At scale (100+ communities), do NOT hand-label one at a time and do NOT fall back to a naive path-prefix heuristic (e.g. title-casing the common directory prefix).** Both break down past a handful of communities — the heuristic in particular produces duplicated-segment junk ("Backend Backend", "Frontend Frontend") and word-soup names that look plausible until you actually read them. Verified in production on a 1000+ community monorepo graph (apa, 2026-07-03): a heuristic pass needed a full redo. + +What works instead, for communities with >=20 nodes (the ones that actually surface in God Nodes / navigation): +1. For each such community, sample its ~15 highest-degree (most connected) member nodes — not the full member list, communities can have 300+ nodes. +2. Batch communities into groups of ~15-20 per subagent (write each batch's samples to a small JSON file under `graphify-out/`). +3. Dispatch one `general-purpose` subagent per batch (all in the same message, in parallel — same pattern as Step 3 Part B), each told to read its batch file and write a JSON `{community_id: name}` map based on genuine reasoning about the sampled node labels/paths, not just repeating directory names. +4. Merge all batch results. + +For the long tail of small communities (<20 nodes) where subagent cost isn't justified, a path-prefix heuristic is an acceptable lighter-weight fallback — but dedupe repeated words/segments, strip trailing file extensions, and spot-check a random sample of ~15 outputs across different sizes before trusting it. If duplicated-word or generic ("Community N") names appear in the sample, fix the heuristic before applying it to the rest. + Then regenerate the report and save the labels for the visualizer: +**Reconcile the FULL label set for cross-batch duplicates before regenerating the report — this is a required final pass, not optional polish.** Subagent batches only see their own slice and the small-community heuristic naturally collapses siblings from the same directory (e.g. many different test-file clusters under the same folder), so duplicates are expected even after the steps above — verified in production: 338 of 1083 communities (31%) shared a label with at least one other community before reconciliation (apa monorepo, 2026-07-03). After all labels are assembled: +1. Group community IDs by their current label. +2. For every group with >1 member, pick a distinguishing keyword from each member's OWN node labels (word-frequency count, excluding stopwords and words already in the shared base label) and append it, e.g. `"Agent Gateway MCP Client" -> "Agent Gateway MCP Client — Server"` / `"— Additional"`. +3. If a collision still remains after that (rare — e.g. multiple communities extracted from the very same source image), append a numeric suffix `" (1)"`, `" (2)"` as a last resort. +4. Re-check for zero duplicates before moving on. + ```bash $(cat graphify-out/.graphify_python) -c " import sys, json @@ -530,6 +546,8 @@ graphify export html # auto-aggregates to community view if graph > 5000 nodes # or: graphify export html --no-viz ``` +**If the resulting view (aggregated community graph, or the raw graph for small corpora) has more than ~300 nodes, ALSO generate a Sigma.js + graphology version and present that as the primary interactive file instead of vis-network's `graph.html`.** vis-network runs a live single-threaded JS forceAtlas2 physics simulation on load (200 iterations before it disables itself) — verified in production this is genuinely slow/laggy past a few hundred nodes (apa monorepo, 1083-community aggregated view, 2026-07-03), independent of hardware. Sigma.js (WebGL rendering) with the layout **precomputed offline in Python** (`nx.forceatlas2_layout`, no client-side physics at all) fixes this at the root rather than swapping one JS physics engine for another. See `references/sigma-viz.md` for the full recipe (meta-graph construction, offline layout, self-contained HTML template loading sigma/graphology from esm.sh). Below the 300-node threshold, vis-network's `graph.html` is fine as-is — the extra build step isn't worth it. + ### Steps 6b-8 - Wiki, Neo4j, FalkorDB, SVG, GraphML, MCP, benchmark (only on their flags) These run only when their flag is present (`--wiki`, `--neo4j`/`--neo4j-push`, `--falkordb`/`--falkordb-push`, `--svg`, `--graphml`, `--mcp`) or, for the token-reduction benchmark, when `total_words` exceeds 5,000. A default run with no export flags skips all of them. See `references/exports.md` for each one. Run any `--wiki` export before Step 9 cleanup so `.graphify_labels.json` is still available. diff --git a/graphify/skill-windows.md b/graphify/skill-windows.md index 6ab0e027c..cd9091aef 100644 --- a/graphify/skill-windows.md +++ b/graphify/skill-windows.md @@ -1,5 +1,5 @@ --- -name: graphify-windows +name: graphify description: "Use for any question about a codebase, its architecture, file relationships, or project content — especially when graphify-out/ exists, where the question should be treated as a graphify query first. Turns any input (code, docs, papers, images, videos) into a persistent knowledge graph with god nodes, community detection, and query/path/explain tools." --- @@ -499,8 +499,24 @@ Substitute `IS_DIRECTED` and `INPUT_PATH` as in Step 4. If a `GRAPH HEALTH WARNI Read `graphify-out/.graphify_analysis.json`. For each community key, look at its node labels and write a 2-5 word plain-language name (e.g. "Attention Mechanism", "Training Pipeline", "Data Loading"). +**At scale (100+ communities), do NOT hand-label one at a time and do NOT fall back to a naive path-prefix heuristic (e.g. title-casing the common directory prefix).** Both break down past a handful of communities — the heuristic in particular produces duplicated-segment junk ("Backend Backend", "Frontend Frontend") and word-soup names that look plausible until you actually read them. Verified in production on a 1000+ community monorepo graph (apa, 2026-07-03): a heuristic pass needed a full redo. + +What works instead, for communities with >=20 nodes (the ones that actually surface in God Nodes / navigation): +1. For each such community, sample its ~15 highest-degree (most connected) member nodes — not the full member list, communities can have 300+ nodes. +2. Batch communities into groups of ~15-20 per subagent (write each batch's samples to a small JSON file under `graphify-out/`). +3. Dispatch one `general-purpose` subagent per batch (all in the same message, in parallel — same pattern as Step 3 Part B), each told to read its batch file and write a JSON `{community_id: name}` map based on genuine reasoning about the sampled node labels/paths, not just repeating directory names. +4. Merge all batch results. + +For the long tail of small communities (<20 nodes) where subagent cost isn't justified, a path-prefix heuristic is an acceptable lighter-weight fallback — but dedupe repeated words/segments, strip trailing file extensions, and spot-check a random sample of ~15 outputs across different sizes before trusting it. If duplicated-word or generic ("Community N") names appear in the sample, fix the heuristic before applying it to the rest. + Then regenerate the report and save the labels for the visualizer: +**Reconcile the FULL label set for cross-batch duplicates before regenerating the report — this is a required final pass, not optional polish.** Subagent batches only see their own slice and the small-community heuristic naturally collapses siblings from the same directory (e.g. many different test-file clusters under the same folder), so duplicates are expected even after the steps above — verified in production: 338 of 1083 communities (31%) shared a label with at least one other community before reconciliation (apa monorepo, 2026-07-03). After all labels are assembled: +1. Group community IDs by their current label. +2. For every group with >1 member, pick a distinguishing keyword from each member's OWN node labels (word-frequency count, excluding stopwords and words already in the shared base label) and append it, e.g. `"Agent Gateway MCP Client" -> "Agent Gateway MCP Client — Server"` / `"— Additional"`. +3. If a collision still remains after that (rare — e.g. multiple communities extracted from the very same source image), append a numeric suffix `" (1)"`, `" (2)"` as a last resort. +4. Re-check for zero duplicates before moving on. + ```bash $(cat graphify-out/.graphify_python) -c " import sys, json @@ -556,6 +572,8 @@ graphify export html # auto-aggregates to community view if graph > 5000 nodes # or: graphify export html --no-viz ``` +**If the resulting view (aggregated community graph, or the raw graph for small corpora) has more than ~300 nodes, ALSO generate a Sigma.js + graphology version and present that as the primary interactive file instead of vis-network's `graph.html`.** vis-network runs a live single-threaded JS forceAtlas2 physics simulation on load (200 iterations before it disables itself) — verified in production this is genuinely slow/laggy past a few hundred nodes (apa monorepo, 1083-community aggregated view, 2026-07-03), independent of hardware. Sigma.js (WebGL rendering) with the layout **precomputed offline in Python** (`nx.forceatlas2_layout`, no client-side physics at all) fixes this at the root rather than swapping one JS physics engine for another. See `references/sigma-viz.md` for the full recipe (meta-graph construction, offline layout, self-contained HTML template loading sigma/graphology from esm.sh). Below the 300-node threshold, vis-network's `graph.html` is fine as-is — the extra build step isn't worth it. + ### Steps 6b-8 - Wiki, Neo4j, FalkorDB, SVG, GraphML, MCP, benchmark (only on their flags) These run only when their flag is present (`--wiki`, `--neo4j`/`--neo4j-push`, `--falkordb`/`--falkordb-push`, `--svg`, `--graphml`, `--mcp`) or, for the token-reduction benchmark, when `total_words` exceeds 5,000. A default run with no export flags skips all of them. See `references/exports.md` for each one. Run any `--wiki` export before Step 9 cleanup so `.graphify_labels.json` is still available. diff --git a/graphify/skill.md b/graphify/skill.md index b3542438d..5c337677e 100644 --- a/graphify/skill.md +++ b/graphify/skill.md @@ -477,8 +477,24 @@ Substitute `IS_DIRECTED` and `INPUT_PATH` as in Step 4. If a `GRAPH HEALTH WARNI Read `graphify-out/.graphify_analysis.json`. For each community key, look at its node labels and write a 2-5 word plain-language name (e.g. "Attention Mechanism", "Training Pipeline", "Data Loading"). +**At scale (100+ communities), do NOT hand-label one at a time and do NOT fall back to a naive path-prefix heuristic (e.g. title-casing the common directory prefix).** Both break down past a handful of communities — the heuristic in particular produces duplicated-segment junk ("Backend Backend", "Frontend Frontend") and word-soup names that look plausible until you actually read them. Verified in production on a 1000+ community monorepo graph (apa, 2026-07-03): a heuristic pass needed a full redo. + +What works instead, for communities with >=20 nodes (the ones that actually surface in God Nodes / navigation): +1. For each such community, sample its ~15 highest-degree (most connected) member nodes — not the full member list, communities can have 300+ nodes. +2. Batch communities into groups of ~15-20 per subagent (write each batch's samples to a small JSON file under `graphify-out/`). +3. Dispatch one `general-purpose` subagent per batch (all in the same message, in parallel — same pattern as Step 3 Part B), each told to read its batch file and write a JSON `{community_id: name}` map based on genuine reasoning about the sampled node labels/paths, not just repeating directory names. +4. Merge all batch results. + +For the long tail of small communities (<20 nodes) where subagent cost isn't justified, a path-prefix heuristic is an acceptable lighter-weight fallback — but dedupe repeated words/segments, strip trailing file extensions, and spot-check a random sample of ~15 outputs across different sizes before trusting it. If duplicated-word or generic ("Community N") names appear in the sample, fix the heuristic before applying it to the rest. + Then regenerate the report and save the labels for the visualizer: +**Reconcile the FULL label set for cross-batch duplicates before regenerating the report — this is a required final pass, not optional polish.** Subagent batches only see their own slice and the small-community heuristic naturally collapses siblings from the same directory (e.g. many different test-file clusters under the same folder), so duplicates are expected even after the steps above — verified in production: 338 of 1083 communities (31%) shared a label with at least one other community before reconciliation (apa monorepo, 2026-07-03). After all labels are assembled: +1. Group community IDs by their current label. +2. For every group with >1 member, pick a distinguishing keyword from each member's OWN node labels (word-frequency count, excluding stopwords and words already in the shared base label) and append it, e.g. `"Agent Gateway MCP Client" -> "Agent Gateway MCP Client — Server"` / `"— Additional"`. +3. If a collision still remains after that (rare — e.g. multiple communities extracted from the very same source image), append a numeric suffix `" (1)"`, `" (2)"` as a last resort. +4. Re-check for zero duplicates before moving on. + ```bash $(cat graphify-out/.graphify_python) -c " import sys, json @@ -534,6 +550,8 @@ graphify export html # auto-aggregates to community view if graph > 5000 nodes # or: graphify export html --no-viz ``` +**If the resulting view (aggregated community graph, or the raw graph for small corpora) has more than ~300 nodes, ALSO generate a Sigma.js + graphology version and present that as the primary interactive file instead of vis-network's `graph.html`.** vis-network runs a live single-threaded JS forceAtlas2 physics simulation on load (200 iterations before it disables itself) — verified in production this is genuinely slow/laggy past a few hundred nodes (apa monorepo, 1083-community aggregated view, 2026-07-03), independent of hardware. Sigma.js (WebGL rendering) with the layout **precomputed offline in Python** (`nx.forceatlas2_layout`, no client-side physics at all) fixes this at the root rather than swapping one JS physics engine for another. See `references/sigma-viz.md` for the full recipe (meta-graph construction, offline layout, self-contained HTML template loading sigma/graphology from esm.sh). Below the 300-node threshold, vis-network's `graph.html` is fine as-is — the extra build step isn't worth it. + ### Steps 6b-8 - Wiki, Neo4j, FalkorDB, SVG, GraphML, MCP, benchmark (only on their flags) These run only when their flag is present (`--wiki`, `--neo4j`/`--neo4j-push`, `--falkordb`/`--falkordb-push`, `--svg`, `--graphml`, `--mcp`) or, for the token-reduction benchmark, when `total_words` exceeds 5,000. A default run with no export flags skips all of them. See `references/exports.md` for each one. Run any `--wiki` export before Step 9 cleanup so `.graphify_labels.json` is still available. diff --git a/graphify/skills/agents/references/sigma-viz.md b/graphify/skills/agents/references/sigma-viz.md new file mode 100644 index 000000000..0eb161320 --- /dev/null +++ b/graphify/skills/agents/references/sigma-viz.md @@ -0,0 +1,449 @@ +# graphify reference: Sigma.js + graphology visualization for large graphs + +Load this when Step 6's `graph.html` (vis-network) would render more than ~300 nodes — either the aggregated community view on a large corpus, or the raw graph on a smaller one that still clusters into 300+ communities. vis-network runs a live, single-threaded JS forceAtlas2 physics simulation on load (canvas 2D rendering); past a few hundred nodes this stabilization pass is genuinely slow regardless of hardware. The fix is not swapping to a different JS physics engine — it's removing client-side physics entirely: precompute the layout once in Python (fast, uses networkx's optimized implementation) and render only, with sigma.js's WebGL renderer instead of vis-network's canvas 2D renderer. + +Output file: `graphify-out/graph_sigma.html` (self-contained, opens directly like `graph.html`). + +Beyond the raw performance fix, this view also encodes three things vis-network's `graph.html` doesn't surface at a glance: each community's **dominant content kind** (code/document/paper/image/rationale/concept, drawn as a small icon), its **dominant module** (the top-level directory most of its members live under, drawn as color), and a **left-side filter panel** for both entity kind and relation type — so a code-heavy corpus with a handful of docs sprinkled in doesn't read as one undifferentiated blob. + +## Step 1 — build the meta-graph and precompute layout in Python + +Adjust `MIN_COMMUNITY_SIZE` (20 is a reasonable default — communities below this rarely appear in God Nodes / navigation and just add render cost) and `INPUT_PATH`/label source to match the current run. + +```python +import json +import math +import networkx as nx +from pathlib import Path +from collections import Counter, defaultdict + +g_data = json.loads(Path('graphify-out/graph.json').read_text(encoding='utf-8')) +labels = json.loads(Path('graphify-out/.graphify_labels.json').read_text(encoding='utf-8')) + +# Maps every relation graphify emits (AST-structural and LLM-semantic alike) to +# one of six buckets the filter panel toggles as a group. Keep this in sync +# with the relation vocabulary in graphify/extract.py if new relations are +# added there — an unmapped relation falls into 'other' rather than crashing. +RELATION_BUCKETS = { + 'calls': 'calls', 'indirect_call': 'calls', 'instantiates': 'calls', + 'contains': 'structure', 'defines': 'structure', 'method': 'structure', + 'implements': 'structure', 'inherits': 'structure', + 'imports': 'imports', 'imports_from': 'imports', 'dynamic_import': 'imports', + 're_exports': 'imports', 'depends_on': 'imports', 'crate_depends_on': 'imports', + 'requires_env': 'imports', + 'references': 'references', 'references_constant': 'references', 'uses': 'references', + 'uses_static_prop': 'references', 'bound_to': 'references', 'listened_by': 'references', + 'cites': 'docs', 'conceptually_related_to': 'docs', 'shares_data_with': 'docs', + 'semantically_similar_to': 'docs', 'rationale_for': 'docs', + 'participate_in': 'groups', 'implement': 'groups', 'form': 'groups', +} + +def top_level_dir(source_file: str) -> str: + if not source_file or '/' not in source_file: + return '(root)' + return source_file.split('/')[0] + +MIN_COMMUNITY_SIZE = 20 +node_attrs = {n['id']: n for n in g_data['nodes']} +comms = defaultdict(list) +for n in g_data['nodes']: + comms[n['community']].append(n['id']) + +significant = {cid: members for cid, members in comms.items() if len(members) >= MIN_COMMUNITY_SIZE} +dropped = len(comms) - len(significant) +if not significant: + # Every community is below the threshold (plausible on a corpus that + # clusters into many small communities) - fall back to all of them + # rather than crashing on an empty meta-graph downstream. + largest = max((len(m) for m in comms.values()), default=0) + print(f'No community reaches MIN_COMMUNITY_SIZE={MIN_COMMUNITY_SIZE} (largest: {largest}) - showing all {len(comms)} communities instead.') + significant = comms + dropped = 0 +node_to_community = {m: cid for cid, members in significant.items() for m in members} + +meta = nx.Graph() +for cid, members in significant.items(): + # Dominant file_type (majority vote) drives the icon; dominant top-level + # directory drives the color. Both are approximations at the community + # level — a mixed community shows its majority kind/module, not a blend. + type_counts = Counter(node_attrs[m].get('file_type', 'code') for m in members) + dir_counts = Counter(top_level_dir(node_attrs[m].get('source_file', '')) for m in members) + meta.add_node( + cid, member_count=len(members), label=labels.get(str(cid), f'Community {cid}'), + file_type=type_counts.most_common(1)[0][0], module=dir_counts.most_common(1)[0][0], + ) + +edge_counts = Counter() +edge_buckets = defaultdict(Counter) +# NetworkX <= 3.1 serializes edges as 'links'; some older graph.json files use +# 'edges' instead (same compatibility hazard graphify/build.py and +# graphify/affected.py already guard against) - accept either. +links_key = 'links' if 'links' in g_data else 'edges' +for link in g_data[links_key]: + cu, cv = node_to_community.get(link['source']), node_to_community.get(link['target']) + if cu is not None and cv is not None and cu != cv: + key = (min(cu, cv), max(cu, cv)) + edge_counts[key] += 1 + edge_buckets[key][RELATION_BUCKETS.get(link.get('relation', ''), 'other')] += 1 + +# Hyperedges (3+ node group relations - participate_in/implement/form) carry +# no source/target, so they never appear in the loop above. Remap each to the +# distinct communities its members span and count it into every pairwise +# combination, mirroring how graphify/export.py's vis-network aggregated view +# remaps hyperedges to community IDs - otherwise 'groups' in RELATION_BUCKETS +# is unreachable and the filter panel's "Groups" checkbox always no-ops. +for he in g_data.get('hyperedges', []): + he_communities = sorted({node_to_community[m] for m in he.get('nodes', []) if m in node_to_community}) + for i in range(len(he_communities)): + for j in range(i + 1, len(he_communities)): + key = (he_communities[i], he_communities[j]) + edge_counts[key] += 1 + edge_buckets[key][RELATION_BUCKETS.get(he.get('relation', ''), 'groups')] += 1 + +for (cu, cv), w in edge_counts.items(): + meta.add_edge(cu, cv, weight=w, buckets=dict(edge_buckets[(cu, cv)])) + +# offline layout — no client-side physics needed at all +pos = nx.forceatlas2_layout(meta, max_iter=800, gravity=1.0, scaling_ratio=4.0, seed=42, weight='weight') + +degrees = dict(meta.degree()) +max_members = max((meta.nodes[n]['member_count'] for n in meta.nodes), default=1) +xs = [float(p[0]) for p in pos.values()] +ys = [float(p[1]) for p in pos.values()] +xr, yr = (max(xs) - min(xs)) or 1, (max(ys) - min(ys)) or 1 +scaled = {n: ((float(pos[n][0]) - min(xs)) / xr * 1000, (float(pos[n][1]) - min(ys)) / yr * 1000) for n in meta.nodes()} + +# Density-aware sizing: a fixed absolute size range (e.g. "3 to 15") looks +# fine at a few dozen communities but overlaps into an unreadable blob once +# forceAtlas2 packs 300+ communities into the same normalized space — the +# same size value covers a much larger *fraction* of the available room as +# node count grows. Calibrate against the layout's own median nearest- +# neighbor distance instead, so sizes stay legible regardless of density. +def _nearest_neighbor_dists(points): + dists = [] + for i, (x1, y1) in enumerate(points): + best = min((math.hypot(x1 - x2, y1 - y2) for j, (x2, y2) in enumerate(points) if j != i), default=None) + if best is not None: + dists.append(best) + return dists + +nn = _nearest_neighbor_dists(list(scaled.values())) +median_nn = sorted(nn)[len(nn) // 2] if nn else 30.0 +SIZE_MIN = max(2.0, median_nn * 0.12) +SIZE_MAX = max(SIZE_MIN * 2, median_nn * 0.45) + +nodes_out = [] +for n in meta.nodes(): + x, y = scaled[n] + deg, mc = int(degrees.get(n, 0)), int(meta.nodes[n]['member_count']) + nodes_out.append({ + 'key': str(n), 'label': meta.nodes[n]['label'], + 'x': round(x, 2), 'y': round(y, 2), + 'size': round(SIZE_MIN + (SIZE_MAX - SIZE_MIN) * (mc / max_members) ** 0.5, 2), + 'degree': deg, 'members': mc, + 'fileType': meta.nodes[n]['file_type'], 'module': meta.nodes[n]['module'], + }) +edges_out = [ + {'source': str(u), 'target': str(v), 'weight': int(d.get('weight', 1)), 'buckets': d.get('buckets', {})} + for u, v, d in meta.edges(data=True) +] + +Path('graphify-out/.graphify_sigma_data.json').write_text( + json.dumps({'nodes': nodes_out, 'edges': edges_out}, ensure_ascii=False), encoding='utf-8') +print(f'meta graph: {len(nodes_out)} nodes, {len(edges_out)} edges — layout precomputed, size range {SIZE_MIN:.1f}-{SIZE_MAX:.1f}, {dropped} communities below MIN_COMMUNITY_SIZE={MIN_COMMUNITY_SIZE} dropped') +``` + +**Important**: cast every numpy value (`forceatlas2_layout` returns numpy floats) to plain Python `float`/`int` before `json.dumps` — numpy scalars aren't JSON-serializable and will raise `TypeError: Object of type float32 is not JSON serializable`. + +## Step 2 — write the HTML template + +Write this template to `graphify-out/graph_sigma.html`, with a literal `__GRAPH_DATA__` placeholder where the data goes (substituted in Step 3 — do NOT try to embed the JSON directly while authoring the template, string-templating that much escaping inline is error-prone). + +Key implementation notes: +- **Library loading**: sigma@3 ships CJS/ESM only, no browser UMD global. Load libraries as ES modules from `esm.sh` (`https://esm.sh/sigma@3.0.3`, `https://esm.sh/graphology@0.25.4`, `https://esm.sh/@sigma/node-image@3.0.0`) via ` + + +``` + +## Step 3 — inject the data and clean up + +```python +import json +from pathlib import Path + +data = json.loads(Path('graphify-out/.graphify_sigma_data.json').read_text(encoding='utf-8')) +html = Path('graphify-out/graph_sigma.html').read_text(encoding='utf-8') +html = html.replace('__GRAPH_DATA__', json.dumps(data, ensure_ascii=False)) +Path('graphify-out/graph_sigma.html').write_text(html, encoding='utf-8') +Path('graphify-out/.graphify_sigma_data.json').unlink() +``` + +**Before telling the user it's done**, sanity-check the embedded JSON didn't break the ` + + +``` + +## Step 3 — inject the data and clean up + +```python +import json +from pathlib import Path + +data = json.loads(Path('graphify-out/.graphify_sigma_data.json').read_text(encoding='utf-8')) +html = Path('graphify-out/graph_sigma.html').read_text(encoding='utf-8') +html = html.replace('__GRAPH_DATA__', json.dumps(data, ensure_ascii=False)) +Path('graphify-out/graph_sigma.html').write_text(html, encoding='utf-8') +Path('graphify-out/.graphify_sigma_data.json').unlink() +``` + +**Before telling the user it's done**, sanity-check the embedded JSON didn't break the ` + + +``` + +## Step 3 — inject the data and clean up + +```python +import json +from pathlib import Path + +data = json.loads(Path('graphify-out/.graphify_sigma_data.json').read_text(encoding='utf-8')) +html = Path('graphify-out/graph_sigma.html').read_text(encoding='utf-8') +html = html.replace('__GRAPH_DATA__', json.dumps(data, ensure_ascii=False)) +Path('graphify-out/graph_sigma.html').write_text(html, encoding='utf-8') +Path('graphify-out/.graphify_sigma_data.json').unlink() +``` + +**Before telling the user it's done**, sanity-check the embedded JSON didn't break the ` + + +``` + +## Step 3 — inject the data and clean up + +```python +import json +from pathlib import Path + +data = json.loads(Path('graphify-out/.graphify_sigma_data.json').read_text(encoding='utf-8')) +html = Path('graphify-out/graph_sigma.html').read_text(encoding='utf-8') +html = html.replace('__GRAPH_DATA__', json.dumps(data, ensure_ascii=False)) +Path('graphify-out/graph_sigma.html').write_text(html, encoding='utf-8') +Path('graphify-out/.graphify_sigma_data.json').unlink() +``` + +**Before telling the user it's done**, sanity-check the embedded JSON didn't break the ` + + +``` + +## Step 3 — inject the data and clean up + +```python +import json +from pathlib import Path + +data = json.loads(Path('graphify-out/.graphify_sigma_data.json').read_text(encoding='utf-8')) +html = Path('graphify-out/graph_sigma.html').read_text(encoding='utf-8') +html = html.replace('__GRAPH_DATA__', json.dumps(data, ensure_ascii=False)) +Path('graphify-out/graph_sigma.html').write_text(html, encoding='utf-8') +Path('graphify-out/.graphify_sigma_data.json').unlink() +``` + +**Before telling the user it's done**, sanity-check the embedded JSON didn't break the ` + + +``` + +## Step 3 — inject the data and clean up + +```python +import json +from pathlib import Path + +data = json.loads(Path('graphify-out/.graphify_sigma_data.json').read_text(encoding='utf-8')) +html = Path('graphify-out/graph_sigma.html').read_text(encoding='utf-8') +html = html.replace('__GRAPH_DATA__', json.dumps(data, ensure_ascii=False)) +Path('graphify-out/graph_sigma.html').write_text(html, encoding='utf-8') +Path('graphify-out/.graphify_sigma_data.json').unlink() +``` + +**Before telling the user it's done**, sanity-check the embedded JSON didn't break the ` + + +``` + +## Step 3 — inject the data and clean up + +```python +import json +from pathlib import Path + +data = json.loads(Path('graphify-out/.graphify_sigma_data.json').read_text(encoding='utf-8')) +html = Path('graphify-out/graph_sigma.html').read_text(encoding='utf-8') +html = html.replace('__GRAPH_DATA__', json.dumps(data, ensure_ascii=False)) +Path('graphify-out/graph_sigma.html').write_text(html, encoding='utf-8') +Path('graphify-out/.graphify_sigma_data.json').unlink() +``` + +**Before telling the user it's done**, sanity-check the embedded JSON didn't break the ` + + +``` + +## Step 3 — inject the data and clean up + +```python +import json +from pathlib import Path + +data = json.loads(Path('graphify-out/.graphify_sigma_data.json').read_text(encoding='utf-8')) +html = Path('graphify-out/graph_sigma.html').read_text(encoding='utf-8') +html = html.replace('__GRAPH_DATA__', json.dumps(data, ensure_ascii=False)) +Path('graphify-out/graph_sigma.html').write_text(html, encoding='utf-8') +Path('graphify-out/.graphify_sigma_data.json').unlink() +``` + +**Before telling the user it's done**, sanity-check the embedded JSON didn't break the ` + + +``` + +## Step 3 — inject the data and clean up + +```python +import json +from pathlib import Path + +data = json.loads(Path('graphify-out/.graphify_sigma_data.json').read_text(encoding='utf-8')) +html = Path('graphify-out/graph_sigma.html').read_text(encoding='utf-8') +html = html.replace('__GRAPH_DATA__', json.dumps(data, ensure_ascii=False)) +Path('graphify-out/graph_sigma.html').write_text(html, encoding='utf-8') +Path('graphify-out/.graphify_sigma_data.json').unlink() +``` + +**Before telling the user it's done**, sanity-check the embedded JSON didn't break the ` + + +``` + +## Step 3 — inject the data and clean up + +```python +import json +from pathlib import Path + +data = json.loads(Path('graphify-out/.graphify_sigma_data.json').read_text(encoding='utf-8')) +html = Path('graphify-out/graph_sigma.html').read_text(encoding='utf-8') +html = html.replace('__GRAPH_DATA__', json.dumps(data, ensure_ascii=False)) +Path('graphify-out/graph_sigma.html').write_text(html, encoding='utf-8') +Path('graphify-out/.graphify_sigma_data.json').unlink() +``` + +**Before telling the user it's done**, sanity-check the embedded JSON didn't break the ` + + +``` + +## Step 3 — inject the data and clean up + +```python +import json +from pathlib import Path + +data = json.loads(Path('graphify-out/.graphify_sigma_data.json').read_text(encoding='utf-8')) +html = Path('graphify-out/graph_sigma.html').read_text(encoding='utf-8') +html = html.replace('__GRAPH_DATA__', json.dumps(data, ensure_ascii=False)) +Path('graphify-out/graph_sigma.html').write_text(html, encoding='utf-8') +Path('graphify-out/.graphify_sigma_data.json').unlink() +``` + +**Before telling the user it's done**, sanity-check the embedded JSON didn't break the ` + + +``` + +## Step 3 — inject the data and clean up + +```python +import json +from pathlib import Path + +data = json.loads(Path('graphify-out/.graphify_sigma_data.json').read_text(encoding='utf-8')) +html = Path('graphify-out/graph_sigma.html').read_text(encoding='utf-8') +html = html.replace('__GRAPH_DATA__', json.dumps(data, ensure_ascii=False)) +Path('graphify-out/graph_sigma.html').write_text(html, encoding='utf-8') +Path('graphify-out/.graphify_sigma_data.json').unlink() +``` + +**Before telling the user it's done**, sanity-check the embedded JSON didn't break the ` + + +``` + +## Step 3 — inject the data and clean up + +```python +import json +from pathlib import Path + +data = json.loads(Path('graphify-out/.graphify_sigma_data.json').read_text(encoding='utf-8')) +html = Path('graphify-out/graph_sigma.html').read_text(encoding='utf-8') +html = html.replace('__GRAPH_DATA__', json.dumps(data, ensure_ascii=False)) +Path('graphify-out/graph_sigma.html').write_text(html, encoding='utf-8') +Path('graphify-out/.graphify_sigma_data.json').unlink() +``` + +**Before telling the user it's done**, sanity-check the embedded JSON didn't break the ` + + +``` + +## Step 3 — inject the data and clean up + +```python +import json +from pathlib import Path + +data = json.loads(Path('graphify-out/.graphify_sigma_data.json').read_text(encoding='utf-8')) +html = Path('graphify-out/graph_sigma.html').read_text(encoding='utf-8') +html = html.replace('__GRAPH_DATA__', json.dumps(data, ensure_ascii=False)) +Path('graphify-out/graph_sigma.html').write_text(html, encoding='utf-8') +Path('graphify-out/.graphify_sigma_data.json').unlink() +``` + +**Before telling the user it's done**, sanity-check the embedded JSON didn't break the ` + + +``` + +## Step 3 — inject the data and clean up + +```python +import json +from pathlib import Path + +data = json.loads(Path('graphify-out/.graphify_sigma_data.json').read_text(encoding='utf-8')) +html = Path('graphify-out/graph_sigma.html').read_text(encoding='utf-8') +html = html.replace('__GRAPH_DATA__', json.dumps(data, ensure_ascii=False)) +Path('graphify-out/graph_sigma.html').write_text(html, encoding='utf-8') +Path('graphify-out/.graphify_sigma_data.json').unlink() +``` + +**Before telling the user it's done**, sanity-check the embedded JSON didn't break the ` + + +``` + +## Step 3 — inject the data and clean up + +```python +import json +from pathlib import Path + +data = json.loads(Path('graphify-out/.graphify_sigma_data.json').read_text(encoding='utf-8')) +html = Path('graphify-out/graph_sigma.html').read_text(encoding='utf-8') +html = html.replace('__GRAPH_DATA__', json.dumps(data, ensure_ascii=False)) +Path('graphify-out/graph_sigma.html').write_text(html, encoding='utf-8') +Path('graphify-out/.graphify_sigma_data.json').unlink() +``` + +**Before telling the user it's done**, sanity-check the embedded JSON didn't break the ` + + +``` + +## Step 3 — inject the data and clean up + +```python +import json +from pathlib import Path + +data = json.loads(Path('graphify-out/.graphify_sigma_data.json').read_text(encoding='utf-8')) +html = Path('graphify-out/graph_sigma.html').read_text(encoding='utf-8') +html = html.replace('__GRAPH_DATA__', json.dumps(data, ensure_ascii=False)) +Path('graphify-out/graph_sigma.html').write_text(html, encoding='utf-8') +Path('graphify-out/.graphify_sigma_data.json').unlink() +``` + +**Before telling the user it's done**, sanity-check the embedded JSON didn't break the ` + + +``` + +## Step 3 — inject the data and clean up + +```python +import json +from pathlib import Path + +data = json.loads(Path('graphify-out/.graphify_sigma_data.json').read_text(encoding='utf-8')) +html = Path('graphify-out/graph_sigma.html').read_text(encoding='utf-8') +html = html.replace('__GRAPH_DATA__', json.dumps(data, ensure_ascii=False)) +Path('graphify-out/graph_sigma.html').write_text(html, encoding='utf-8') +Path('graphify-out/.graphify_sigma_data.json').unlink() +``` + +**Before telling the user it's done**, sanity-check the embedded JSON didn't break the ` + + +``` + +## Step 3 — inject the data and clean up + +```python +import json +from pathlib import Path + +data = json.loads(Path('graphify-out/.graphify_sigma_data.json').read_text(encoding='utf-8')) +html = Path('graphify-out/graph_sigma.html').read_text(encoding='utf-8') +html = html.replace('__GRAPH_DATA__', json.dumps(data, ensure_ascii=False)) +Path('graphify-out/graph_sigma.html').write_text(html, encoding='utf-8') +Path('graphify-out/.graphify_sigma_data.json').unlink() +``` + +**Before telling the user it's done**, sanity-check the embedded JSON didn't break the ` + + +``` + +## Step 3 — inject the data and clean up + +```python +import json +from pathlib import Path + +data = json.loads(Path('graphify-out/.graphify_sigma_data.json').read_text(encoding='utf-8')) +html = Path('graphify-out/graph_sigma.html').read_text(encoding='utf-8') +html = html.replace('__GRAPH_DATA__', json.dumps(data, ensure_ascii=False)) +Path('graphify-out/graph_sigma.html').write_text(html, encoding='utf-8') +Path('graphify-out/.graphify_sigma_data.json').unlink() +``` + +**Before telling the user it's done**, sanity-check the embedded JSON didn't break the ` + + +``` + +## Step 3 — inject the data and clean up + +```python +import json +from pathlib import Path + +data = json.loads(Path('graphify-out/.graphify_sigma_data.json').read_text(encoding='utf-8')) +html = Path('graphify-out/graph_sigma.html').read_text(encoding='utf-8') +html = html.replace('__GRAPH_DATA__', json.dumps(data, ensure_ascii=False)) +Path('graphify-out/graph_sigma.html').write_text(html, encoding='utf-8') +Path('graphify-out/.graphify_sigma_data.json').unlink() +``` + +**Before telling the user it's done**, sanity-check the embedded JSON didn't break the ` + + +``` + +## Step 3 — inject the data and clean up + +```python +import json +from pathlib import Path + +data = json.loads(Path('graphify-out/.graphify_sigma_data.json').read_text(encoding='utf-8')) +html = Path('graphify-out/graph_sigma.html').read_text(encoding='utf-8') +html = html.replace('__GRAPH_DATA__', json.dumps(data, ensure_ascii=False)) +Path('graphify-out/graph_sigma.html').write_text(html, encoding='utf-8') +Path('graphify-out/.graphify_sigma_data.json').unlink() +``` + +**Before telling the user it's done**, sanity-check the embedded JSON didn't break the ` + + +``` + +## Step 3 — inject the data and clean up + +```python +import json +from pathlib import Path + +data = json.loads(Path('graphify-out/.graphify_sigma_data.json').read_text(encoding='utf-8')) +html = Path('graphify-out/graph_sigma.html').read_text(encoding='utf-8') +html = html.replace('__GRAPH_DATA__', json.dumps(data, ensure_ascii=False)) +Path('graphify-out/graph_sigma.html').write_text(html, encoding='utf-8') +Path('graphify-out/.graphify_sigma_data.json').unlink() +``` + +**Before telling the user it's done**, sanity-check the embedded JSON didn't break the ` + + +``` + +## Step 3 — inject the data and clean up + +```python +import json +from pathlib import Path + +data = json.loads(Path('graphify-out/.graphify_sigma_data.json').read_text(encoding='utf-8')) +html = Path('graphify-out/graph_sigma.html').read_text(encoding='utf-8') +html = html.replace('__GRAPH_DATA__', json.dumps(data, ensure_ascii=False)) +Path('graphify-out/graph_sigma.html').write_text(html, encoding='utf-8') +Path('graphify-out/.graphify_sigma_data.json').unlink() +``` + +**Before telling the user it's done**, sanity-check the embedded JSON didn't break the ` + + +``` + +## Step 3 — inject the data and clean up + +```python +import json +from pathlib import Path + +data = json.loads(Path('graphify-out/.graphify_sigma_data.json').read_text(encoding='utf-8')) +html = Path('graphify-out/graph_sigma.html').read_text(encoding='utf-8') +html = html.replace('__GRAPH_DATA__', json.dumps(data, ensure_ascii=False)) +Path('graphify-out/graph_sigma.html').write_text(html, encoding='utf-8') +Path('graphify-out/.graphify_sigma_data.json').unlink() +``` + +**Before telling the user it's done**, sanity-check the embedded JSON didn't break the ` + + +``` + +## Step 3 — inject the data and clean up + +```python +import json +from pathlib import Path + +data = json.loads(Path('graphify-out/.graphify_sigma_data.json').read_text(encoding='utf-8')) +html = Path('graphify-out/graph_sigma.html').read_text(encoding='utf-8') +html = html.replace('__GRAPH_DATA__', json.dumps(data, ensure_ascii=False)) +Path('graphify-out/graph_sigma.html').write_text(html, encoding='utf-8') +Path('graphify-out/.graphify_sigma_data.json').unlink() +``` + +**Before telling the user it's done**, sanity-check the embedded JSON didn't break the ` + + +``` + +## Step 3 — inject the data and clean up + +```python +import json +from pathlib import Path + +data = json.loads(Path('graphify-out/.graphify_sigma_data.json').read_text(encoding='utf-8')) +html = Path('graphify-out/graph_sigma.html').read_text(encoding='utf-8') +html = html.replace('__GRAPH_DATA__', json.dumps(data, ensure_ascii=False)) +Path('graphify-out/graph_sigma.html').write_text(html, encoding='utf-8') +Path('graphify-out/.graphify_sigma_data.json').unlink() +``` + +**Before telling the user it's done**, sanity-check the embedded JSON didn't break the ` + + +``` + +## Step 3 — inject the data and clean up + +```python +import json +from pathlib import Path + +data = json.loads(Path('graphify-out/.graphify_sigma_data.json').read_text(encoding='utf-8')) +html = Path('graphify-out/graph_sigma.html').read_text(encoding='utf-8') +html = html.replace('__GRAPH_DATA__', json.dumps(data, ensure_ascii=False)) +Path('graphify-out/graph_sigma.html').write_text(html, encoding='utf-8') +Path('graphify-out/.graphify_sigma_data.json').unlink() +``` + +**Before telling the user it's done**, sanity-check the embedded JSON didn't break the ` + + +``` + +## Step 3 — inject the data and clean up + +```python +import json +from pathlib import Path + +data = json.loads(Path('graphify-out/.graphify_sigma_data.json').read_text(encoding='utf-8')) +html = Path('graphify-out/graph_sigma.html').read_text(encoding='utf-8') +html = html.replace('__GRAPH_DATA__', json.dumps(data, ensure_ascii=False)) +Path('graphify-out/graph_sigma.html').write_text(html, encoding='utf-8') +Path('graphify-out/.graphify_sigma_data.json').unlink() +``` + +**Before telling the user it's done**, sanity-check the embedded JSON didn't break the `