State: experimental local model-building code plus evaluation infrastructure; no LineLoom-trained weights, completed GPU fine-tune, approved dataset, or real model benchmark results exist yet.
LineLoom's VS Code extension already supports more languages; this dedicated model track is intentionally narrower:
| Scope | VS Code language ID | Extension |
|---|---|---|
| Rust | rust |
.rs |
| C | c |
.c, .h |
| C++ | cpp |
.cc, .cpp, .cxx, .hpp |
| Python | python |
.py, .pyi |
| TypeScript | typescript / typescriptreact |
.ts, .tsx |
C and C++ are separate evaluation targets. JavaScript, Java, Kotlin and Go remain supported by the extension with existing Ollama models, but are not initial dedicated-model targets.
The checked-in fixtures/completion-v1.json contains 40 small, original, synthetic examples, eight per language. It is not a training corpus. A strict exact string comparison is a narrow regression metric, not a claim that the generated code compiles or is useful. Equivalent completions can be marked wrong.
npm run model:eval:dry
npm run test:model-eval
# Optional; requires running Ollama and an installed model:
npm run model:eval -- --model qwen2.5-coder:1.5b --runs 3 --mode auto
# Compare syntax-sensitive or suffix behavior in isolation:
npm run model:eval -- --language cpp --category suffix --runs 2
# For a future locally installed and reviewed FIM-capable LineLoom Code model:
npm run model:eval -- --model lineloom-code:dev --mode auto --language rust- The evaluator is opt-in and talks only to
http://127.0.0.1:11434/api/generate. It has no configurable remote endpoint and no model downloads. --dry-runis fully offline. No evaluator command reads editor buffers, repositories or personal source.--runsallows 1–5 sequential repetitions. Language filters:rust,c,cpp,python,typescript, orall. The optional--categoryfilter selectsexpression,statement,suffix,type,indentation, orall.- Each language has eight hand-authored, synthetic checks. Category-specific results distinguish common expressions, statements, suffix awareness, type usage, and indentation. Synthetic strict matches can reject valid alternative implementations.
autouses prefix/suffix generation for explicitly recognizedqwen2.5-coderandlineloom-codemodel tags, matching extension routing; unrecognized models use a bounded instruction format.--mode fimforces FIM only when the locally installed model actually supports it, and--mode instructionforces the generic protocol.- Only aggregate counts, per-language/per-category strict matches, nearest-rank p50/p95 duration/first-token latency, and fixture-level outcome totals without fixture IDs are printed.
fixtureOutcomesreports how many synthetic fixtures were evaluated, repeated, matched at least once, matched on every attempt, or had mixed exact-match outcomes. Failed requests count as non-matches.exactEveryAttemptalso includes one-run fixtures; userepeatedto distinguish repeated observations. These are descriptive synthetic results, not statistical pass@k, real-editor acceptance or proven model quality. Model text, prompts, paths and server errors are not logged. A failed request returns a failure count and a nonzero process exit code. - The evaluator stream decoder validates NDJSON finalization, malformed fields, UTF-8 boundaries and strict output/byte/record limits. It requires clean EOF after a terminal frame (no hidden later chunks) and uses the existing per-request timeout to abort stalled servers after
done. Cancellation cleanup is best-effort and non-blocking: a misbehaving stream whosecancel()promise never settles must not hang a timed-out or invalid benchmark request. This is verified with simulated in-memory streams, not real-model measurements. The VS Code extension still cancels promptly atdone, rejecting only already-buffered contradictory events. - The per-request deadline is enforced independently of whether a faulty fetch/stream transport acknowledges
AbortSignal; uncooperative test transports now return source-free failed attempts instead of indefinitely blocking the optional benchmark. This is a harness reliability guard, not evidence of local model latency, throughput or hardware behavior. - Run one named model and mode at a time on the same local machine, with warmed and cold trials recorded separately in human-written notes. Do not compare claims across different hardware without annotation.
- For a voluntary, real local two-model comparison without saving separate JSON reports, use
npm run model:eval:pair:dry -- --baseline model-a:tag --candidate model-b:tag --mode fim --runs 3to verify the offline scope, then usenpm run model:eval:pair -- --baseline model-a:tag --candidate model-b:tag --mode fim --runs 3if both models are already installed in Ollama. The prompt mode must be explicitly the same for both models (eitherfimorinstruction). Each repetition runs one complete fixture pass for each model; first-model order alternates across repetitions. Model identifiers and generated source are omitted from aggregate output. The runner does not control cache warm-up, loaded weights, quantization or hardware and does not prove real-editor usefulness.pairedStrictExact.fixtureStabilitycounts each unique synthetic fixture once and distinguishes repeatable candidate/baseline wins, ties, mixed results, incomplete attempts and single-run results; repeated requests are correlated, not independent evidence. - Human review of syntax correctness, unsafe code, indentation, duplicate suffix, irrelevant outputs, keyboard UX and actual VS Code acceptance remains NOT RUN until recorded separately.
See the reproducible local benchmark protocol, training and release gates and dataset and contributor policy. Do not confuse the extension's version numbers with any future model version.
The extension now recognizes an explicitly configured local Ollama tag of
lineloom-code or lineloom-code:<tag> as a native suffix-aware FIM model,
just as it recognizes installed qwen2.5-coder models. This sends the bounded
cursor prefix and optional suffix to Ollama rather than wrapping them in
generic chat instructions. When the user explicitly selects additional
reference files, the bounded instruction path is retained to preserve their
untrusted-data boundaries. Other/unrecognized model names keep the existing
instruction behavior.
A tag is only a routing convention, not proof of checkpoint capability. Only assign such a tag to a locally installed model verified to accept Ollama FIM requests; simply renaming an unrelated chat model does not enable FIM. No model is downloaded, trained, selected by default or published by this change. Before claiming improved autocomplete, collect real-model and physical-editor evidence separately.
The opt-in offline evaluator now routes lineloom-code and
lineloom-code:<tag> via native FIM in --mode auto, matching the
extension's explicit local-tag behavior when no related files are selected.
The prior synthetic v3 evaluator chose instructions for this tag; comparing
those differently routed reports would be misleading. The benchmark
identifier is now lineloom-synthetic-v5 and legacy v1–v4 Ollama
reports are intentionally rejected by the offline comparator. The v5 model:eval runner also inspects generated TypeScript syntax
in memory with the locally installed TypeScript parser, reporting only
per-language aggregate parser-pass/fail/unchecked counts. Rust, C, C++
and Python generated suggestions are explicitly unchecked in the Ollama
path. Python AST and TypeScript parser-only grammar analysis exist separately
in the explicitly approved local Hugging Face smoke runner; generated
Rust, C and C++ remain unchecked. Neither path executes generated code,
and neither is equivalent to model quality, type correctness or manual
editor acceptance. The new source-free comparator rejects fabricated
TypeScript syntax metrics and flags increased parser failures.
Re-run
baseline and candidate against the same checkout, explicit mode and
fixture scope; do not mix them with Hugging Face smoke v2 reports.
The change adds no installed model, real benchmark result, or automatic
download. Explicit --mode instruction still allows a controlled
generic-model baseline.
The opt-in, loopback-only synthetic model evaluator now reports a
reasoningEnvelope quality signal when a local model begins a completion
with a standalone <think> section, including the same opening tag after
recognized Ollama chat or FIM markers. Both complete and truncated reasoning
preambles are flagged. Marker-like text inside ordinary code strings or
later in the response is not classified as a leading reasoning envelope.
This is a source-free format diagnostic; raw reasoning text is not placed in benchmark reports, and the strict exact-continuation metric is unchanged. The production extension separately filters conservative, complete leading reasoning blocks before offering ghost text. A format signal is not evidence that a model produces accurate or safe completions.
Because the available format-signal set has changed, the current benchmark
protocol is lineloom-synthetic-v12. v11 and older saved reports must
not be compared against v12 results. Re-run both models using the same
fixtures, mode and repetition count to obtain comparable aggregate results.
No new trained weights, actual inference measurements or publisher release
are claimed by this implementation.
The synthetic evaluator now flags supplementary invisible Unicode tag
characters U+E0000–U+E007F as unsafeUnicode, matching the VS Code
completion filter. This changes how format diagnostics are counted even
though the fixture set and FIM/instruction request generation are otherwise
unchanged. At that milestone, the report protocol was lineloom-synthetic-v11:
v10 and older saved reports must not be compared with v11 results, even
when their request-spec and fixture hashes match. Re-run both actual
models under v11 to obtain comparable source-free numbers; no model
run or quality improvement is claimed. Existing checkpoint, rights,
training and physical editor release gates remain unmet.
The optional local synthetic evaluator now quotes untrusted fixture source
using the same source-line data-lane convention as the VS Code extension:
| on each line, not forgeable <before>/<after> wrappers.
It displays unusual newline separators as escaped Unicode sequences and
keeps the native-FIM request prefix/suffix byte-exact. The request SHA-256
binds this new format. Because it changes the instruction prompt,
the top-level report protocol is now lineloom-synthetic-v10.
Version v9 and earlier reports are not comparable, including for the
unchanged FIM route; rerun both locally under v10.
The benchmark is still not the full editor prompt: it intentionally does not reproduce per-language Rust/C/C++ static hints or selected related files. Matching a formatting convention does not prove model quality, prompt-injection protection or compiler-valid completions. No local inference/training, model download, weight release or new published extension version was performed. See the protocol framing section.
Real opt-in, local Ollama evaluation can now record terminal eval_count
and eval_duration metadata as source-free tokenAccounting. Its
per-language totals distinguish checked responses from missing (unchecked)
metadata. The total tokens-per-second figure is derived from Ollama's
reported output tokens divided by its generation-only duration; it is
null when no duration is measured, and does not attest hardware, server
configuration, cache conditions or code quality.
The offline comparison validates the counter totals and refuses to flag a meaningful completed-request p95 slowdown unless the number of completed requests, token telemetry coverage and per-language output-token totals match. A model generating longer completions is not silently judged slower based on unadjusted completion latency. These are still just synthetic measurements; user-editor acceptance, generation validity and real checkpoint testing must be verified independently.
The historical v9 telemetry milestone advanced the Ollama report identity
to lineloom-synthetic-v9. The current protocol is v10 (above).
Older reports must be rerun on the reviewed fixture set to compare.
See evaluation protocol section 4d.
The optional local evaluator now records nativeStructureDiagnostics for
completed Rust, C and C++ model requests, with per-language counts of
clean, mismatch and unchecked results. Unsupported/oversized inputs
and missing measurements are unchecked, not clean. The existing
nativeDelimiterMismatch signal is tied to mismatch counts, and the
source-free comparator validates the relationship before allowing any
comparison. A changed checked/unchecked coverage for any native language is
flagged for review, even if the overall inspected total is unchanged. Increases
in Rust, C or C++ mismatch warnings are tracked independently, so improvements
in one language cannot silently cancel a regression in another. Neither a
clean lexical scan nor a favorable comparator output proves model correctness.
This is a bounded lexical heuristic, not a parser, Rust/C/C++ compiler
check or safe-to-run verdict. The Ollama benchmark identifier is now
lineloom-synthetic-v8, so model baselines must be regenerated
under v8; v7 or earlier reports cannot be mixed into a controlled comparison.
See evaluation protocol section 4c.
The C/C++ delimiter scanner accounts for backslash-LF and backslash-CRLF
physical-line splicing before recognizing line comments or brackets.
Otherwise a continued // comment could be mistaken for unmatched code.
Rust is not normalized this way. Both the original input-size limit and
the unchecked result for unsupported cases are preserved. This is a
scanner accuracy fix, not a new parser, model benchmark, compiler
verification or extension release. Because it changes scanner evidence, the
reported native-scanner method is now bounded-native-delimiter-coverage-v2.
Old v1 method reports are rejected by the comparator, even when the
broader synthetic benchmark protocol is unchanged. Rerun both baseline
and candidate using the same checkout and method before comparing results.
The real local Ollama synthetic evaluator adds a bounded lexical diagnostic
for generated Rust, C and C++ FIM insertions. Mismatched delimiters and
unterminated strings/comments are counted as nativeDelimiterMismatch
format signals. It does not invoke compilers or interpret code; clean is
not compiler-valid, and generated Rust/C/C++ grammar outcomes remain
unchecked. Compare candidate models only after rerunning both under
lineloom-synthetic-v7. See
the evaluation protocol.
The local synthetic Ollama evaluator can now check generated Python grammar
with an isolated, bounded Python 3 stdlib ast.parse subprocess, alongside
its locked TypeScript parser. Neither path executes generated continuations.
If local Python is unavailable, the result is explicitly unchecked, never
a false parser success. The source-free comparator requires matching Python
parser coverage and reports parser failure deltas. Rust, C and C++ model
suggestions remain grammar-unchecked.
The current report format is lineloom-synthetic-v12 and cannot be compared
to previous v1–v11 Ollama reports; rerun both models with the same parser
environment. Historical notes above describe the earlier v5 development
milestone. No local inference, trained LineLoom Code weights, model-quality
baseline, or release publication is established by these changes.
Existing LineLoom endpoint approval, workspace trust, file exclusions, per-language model selection, cancellation and bounded prompt windows remain in force. The benchmark introduces no new inference path inside the extension; it is an explicitly invoked development command.
The eventual first-party model can be distributed only after a separate reviewed model license, weight provenance, privacy assessment, local compatibility test and maintainer release approval.
The optional offline comparator checks that two reports use the same SHA-256
fingerprint of all selected public fixture contents, the same resolved
prompt mode, benchmark protocol, number of fixtures, and repetitions.
New evaluator summaries include benchmarkProtocol,
fixtureSetSha256 and runsPerFixture (older summaries are not comparable).
It also rejects malformed numeric totals and inconsistent per-language or
per-category counts. Run separately against an already installed local model:
npm run model:eval -- --model qwen2.5-coder:1.5b --mode fim --runs 3 > baseline.json
npm run model:eval -- --model YOUR-LOCAL-MODEL --mode fim --runs 3 > candidate.json
npm run model:compare -- --baseline baseline.json --candidate candidate.jsonSecurity and interpretation: this command is offline and reads only the two explicit report files, each limited to 32 KiB, plus the fixed checked-in public synthetic fixture manifest. The comparator verifies each claimed fixture-set and request-spec hash against actual supported language/category selections, not merely against the other report; it also checks exact per-language/category attempt allocations. These checks are not cryptographic attestation of an actual model run and cannot establish hardware or model identity. It displays only allowlisted aggregate count/latency changes and per-language/per-category exact-match deltas, without echoing the model tags, filenames, JSON payloads, generated code, source, or file paths. Keep original report filenames and hardware details in private local notes. The comparator cannot verify CPU/GPU, thermal or quantization equivalence, and completed-only latency percentiles are not representative when failures differ. Higher strict equality is not proof of safer/correct code. No actual Ollama model has been benchmarked here. This tool is unreleased development and not included in GitHub v1.4.0.
The offline report comparator now also returns a strictly allowlisted
regressionReview with indicators for additional request failures,
fewer strict matches (including per-language declines), fewer completions
without format flags, more recognized format artifacts, and substantial
completed-request p95 increases under equal completion counts. This is
not a pass/fail model quality score; it cannot verify hardware parity,
code compilation, practical usability, or the authenticity of a report.
No source snippets, model names or custom user report fields are echoed.
LineLoom Code now has a read-only offline preflight for potential training samples. For a local, independently reviewed, permissioned manifest:
npm run test:model-data-audit
npm run model:data-audit -- --manifest local-reviewed-dataset.jsonThe audit checks language/split coverage, repository-group segregation, source revision and review declarations, duplicate and near-duplicate samples, overlap with the checked-in 40 synthetic evaluation fixtures, and conservative secret/PII red flags. It emits sanitized totals only; it does not train, copy, download, export, distribute, or persist any source. See data governance for the private manifest schema, limitations, mandatory human reviews, and failure behavior. No approved training corpus or model weights exist yet.
The local-only benchmark now distinguishes seven conservative format signals from strict exact-match equality: empty output, markdown fencing, chat preambles, model protocol tokens, unsafe/hidden Unicode, copied prefix text and copied suffix text. Completed suggestions can have multiple flags. No generated text, snippets or per-fixture flags are written to reports. Only aggregate counts and a count of completions with no detected format flags are included.
The benchmark protocol identifier is now lineloom-synthetic-v2.
Existing v1 JSON reports are not comparable with v2: rerun both baseline
and candidate models under the same new protocol rather than quietly
mixing scoring definitions. The offline comparator now reports per-signal
candidate-minus-baseline counts and rejects missing/inconsistent signal
summaries. Signals can be false positives or false negatives and do not
prove compilability, safety, relevance, acceptance or model superiority.
No actual Ollama baseline measurements have been made by this change.
Current preflight hardening: every target language must have declared samples in all three repository-separated splits (train, validation, test). Lightly edited copies of the public synthetic evaluation fixtures are additionally screened using bounded character shingles. Audit outputs include only numerical per-language/per-split totals; this is not enough data for a real model, and these heuristics do not replace independently reviewed rights, fork relationships, or contamination scans.
A request-spec SHA-256 now accompanies the original fixture-content hash
in dry-run and real local benchmark summaries. It deterministically covers
the actual constructed synthetic requests, including prefix/prompt, FIM
suffix, streaming and explicit Ollama options (temperature: 0,
num_predict: 64, num_ctx: 2048), excluding the model tag so different
models can be compared. It never logs raw prompts, generated code or
private workspace content. The offline comparator refuses missing or
different request-spec hashes and still checks fixture hash, mode,
repetitions and valid numeric outcomes.
Protocol is now lineloom-synthetic-v3. Reports from v1/v2 cannot
be compared to v3. Regenerate both reports on a matching checkout and use an
explicit prompt mode. Matching hashes cannot verify model/Modelfile
identity, server-side defaults, quantization, OS/GPU, system load,
correctness or human acceptance. No real model was benchmarked here.
Request-path test hardening (unreleased source): a new offline testable synthetic evaluator verifies the entire request → bounded NDJSON parsing → completion-format flags/strict equality → source-free result flow using an injected mock transport. It fails closed for malformed replies, HTTP errors and aborted requests, and tests deadline races. No real Ollama inference, network access in tests, approved corpus or trained model is implied.
An actual opt-in, locally runnable LoRA fine-tuning experiment now lives under train/. It uses the project's audited, manually supplied five-language FIM manifest and reviewed locally installed Qwen2.5-Coder-0.5B base candidate, not new pretrained weights. No checkpoint or training set is auto-downloaded or bundled. Python stdlib + Node can produce only a dry planning report; full training requires separate explicit data rights/checkpoint approvals, optional locally installed ML packages, an immutable revision declaration, and a pre-existing local safetensors model. The resulting unreviewed LoRA adapter is saved only in a locally ignored or outside-repository directory; no Ollama export/quantization or marketplace/model release is performed.
Data are FIM-encoded with Qwen's documented special tokens, loss applies
only to the missing code and EOS, and test split remains sealed. The default
plan never calls PyTorch or a network. Run npm run test:model-training
for deterministic offline standard-library unit tests. Real training
has not been executed here, and minimal mechanically passing input is
far from a useful training dataset. See train/README
and data policy.
The train/ directory now contains a voluntary
local-only training program using a separately reviewed
Qwen2.5-Coder-0.5B
base candidate and only explicitly supplied source records which pass
the current offline dataset audit. It builds Qwen-style FIM examples,
masks prefix and suffix loss, trains only on declared training records,
validates separately and never consumes the held-out test split. The
program requires explicit rights/checkpoint approval flags, an
independently reviewed local safetensors checkpoint and optional
user-installed ML dependencies before starting optimization. All
candidate inputs/outputs stay local; adapter directories are Git-ignored
and excluded from the VSIX. The default command makes a sanitized
offline plan only, and npm run test:model-training exercises its
FIM encoding and approval gates without downloading data or weights.
No approved corpus, GPU training session, adapter/weights, actual quality measurements or deployable model were created by this source change. The 15-record mechanical audit minimum cannot produce a useful trained model, and reviewed licensing/provenance remain required.
Local FIM adapter verification (development after v1.4.1): a separate optional Python smoke runner supports a plan-only mode and an explicit opt-in local base-versus-LoRA-adapter check over the original 40 synthetic fixtures. It runs no network requests, prints no generated code, and reports only source-free per-language strict matches. This is distinct from Ollama benchmark protocol v3; the two cannot be directly compared, and neither is a full correctness/accessibility/security evaluation.
The opt-in LoRA prototype now provides a seeded, language-balanced microbatch sampler and per-language validation loss alongside sample-weighted and macro validation means. The separate train/validation/test boundaries remain enforced, and the test holdout is not passed to the training loop or validation metrics. Offline synthetic sampler tests require only Python standard library.
This only improves experiment design and diagnostics. It does not mean any real weights were trained, that FIM output quality improved, or that a useful five-language dataset, real baseline, GPU or local adapter exists. Experiment instructions.
The separate optional local Hugging Face synthetic smoke now tags each real base/LoRA run with a deterministic hash of the fixed public fixture subset, the ordered FIM request contract, declared base revision and SHA-256 of the local base config and tokenizer metadata. The default dry plan has no base metadata because it opens no model files.
npm run model:smoke:compare -- --base base-smoke.json --adapter adapter-smoke.json
compares two manually captured real smoke JSON reports offline.
It validates the two report kinds, recomputes the fixed fixture and
request hashes locally, checks exact per-language totals and refuses
changed checkpoint declarations, corrupt/mixed/legacy JSON or differing
generation settings. Output contains only aggregate differences and
fixed caveats; neither user report paths nor generated source are echoed.
This adds no trained weights, checkpoint downloads, adapter export or measured inference. Report hashes do not prove the identity of model weights, adapter application or equivalent execution hardware. Read the full local model instructions and do not compare this HF v2 protocol with Ollama synthetic v3.
The local LoRA trainer now writes an unreviewed, source-free
lineloom-local-adapter-receipt-v1 metadata record with relative
adapter file SHA-256 digests, local base config/tokenizer digests,
training counts and the declared checkpoint revision. An offline
npm run model:adapter:verify -- --base-dir ... --base-revision ... --adapter-dir ...
command detects missing, changed, unexpected or symlinked adapter files.
Actual adapter-vs-base HF smoke and comparisons require this matching
receipt before treating an adapter as eligible.
The receipt detects accidental mismatches, not whether weights were honestly trained, legally permissioned, correct or authentic. The full base weights are not hashed or proven and the receipt is not signed. No checkpoint, dataset, trained weights, GPU run or released model was produced by this code. See local details.