[Compiler] Implement shared tokenizer + recursive-descent parser (ADR 0010)

[cssbruno]

Follow-up from #299 (ADR 0010) and PR #300.

Problem

ADR 0010 (docs/engineering/decisions/0010-tokenizer-recursive-descent-parser.md)
accepted the direction: replace the line-oriented parser with a single shared
tokenizer and a recursive-descent parser with error recovery, behind the stable
gwdkast AST seam. The decision is recorded; the implementation is not started.
This issue tracks building it.

Relevant code:

• internal/lang/lexer.go: the rune scanner to promote into the shared lexer
• internal/parser/syntax.go, patterns.go: the line-oriented parser and
  per-line lexLine to retire
• internal/parser/braces.go: brace state that becomes ordinary lexer state
• internal/gwdkast: the stable AST output contract

Scope

Implement the ADR in phases behind the AST seam:

1. Promote internal/lang's scanner into the shared tokenizer (with byte
   offsets, [Compiler] Carry byte offsets in source positions #294 follow-up) and retire internal/parser lexLine.
2. Build a recursive-descent parser to gwdkast.File with error recovery
   (synchronize at declaration boundaries and block braces; accumulate
   diagnostics instead of returning on the first error).
3. Gate with golden AST-equivalence tests against the line-oriented parser and
   the conformance corpus ([Language] Add a machine-checked .gwdk conformance corpus #295); cut over per declaration kind; remove the
   line-oriented path when equivalence holds.

The AST-backed formatter and granular per-construct diagnostic codes deferred in
#250 consume this parser and are out of scope here.

Acceptance Criteria

• One shared tokenizer feeds both the compiler parser and editor/CLI tooling.
• One syntax error no longer hides the rest of the file; diagnostics accumulate.
• AST output matches the line-oriented parser for the supported subset until
  cutover, verified by golden equivalence tests and the corpus.
• The line-oriented parser and lexLine are removed once equivalence holds.

Verification

go test ./internal/parser ./internal/lang ./internal/compiler ./cmd/gowdk

[cssbruno]

Absorbing #301 (byte-offset population from the front-end) into phase 1 of this issue: the shared tokenizer emits byte offsets on every token, so offset population is part of building the tokenizer rather than a separate task against the line-oriented parser (where it would be dormant). #294 already landed the source.SourcePosition.Offset substrate and PositionAt/OffsetOf helpers this consumes.

[cssbruno]

Phase 1 started in #308: the shared internal/lang tokenizer now records each token's 0-based byte Offset (verified byte-accurate against source.OffsetOf). This folds in the closed #301 and is the exact-span substrate the recursive-descent parser will consume.

Remaining phases (genuinely multi-PR, not started):

• Phase 2: recursive-descent parser over the token stream producing gwdkast.File, with error recovery (synchronize at declaration boundaries / block braces; accumulate diagnostics instead of returning on the first error).
• Phase 3: gate with golden AST-equivalence tests against the line-oriented parser + the conformance corpus ([Language] Add a machine-checked .gwdk conformance corpus #295); cut over per declaration kind; remove the line-oriented path and lexLine once equivalence holds.

The AST-backed formatter and granular per-construct diagnostic codes (#250) consume phase 2's output and stay out of scope here.

[cssbruno]

Phase 2 added in #308: the recursive-descent outline parser (internal/lang/outline.go) is the first live consumer of the tokenizer — it parses the top-level declaration structure with error recovery (a malformed line no longer hides the rest) and brace-aware block ranges, and is wired as the LSP textDocument/documentSymbol provider. This makes the phase-1 byte-offset work non-dormant.

Still open here: phase 3 — extend the recursive-descent parser to produce a full gwdkast.File (block bodies, view markup) with AST-equivalence gating against the line-oriented parser, then cut over per declaration kind and retire the line parser + lexLine. That's the large remaining rewrite.

[cssbruno]

Phase 3 started in #309: lang.ParseTopLevel is the first recursive-descent slice that emits real gwdkast nodes (package, imports, uses) over the shared tokenizer, anchored by an equivalence test against parser.ParseSyntax and a recovery test that shows it surfacing declarations past a malformed line the line parser bails on.

Remaining phase-3 work (the large cutover): extend ParseTopLevel to a full gwdkast.File (metadata routing into typed fields, block bodies, view markup, component contracts, endpoints) under the equivalence harness, then cut over per declaration kind and retire parser.ParseSyntax + lexLine. The tokenizer (phase 1), outline parser + recovery (phase 2), and now the equivalence/recovery harness + declaration nodes (phase 3 start) are the foundation that work builds on.