libphptok

A standalone PHP tokenizer in C, built from PHP's own lexer.

Instead of hand-writing (and forever chasing) a PHP grammar, libphptok vendors Zend/zend_language_scanner.l straight from php-src, compiles it free of the Zend runtime behind a small stub layer, and exposes it through a tiny C API. Output matches PHP's PhpToken::tokenize() byte-for-byte, verified per php-src version against the real engine — so you get exact PHP lexing with no PHP runtime, no per-file php subprocess, and no grammar drift.

Pinned baseline: php-8.5.7 (a stable release tag; also the version the differential harness checks against locally).

Why this exists

token_get_all() is only reachable from inside a running PHP process. Anything outside PHP that needs to tokenize PHP — a linter, LSP server, static analyser, or clone detector written in C/Rust/Go/Nim — otherwise has to shell out to php per file or embed a hand-maintained grammar (e.g. tree-sitter-php) that only approximates the real lexer. libphptok is the third option: the authoritative lexer itself, as a linkable library, kept honest by differential testing.

API

#include <phptok.h>

phptok_tokenizer *t = phptok_new(src, len);   // copies src
phptok_token tok;
while (phptok_next(t, &tok)) {
    // tok.id (T_* or single-char < 256), tok.line, tok.text, tok.text_len
    // tok.text points into tokenizer memory; valid until phptok_free
    printf("%s @ line %d\n", phptok_token_name(tok.id), tok.line);
}
phptok_free(t);

Each token also carries start_byte/end_byte (0-based; start_byte equals PHP's PhpToken->pos), verified against the engine by the differential test.

Not reentrant: the underlying Zend scanner uses global state, so one tokenizer is active at a time (as in PHP itself). Fine for one-shot / batch tokenizing.

Swapping from tree-sitter-php

Read this first — it depends entirely on how you use tree-sitter-php. tree-sitter produces a parse tree; libphptok produces a token stream. What bridges and what doesn't:

You use tree-sitter-php for…	Swappable?
syntax highlighting, folding, token-type scanning (leaf level)	Yes — via phptok_ts.h
byte ranges / (row, column) points per token	Yes — same shim
named nodes, fields, child/parent, tree queries (.scm)	No — that's tree structure; libphptok has no parser by design

For the swappable cases, include/phptok_ts.h is a header-only shim presenting tokens as tree-sitter-flavored leaf nodes with familiar names:

#include <phptok_ts.h>

phptok_tokenizer *t = phptok_new(src, len);
phptok_ts_cursor cur; phptok_ts_cursor_init(&cur, t);
phptok_ts_node n;
while (phptok_ts_cursor_next(&cur, &n)) {
    // n.type (const char*), n.symbol, n.start_byte, n.end_byte,
    // n.start_point.{row,column}, n.end_point.{row,column}   (all 0-based)
}
phptok_free(t);

Name mapping: ts_node_type → n.type, ts_node_start_byte → n.start_byte, ts_node_start_point → n.start_point, TSTreeCursor walk → phptok_ts_cursor_next. Rows/columns are 0-based and columns are byte offsets within the line, exactly as tree-sitter's TSPoint. The shim's point math is verified against the engine (make test also runs the --bridge differential: start row from PhpToken->line, start byte from ->pos).

Build

make            # -> build/libphptok.a and build/phptok-dump
make test       # differential test vs. the local PHP engine over tests/corpus
make clean

Requires re2c (grammar -> scanner C), a C compiler, and python3 (the two generator scripts). php is needed only for make test, not to build.

How it's built (and stays maintainable)

Every generated artifact lands in build/; nothing generated is committed.

Step	Tool	Input -> Output
1. scan	re2c	vendor/<tag>/zend_language_scanner.l -> build/zend_language_scanner.c
2. strip	bin/strip-scanner.py	full scanner -> just lex_scan() + the ~20 helpers it reaches (drops the compiler/parser/AST/highlighter functions in the same file, unreachable from the lexer-only tokenize() path)
3. tokens	bin/gen-tokens.py	vendor/<tag>/zend_language_parser.y + vendor/<tag>/token-values.txt -> build/phptok_tokens.h (enum + name table, incl. token_name() display overrides)
4. compile	cc	scanner + src/stubs/ + src/phptok.c -> build/libphptok.a

Token ids are PHP's real constant values (T_STRING == 262, …), read from vendor/<tag>/token-values.txt (dumped once per version by bin/dump-token-values.php). This matters wherever the numeric id is observed downstream — e.g. phpcpd's id & 255 signature byte, where a T_* low byte can alias a single-char token; a faithful consumer must reproduce PHP's exact numbering.

The stub layer (src/stubs/) supplies standalone replacements for the Zend symbols the scanner references (zend_string, SCNG/CG/EG globals, the allocator, stacks). See src/stubs/README.md for the full dependency table.

Updating to a new PHP version

make extract PHP_TAG=php-8.6.0     # vendor grammar + token-values, diff vs last tag
make test PHP_TAG=php-8.6.0        # rebuild and re-verify against the engine

(make extract dumps token-values.txt from the php on PATH, which should be the same version as the tag — the differential test catches a mismatch.)

bin/extract-scanner.sh classifies each version bump as grammar-only (new token rules — usually just works) or plumbing-touching (SCNG/CG/EG/ arena changes — review the stub layer). Historically bumps are small and additive (see the checked-in *.diff-from-* reports). The differential test is the backstop: if a change breaks byte-exactness, it fails loudly against the real engine.

Layout

include/phptok.h           public API (tokenizer)
include/phptok_ts.h         public API (tree-sitter compatibility shim, header-only)
src/phptok.c               API implementation (drives lex_scan)
src/tool/phptok-dump.c     token CLI (public-API-only) + native test side
src/tool/phptok-ts-dump.c  bridge CLI + native side of the --bridge test
src/stubs/                 Zend-runtime replacements
  phptok_zend_stubs.h        the replacements
  zend/*.h                   stub headers the vendored scanner #includes
  globals.c                  the global state instances
bin/                       extractor + generators + oracles + differential runner
  dump-tokens.php            token oracle (PhpToken)
  dump-ts.php                bridge oracle (points derived from PhpToken + source)
vendor/<tag>/              vendored php-src grammar (the only upstream input)
tests/corpus/              PHP fixtures exercised by the differential test

LICENSE

BSD-3-Clause: Copyright 2026 Luciano Federico Pereira