Skip to content

REPORT.md

Latest commit

 

History

History
243 lines (193 loc) · 17.6 KB

REPORT.md

File metadata and controls

243 lines (193 loc) · 17.6 KB

Project Status Report — Native C++ Scripting Engine ✅

Summary

This document summarizes what has been completed so far and provides a detailed, actionable checklist of remaining work to follow the plan in .github/prompts/plan-nativeCppScriptingEngine.prompt.md.

Completed Work ✅

  • Project scaffolding and repository metadata

    • Added CMakeLists.txt (top-level) with C++20 defaults and Ninja-friendly configuration
    • Created README.md, LICENSE (MIT), and .gitignore
    • CI skeleton: .github/workflows/ci.yml (basic CMake/Ninja job)

  • Basic code structure and AoT runtime stubs

    • src/main.cpp — entry point, run loop, subsystem initialization
    • Subsystem stubs:
      • src/engine/engine.h, src/engine/engine.cpp (JIT scaffold)
      • src/sandbox/sandbox.h, src/sandbox/sandbox.cpp (Landlock/seccomp stubs)
      • src/services/services.h, src/services/services.cpp (SQLite, MailApp, PropertyStore stubs)

  • Configuration and scripts

    • config/server.yaml (placeholder server settings)
    • config/syscalls.conf (placeholder seccomp whitelist)
    • scripts/README.md (user script layout)

  • Build/test attempts

    • Local build attempted in CI style, but cmake is not installed in the current environment (build step aborted with cmake: command not found).
      • Added a reproducible Docker-based dev image (Dockerfile.dev) and a small helper script at scripts/dev_setup.sh to build and run the image locally.

Additional completed items (implementation work)

  • Prototype Admin UI: src/web/ui/index.html (static SPA placeholder). ✅
  • Prototype HTTP static server: src/web/simple_http.{h,cpp} and src/web/static_server.*, wired into main and serving src/web/ui on port 8081. ✅
  • Graceful shutdown (SIGINT/SIGTERM handling) in src/main.cpp. ✅
  • /api/status endpoint that reports uptime and subsystem health (engine/services). ✅
  • Subsystem health API additions: engine::is_initialized() and services::is_initialized(). ✅
  • Integration test: tests/api_status_test.sh and CI step to run it. ✅
  • Updated CMakeLists.txt to include src/web and src/triggers sources. ✅

Detailed Checklist — Remaining Work (actionable, phase-aligned) 🔧

Phase 0 — Essential infra and CI

  • Add third_party/ layout and submodule strategy document (LLVM, Drogon, SQLite, mailio, libcron)
  • Add reproducible build instructions for local dev (docker or dev script that ensures musl toolchain, CMake >= 3.22, Ninja)
  • Improve CI to build dependencies (split job: deps build, core build, unit tests)
  • Add CONTRIBUTING.md with developer workflow and Docker dev image usage (new)
  • Add Docker-based dev image and helper script (see Dockerfile.dev, scripts/dev_setup.sh)
  • Improve CI to build dependencies (split job: deps build, core build, unit tests)

Phase 1 — Core JIT engine (engine/)

  • Evaluate Cling vs ClangREPL — ClangREPL chosen (see notes)
  • Add third_party/llvm/ submodule or FetchContent setup for minimal LLVM required for chosen JIT
  • Implement ClangREPL bootstrap:
    • Minimal startup harness that preloads a small "runtime universe" header set
    • Compile & execute a tiny user script (smoke test)
    • Measure cold start and aim for <100ms baseline (profiling harness)
  • Add CMake targets to optionally build the JIT tooling and unit test harness

Notes / next steps (JIT): We chose ClangREPL (LLVM 16+) as the JIT backend for the project. The concrete next steps for contributors:

  • Add third_party/llvm/ via git submodule or use FetchContent in CMake to fetch a minimal LLVM build configured with -DLLVM_TARGETS_TO_BUILD=\"host\" and -DLLVM_BUILD_TOOLS=OFF.
  • Create an engine/jit_bootstrap.cpp with a minimal ClangREPL harness that:
    • Preloads commonly used headers (iostream, string, vector, ) into the REPL session.
    • Compiles and runs a simple snippet (e.g., std::cout << "hello";) as a smoke test.
  • Add a CMake option -DENGINE_JIT=ON to build and run the harness during CI for smoke tests.

Phase 1 — Security sandbox (sandbox/)

  • Implement Landlock wrapper (C++):
    • Runtime kernel capability check (detect <5.13 and fail if missing)
    • Ruleset builder and per-script rules application
    • Unit tests and integration tests (requires kernel with Landlock)
  • Implement Seccomp-BPF loader that reads config/syscalls.conf
  • Implement cgroups v2 integration for ephemeral execution cgroups (CPU, memory, pids)
  • Decide and implement fallback policy for older kernels (strict logging fallback vs. hard requirement)

Notes / next steps (Sandbox): The repository contains a sandbox stub; next actions for implementers:

  • Implement a runtime detection helper that calls landlock_create_ruleset (or checks /proc/self/attr availability) and returns a clearly documented capability flag. Add an API sandbox::is_landlock_available().
  • Implement a RulesetBuilder that reads a JSON/YAML policy (paths + access flags) and applies the Landlock ruleset for the current process before running user code.
  • Provide a conservative fallback mode when Landlock is not available: deny all write attempts by default, enforce strict seccomp policy from config/syscalls.conf, and log the degraded mode clearly in startup logs. Document that Landlock is recommended for production.

Updates: A sample policy file has been added at config/landlock_policy.conf (format: <path> <ro|rw>). The sandbox::apply_default_policy() routine now attempts to load this file and applies the configured per-path rules via the RulesetBuilder. If the file is absent, a default minimal ruleset is created and applied. Remember: Landlock is a hard requirement (kernel ≥5.13); the process will abort startup if Landlock is unavailable.

Seccomp update: Added support to load a seccomp syscall whitelist from config/syscalls.conf (one syscall name per line). A sample file config/syscalls.example.conf is provided. When libseccomp is available at compile time, the runtime will apply the whitelist after the Landlock ruleset. The conservative security posture is to require both Landlock and an applied seccomp whitelist for startup.

Phase 2 — Integrated services (services/)

  • SQLite wrapper with WAL and connection pooling
  • MailApp wrapper over mailio or chosen alternative (decide Boost strategy)
    • Evaluate statically vendoring Boost vs alternate lightweight SMTP library
  • PropertyStore key-value API
  • Quota tracking (per-script invocation/day limits, mail quotas)
  • Add unit & integration tests for services

Notes / next steps (Services): To avoid heavy Boost dependency in early phases, prefer a lightweight SMTP prototype:

  • Prototype MailApp over libcurl's SMTP support or a minimal SMTP client implemented as a small internal module (only send mail, no IMAP). If more features are required later, re-evaluate mailio+Boost with static vendoring.
  • Implement services/sqlite_pool.cpp with a small connection pool and WAL mode enabled; add tests that validate concurrent reads/writes.
  • Add PropertyStore backed by SQLite and expose a C++ API that is easy to use for scripts.

Phase 3 — Web framework & triggers (web/, triggers/)

  • Integrate Drogon (v1.9.x) or evaluate Oat++ (size tradeoffs)
  • Implement request dispatcher to map HTTP endpoints to JIT handlers (doGet/doPost contract)
  • Implement HtmlService templating and streaming responses
  • Integrate libcron for scheduled triggers and TriggerBuilder API
  • Add tests and a small sample webapp in scripts/webapps/ for CI
  • Add tests and a small sample webapp in scripts/webapps/ for CI (added tests/api_status_test.sh smoke test that verifies /api/status) Notes / next steps (Web): We've added a minimal prototype static server and SPA; next steps are:
  • Integrate Drogon behind a CMake option -DWEB_FRAMEWORK=DROGON and provide example controllers that dispatch requests into the JIT engine.
  • Implement a request Event object passed to scripts with doGet(Event) semantics and build a simple Dispatcher that maps routes to compiled script handlers.
  • Add an administrative API for script lifecycle (upload, enable/disable, list, logs) and hook the SPA to these endpoints.

Web UI (admin console)

  • Design single-page Admin UI (React/Vue/Vanilla) for managing scripts, triggers, and viewing logs/metrics
  • Implement static asset serving and API endpoints for:
    • script listing/upload/manage
    • trigger scheduling and management
    • logs and real-time status
  • Add a small prototype SPA under src/web/ui/index.html (done: placeholder UI)
  • Add tests that exercise the web UI endpoints and include a small sample app in scripts/webapps/

Notes / next steps (Web UI): The prototype SPA exists; implementers should:

  • Choose a small framework (Vanilla/Preact/Alpine) or keep the UI static for minimal runtime size.
  • Add API endpoints (scripts, triggers, logs) and write integration tests that exercise the entire flow (upload → run → view logs).
  • Consider embedding the SPA into the single binary (resource arrays) for single-binary distribution.

Phase 4 — Library system & binary consolidation

  • Implement versioned module loader with semantic version resolution
  • Final static build strategy:
    • musl static link targets
    • LTO and symbol visibility for size reduction
    • Investigate final binary size target (~34MB) and measurement harness
  • Cross compilation and minimal runtime packaging

Notes / next steps (Binary consolidation): For the final single-binary deliverable:

  • Build dependencies statically into .a archives and link against musl; provide a documented Docker cross-build image that contains the musl toolchain and matching libc headers.
  • Enable LTO (-flto), -ffunction-sections -fdata-sections and -Wl,--gc-sections, set CMAKE_EXE_LINKER_FLAGS appropriately, and use strip to minimize size. Provide a reproducible measurement script that reports final binary size.
  • Carefully review LLVM linkage: minimize the LLVM surface area by building only required components and using -DLLVM_TARGETS_TO_BUILD="host".

Documentation & Misc

  • API docs for script authors (auto-inclusion headers, doGet/doPost, MailApp, PropertyStore)
  • Security policy docs for administrators (Landlock rules, seccomp whitelist guidance)
  • Release/deployment notes and example Dockerfile for reproducible builds

Developer Instructions — Getting started 🛠️

These instructions are targeted at contributors who will implement the remaining phases.

Prerequisites

  • Install development tools: cmake (>=3.22), ninja-build, clang, gcc, make, libssl-dev (for OpenSSL) and pkg-config.
  • Recommended: use the included Docker-based dev image (create one based on debian:bookworm or ubuntu:24.04) and install musl-tools, build-essential, cmake, ninja-build, and clang to build static musl binaries.

Dev image (new): A Dockerfile.dev and a helper script scripts/dev_setup.sh were added to provide a reproducible development environment. Quick usage:

  • Build the image: ./scripts/dev_setup.sh build
  • Start an interactive shell with the repo mounted: ./scripts/dev_setup.sh shell
  • Build inside the image: ./scripts/dev_setup.sh build-run

This addresses the earlier cmake: command not found issue and can be used both locally and in CI to ensure consistent tooling.

Quick local build

  1. Create a build dir: cmake -S . -B build -G Ninja -DCMAKE_BUILD_TYPE=Release
  2. Build: cmake --build build -j $(nproc)
  3. Run the binary: ./build/native_node (UI served at http://localhost:8081)

Run tests

  • Unit/integration test pattern: simple shell scripts under tests/.
  • Run the api status test locally: chmod +x tests/api_status_test.sh && ./tests/api_status_test.sh.

Developing the JIT (ClangREPL) integration

  1. Add FetchContent or a submodule for llvm configured minimally (see notes above).
  2. Add src/engine/jit_bootstrap.cpp that starts a ClangREPL session and exposes a minimal API for evaluating snippets.
  3. Protect JIT code behind CMake option -DENGINE_JIT=ON and add a smoke test that runs during CI.

Implementing Landlock and sandbox policies

  1. Add sandbox/detect.cpp to expose sandbox::is_landlock_available().
  2. Implement RulesetBuilder and a policy loader that reads JSON/YAML from config/.
  3. Implement the conservative fallback (strict seccomp + deny write) and document the security implications.

MailApp & services

  1. Start with a small SMTP sender using libcurl (no Boost). Add CMake option -DUSE_MAILIO=ON if switching to mailio later.
  2. Implement services/sqlite_pool.cpp and tests that validate WAL and concurrency.

Web framework & UI

  1. Integrate Drogon behind an option -DWEB_FRAMEWORK=DROGON and provide example controllers.
  2. Implement the request dispatcher (maps URL to ScriptHandler), and test via the Admin UI.
  3. Consider embedding static assets into the binary for single-binary distribution (e.g., xxd -i or a small resource compiler).

CI and reproducible builds

  • Make CI build steps explicit: deps build, core build, run unit tests, run integration tests.
  • Provide a Dockerfile.dev for reproducible builds and cross-builds; add a CI job that runs builds inside that image.

Contribution & code style

  • Create feature branches named feat/<short-desc> or fix/<short-desc>; open PRs against main for review.
  • Include tests for all non-trivial functionality and document configuration changes in config/.
  • Keep commits small and focused; add a short description of design decisions when implementing security or JIT changes.

If you'd like, I can also add a CONTRIBUTING.md, a sample Dockerfile.dev, and CI changes to implement the recommended build pipeline next.

Open design questions (need your guidance) ⚖️

  • Landlock requirement: should kernel ≥5.13 be a hard requirement, or should we provide a degraded fallback mode for older kernels?
  • JIT backend: ClangREPL (chosen) (LLVM 16+).
  • Mail library & Boost: should we statically vendor Boost (increase binary complexity), or choose a smaller SMTP library to avoid Boost?

Immediate next actions (recommended) ▶️

  1. Landlock policy chosen: kernel ≥ 5.13 is a hard requirement (no degraded fallback mode).
  2. JIT backend chosen: ClangREPL (LLVM 16+).
  3. Add third_party/ and vendor minimal LLVM for the chosen JIT, and add CMake targets to build a small Cling/REPL test harness.
  4. Add a Docker-based dev environment for reproducible builds (ensure cmake, ninja, and toolchain for musl available).

Note: I attempted a local build as a smoke test; cmake was not available in the environment (error: cmake: command not found). Adding a Docker dev image or updating CI to include the dependencies will allow us to run the build jobs automatically.

Report generated by GitHub Copilot (Raptor mini (Preview)).

Feature Google Apps Script (GAS) native_node (current repo) Gap / Notes
Language & Execution Model JavaScript (V8-based managed runtime) — instant edit/exec C++ JIT (ClangREPL/LLVM planned) — aims for REPL/JIT ⚠️ Partial: JIT stub added; ClangREPL not yet integrated
REPL / Instant deploy Built-in editor + instant deploy Planned JIT REPL & instant deploy (engine scaffold) ⚠️ Partial: smoke test and flags added; full REPL missing
Web App endpoints (doGet/doPost) First-class: deploy webapps easily Static server exists; dispatcher → JIT planned (Drogon) ⚠️ Partial: static UI done; dynamic dispatch not implemented
HTML templating (HtmlService) Yes (templating & sandboxing) Planned (HtmlService in design) ❌ Not implemented
Triggers / Scheduling Time/event triggers built-in Planned (libcron) ❌ Not implemented
Mail (MailApp) Integrated MailApp (SMTP) Planned (Mail prototype via libcurl) ❌ Not implemented
Persistence / Properties PropertiesService, CacheService, integrated storage Planned: SQLite + PropertyStore (stubs) ❌ Not implemented
Sandboxing & Security Host-managed JS sandbox and API quotas Kernel-level sandboxing (Landlock + seccomp + cgroups) planned and partially implemented ✅ Partial: Landlock REQUIRED and runtime detection implemented; RulesetBuilder skeleton + test added
Per-path filesystem rules Very limited (managed environment) Planned per-path Landlock rules (skeleton exists) ⚠️ Partial: add rule APIs & policy loader needed
Syscall & resource control Enforced by GAS platform Seccomp + cgroups planned ❌ Seccomp & cgroups not implemented yet
Versioned libraries / modules Library system with versions Planned module loader (design only) ❌ Not implemented
Quotas & limits per script Built-in quota enforcement Planned quota tracking (design) ❌ Not implemented
Admin UI & management Console + deployment tooling Admin SPA placeholder (ui) ⚠️ Partial: UI placeholder but management APIs missing
Observability (logs/metrics) Built-in logging and execution logs Planned (logs + API) ❌ Not implemented
Single-binary deployment Not applicable (cloud service) Aim: single static musl binary (option added) ⚠️ Partial: MUSL_STATIC CMake option + docs added; CI build not yet done
Local & reproducible dev No native local runtime emulation Docker dev image + build scripts exist ✅ Implemented (Docker dev image & scripts)
Cold-start / perf target Managed, varies Target <100ms cold start via JIT ❌ Not yet measured; JIT not integrated