Release 3.1.0

Surveilr v3.1.0 Release Notes

Note: Surveilr is a Resource Surveillance & Integration Engine that helps organizations securely collect, analyze,
and audit critical data across their systems. It creates a SQL queryable private evidence warehouse to ensure your
organization's cybersecurity, quality metrics, and regulatory compliance efforts are backed by reliable data.

This release focuses on production stability with critical database concurrency fixes, platform upgrades, and
dependency updates.

Release Notes

1. Critical Database Lock Fix
   Fixed server crashes during concurrent database access in osquery-ms:

• Previously: Server panicked with "database is locked" errors, causing downtime
• Now: Graceful error handling with proper logging and error responses
• 30-second busy timeout remains active for automatic retry logic

2. SQLPage Platform Upgrade
   Updated to SQLPage v0.37.0:

• Enhanced web UI performance and rendering speed
• Latest security patches and bug fixes included
• Better dashboard visualization performance for surveilr web interfaces

Release 3.0.0

Surveilr v3.0.0 - Drizzle ORM Integration

Summary

Migrated internal schema generation from SQLa to Drizzle ORM - a lightweight, type-safe TypeScript ORM. This establishes the foundation for optional type-safe database queries while maintaining our SQL-first philosophy.

What Changed

Schema Generation (Internal)

• Replaced: SQLa-based lifecycle.sql.ts → Drizzle-generated bootstrap SQL
• New: Type-safe schema definitions in lib/std/drizzle/models.ts and views.ts
• Result: Same RSSD structure with enhanced TypeScript support

Developer Experience

• Added: Optional type-safe query helpers for complex scenarios
• Maintained: SQL views remain the preferred approach for business logic

File Organization

lib/std/drizzle/
├── models.ts              # RSSD table schemas
├── views.ts              # SQL view definitions
├── bootstrap.sql.ts       # Schema generator
└── drizzle-lifecycle.ts   # Migration cells

Migration Impact

• End Users: No changes to surveilr CLI commands
• Developers: Optional access to type-safe queries when needed
• Databases: Same RSSD structure, generated via Drizzle instead of SQLa

Technical Validation

• ✅ Identical 65+ table structure generated
• ✅ All foreign key constraints preserved
• ✅ File ingestion, transforms, multitenancy functional
• ✅ Web UI and code notebook systems working
• ✅ Complete test suite passing

---

Release 2.2.0

surveilr v2.2.0 Release Notes

🚀 What's New

JSONL File Ingestion Support

• New Format Support: Added comprehensive JSONL (JSON Lines) file format ingestion capabilities
• Streaming Data Processing: Efficiently handles large streaming JSON datasets line-by-line
• Automatic Schema Detection: Intelligently detects and processes JSONL file structures.

How JSONL Ingestion Works

Unlike regular JSON files that contain a single JSON object or array, JSONL files contain one valid JSON object per
line:

{"id": 1, "name": "Alice", "timestamp": "2025-09-01T10:00:00Z"}
{"id": 2, "name": "Bob", "timestamp": "2025-09-01T10:01:00Z"}
{"id": 3, "name": "Charlie", "timestamp": "2025-09-01T10:02:00Z"}

Ingestion Process:

1. Line-by-Line Reading: File is read sequentially, one line at a time
2. JSON Validation: Each line is validated as proper JSON
3. Individual Processing: Each JSON object is processed as a separate resource
4. Schema Evolution: Supports varying schemas across lines in the same file
5. Line-Specific URIs: Each line gets a unique URI with line number reference for precise tracking

URI Structure for JSONL:

Each JSON line creates a unique uniform_resource entry with line-specific URI:
/path/to/events.jsonl#L1 # First JSON object
/path/to/events.jsonl#L2 # Second JSON object
/path/to/events.jsonl#L3 # Third JSON object

Example:

File: /data/user-events.jsonl
Line 1: {"user": "alice", "action": "login", "timestamp": "2025-09-01T10:00:00Z"}
Line 2: {"user": "bob", "action": "logout", "timestamp": "2025-09-01T10:05:00Z"}

Results in uniform_resource entries with URIs:

/data/user-events.jsonl#L1
/data/user-events.jsonl#L2

This unique URI scheme allows precise tracking of which specific line in the JSONL file each resource originated
from, enabling accurate data lineage and debugging capabilities.

Release 2.1.0

surveilr v2.1.0 Release Notes

🚀 What's New

1. AI-Powered Natural Language to SQL (ask-ai)

• New Command: surveilr ask-ai sql converts natural language queries into SQL and executes them against your RSSD
• Smart Context Integration: Automatically retrieves relevant AI context from Surveilr notebook cells for
  domain-specific knowledge
• Multiple Output Formats: Support for table, JSON, CSV, and markdown output formats
• Flexible LLM Support: Works with OpenAI-compatible endpoints including local models (Ollama, etc.)

Basic natural language queries

surveilr ask-ai sql "show me all files ingested in the last week"
surveilr ask-ai sql "what devices have been scanned?"
surveilr ask-ai sql "find all JSON files larger than 1MB"

Different output formats

surveilr ask-ai sql "show device information" --output json
surveilr ask-ai sql "list recent sessions" --output csv
surveilr ask-ai sql "security audit summary" --output markdown

Development and debugging

surveilr ask-ai sql "show database tables" --show-query
surveilr ask-ai sql "count all records" --sql-only

🔧 Configuration

Environment variables for LLM configuration

export SURVEILR_LLM_API_KEY="your-api-key"
export SURVEILR_LLM_API_ENDPOINT="https://api.openai.com/v1/chat/completions" # For gpt or your preffered model
export SURVEILR_LLM_MODEL="gpt-4" # Or your preferred model

Release 2.0.0

surveilr v2.0.0 Release Notes

🚀 What's New

1. Enhanced Markdown Transformation Workflow

• Improved mdq Integration: Fixed mdq selector syntax and added comprehensive selector support
• Content Preservation: Markdown transforms no longer null out original content by default
• Better URI Tracking: Transform results now preserve source file paths (e.g., document.pdf/md-select:headers)

2. Dependencies Update

• Upgraded to SQLPage 0.36.1.

Release 1.8.11

surveilr v1.8.11 Release Notes

🎉 New Feature: Automatic Document Processing & Metadata Extraction

🚀 What's New

Surveilr now automatically extracts metadata and converts documents to markdown during ingestion - no configuration
required!

Supported File Types

• PDF files: Full metadata extraction + markdown conversion
• DOCX files: Full metadata extraction + markdown conversion
• Images (PNG, JPEG, GIF, etc.): Metadata extraction (dimensions, format, file size)

Release 1.8.10

Release notes for Resource Surveillance 1.8.10

Release 1.8.9

surveilr v1.8.9 Release Notes

🚀 What's New

1. Surveilr ingestion Improvements

• Image Ingestion Support - Fixed issues with image format ingestion during file processing
• GitHub API Rate Limiting - Enhanced rate limiting handling for GitHub PLM integration

2. Dependencies Update

• OIDC/SSO Support - Added OpenID Connect and Single Sign-On support for surveilr web UI
• SQLPage Upgrade - Updated to latest SQLPage version

Release 1.8.8

surveilr v1.8.8 Release Notes

🚀 What's New

1. Bug-fixes

Ingestion & PLM Issues (#320)

• Fixed TLS crypto provider initialization issues
• Fixed ingestion PLM issues with github

CSV Transform Issues (#194)

• Fixed CSV transform duplicate detection issues

File Carving (#299)

• Fixed file carving functionality

Release 1.8.7

Release notes for Resource Surveillance 1.8.7