04. Developer Guide

This guide is for developers who want to contribute to TritonParse, understand its architecture, or extend its functionality.

🏗️ Architecture Overview

High-Level Architecture

TritonParse consists of three main components:

┌──────────────────────┐    ┌──────────────────────┐    ┌──────────────────────┐
│   Python Backend     │    │   Processing         │    │   Frontend UI        │
│                      │    │                      │    │                      │
│ • Structured Logging │──▶│ • Log Parsing        │──▶│ • React Interface    │
│ • Triton Hooks       │    │ • Source Mapping     │    │ • IR Visualization   │
│ • Trace Generation   │    │ • Data Compression   │    │ • Code Comparison    │
│                      │    │ • Launch Analysis    │    │ • File Diff View     │
└──────────────────────┘    └──────────────────────┘    └──────────────────────┘
         │                           │
         │                           ▼
         │                  ┌──────────────────────┐
         │                  │   Reproducer         │
         │                  │                      │
         └─────────────────▶│ • Script Generation  │
                            │ • Tensor Rebuilding  │
                            │ • Template System    │
                            │ • Context Bundling   │
                            └──────────────────────┘

Component Roles

Python Backend - Captures Triton compilation events and generates structured logs. Core logic for IR parsing, trace processing, and source mapping.

Processing Pipeline - Transforms raw NDJSON logs into structured, compressed data with source mappings and launch analysis.

Frontend UI - React/TypeScript web application for interactive visualization using Monaco Editor for code display.

Reproducer System - Generates standalone Python scripts to reproduce kernel executions from trace files with full context.

💡 See Project Structure below for detailed file organization.

📁 Project Structure

tritonparse/
├── tritonparse/                 # Python package
│   ├── __init__.py
│   ├── __main__.py              # CLI entry point
│   ├── cli.py                   # CLI argument parsing (parse, reproduce, info, diff, bisect, compat-build)
│   ├── structured_logging.py    # Core logging infrastructure to capture events
│   ├── context_manager.py       # TritonParseManager context manager
│   ├── shared_vars.py           # Shared state and variables for the package
│   ├── tp_logger.py             # Logger configuration
│   ├── _json_compat.py          # JSON serialization compatibility
│   ├── clp.py                   # CLP (Compressed Log Processor) format support
│   │
│   ├── parse/                   # Core parsing logic
│   │   ├── common.py            # Common utilities and helper functions
│   │   ├── utils.py             # Main parsing entrypoint (`unified_parse`, `oss_run`)
│   │   ├── trace_processor.py   # Processes raw trace files, groups events
│   │   ├── ir_parser.py         # Extracts source location from various IRs
│   │   ├── ir_analysis.py       # IR analysis utilities
│   │   ├── mapper.py            # Bidirectional mappings between IRs and Python source
│   │   ├── extract_source_mappings.py  # Source mapping extraction
│   │   ├── event_diff.py        # Launch event comparison
│   │   ├── source_type.py       # Source type definitions (TTIR, PTX, etc.)
│   │   └── sourcemap_utils.py   # Source mapping helper functions
│   │
│   ├── info/                    # Kernel query and info CLI
│   │   ├── cli.py               # Info subcommand CLI
│   │   ├── kernel_query.py      # Kernel lookup by name
│   │   └── parse_helper.py      # Parsing utilities for info command
│   │
│   ├── diff/                    # Compilation event comparison/diffing
│   │   ├── cli.py               # Diff subcommand CLI
│   │   ├── core/                # Core diff logic
│   │   │   ├── diff_engine.py   # Single event comparison
│   │   │   ├── trace_diff_engine.py  # Trace-level comparison
│   │   │   ├── event_matcher.py # Event matching
│   │   │   ├── kernel_matcher.py # Kernel matching
│   │   │   └── diff_types.py    # Type definitions
│   │   └── output/              # Output formatting
│   │       ├── event_writer.py  # Write diff events
│   │       └── summary_formatter.py  # Format summaries
│   │
│   ├── bisect/                  # Triton/LLVM/PyTorch regression bisection
│   │   ├── cli.py               # Bisect subcommand CLI
│   │   ├── base_bisector.py     # Base bisector class
│   │   ├── triton_bisector.py   # Triton commit bisection
│   │   ├── torch_bisector.py    # PyTorch commit bisection
│   │   ├── llvm_bisector.py     # LLVM commit bisection
│   │   ├── commit_detector.py   # Detect LLVM bumps in Triton
│   │   ├── pair_tester.py       # Test (Triton, LLVM) commit pairs
│   │   ├── env_manager.py       # Environment setup
│   │   ├── state.py             # Bisect state management
│   │   └── scripts/             # Bash scripts for bisection
│   │
│   ├── compat_builder/          # LLVM compatibility map builder
│   │   ├── cli.py               # Compat-build subcommand CLI
│   │   └── ...                  # Builder logic with AI-assisted fixes
│   │
│   ├── reproducer/              # Reproducer system for generating standalone scripts
│   │   ├── orchestrator.py      # Main reproducer entry point (`reproduce()`)
│   │   ├── cli.py               # CLI argument handling for reproduce command
│   │   ├── placeholder_replacer.py  # Template placeholder substitution
│   │   ├── utils.py             # Tensor creation and path utilities
│   │   ├── types.py             # Type definitions (KernelImportMode enum)
│   │   ├── ast_analyzer.py      # AST analysis for kernel code
│   │   ├── function_extractor.py    # Extract kernel functions
│   │   ├── import_parser.py     # Parse import statements
│   │   ├── import_resolver.py   # Resolve import paths
│   │   ├── multi_file_analyzer.py   # Multi-file analysis
│   │   ├── stub_generator.py    # Generate stubs for missing dependencies
│   │   ├── ingestion/           # Trace file parsing
│   │   │   └── ndjson.py        # NDJSON trace parsing and context extraction
│   │   └── templates/           # Code generation templates
│   │       ├── loader.py        # Template loading utilities
│   │       ├── example.py       # Default reproducer template
│   │       └── tritonbench.py   # TritonBench-compatible template
│   │
│   ├── ai/                      # LLM client abstraction
│   │   ├── client.py            # Unified LLM client interface
│   │   ├── parsers.py           # Response parsers
│   │   └── utils.py             # AI utilities
│   │
│   ├── validation/              # JSON schema validation
│   │   └── schemas/             # JSON schemas for events
│   │       ├── compilation.schema.json
│   │       └── launch.schema.json
│   │
│   └── tools/                   # Utility tools
│       ├── decompress_bin_ndjson.py  # Decompress .bin.ndjson files
│       ├── disasm.py            # Disassembly utilities
│       ├── extract_irs.py       # Extract IRs from trace files
│       ├── load_tensor.py       # Load tensor blob files
│       └── prettify_ndjson.py   # Pretty-print NDJSON
│
├── website/                     # React web application for visualization
│   ├── src/
│   │   ├── components/          # Reusable React components
│   │   │   ├── ArgumentViewer.tsx   # Displays kernel arguments with expandable details
│   │   │   ├── Callstack.tsx        # Call stack visualization
│   │   │   ├── CodeViewer.tsx       # Displays IR code with Monaco editor
│   │   │   ├── CodeComparisonView.tsx  # Side-by-side IR viewing with line mappings
│   │   │   ├── CompilationInfo.tsx  # Compilation metadata display
│   │   │   ├── CopyCodeButton.tsx   # Copy-to-clipboard button
│   │   │   ├── DataSourceSelector.tsx  # File/URL source selector
│   │   │   ├── DiffViewer.tsx       # Side-by-side diff view for text
│   │   │   ├── DiffComparisonView.tsx  # File diff comparison component
│   │   │   ├── ExternalLink.tsx     # External link component
│   │   │   ├── SingleCodeViewer.tsx # Single IR code viewer
│   │   │   ├── StackDiffViewer.tsx  # Stack diff visualization
│   │   │   ├── ToggleSwitch.tsx     # Toggle switch UI component
│   │   │   ├── TritonIRs.tsx        # IR navigation links
│   │   │   └── WelcomeScreen.tsx    # Initial welcome/loading screen
│   │   ├── pages/               # Main application pages
│   │   │   ├── KernelOverview.tsx   # Main analysis view for a kernel
│   │   │   ├── CodeView.tsx         # Focused view for a single IR file
│   │   │   ├── FileDiffView.tsx     # File diff page for cross-trace comparison
│   │   │   └── IRAnalysis.tsx       # IR analysis page
│   │   ├── context/             # React context providers
│   │   │   └── FileDiffSession.tsx  # File diff state management
│   │   ├── utils/               # Utility functions
│   │   │   ├── dataLoader.ts    # Data loading and processing from parsed logs
│   │   │   ├── fbDetection.ts   # Meta internal environment detection
│   │   │   ├── safeImport.ts    # Safe dynamic imports
│   │   │   └── tensor.ts        # Tensor data utilities
│   │   ├── App.tsx              # Main application component with routing
│   │   └── main.tsx             # Application entry point
│   ├── public/                  # Static assets and sample data
│   ├── scripts/                 # Build and deployment scripts
│   ├── package.json             # Frontend dependencies and scripts
│   └── vite.config.ts           # Vite build configuration
│
├── tests/                       # Test suite for the Python package
│   ├── __init__.py
│   ├── test_tritonparse.py      # Main test suite (CPU and CUDA tests)
│   ├── test_add.py              # Manual test example
│   ├── example_output/          # Sample output data
│   └── README.md                # Test documentation
│
├── run.py                       # Main CLI entry point with argparse subcommands
├── pyproject.toml               # Python project configuration
├── Makefile                     # Development commands
├── README.md                    # Project overview
├── CHANGELOG.md                 # Version history
├── CONTRIBUTING.md              # Contribution guidelines
├── CODE_OF_CONDUCT.md           # Code of conduct
└── LICENSE                      # BSD-3 license

🔧 Development Environment Setup

Prerequisites

• Python >= 3.10
• Node.js >= 22.0.0
• Triton >= 3.4.0 (latest version recommended)
• Git for version control

Installation Steps

# 1. Clone repository
git clone https://github.com/meta-pytorch/tritonparse.git
cd tritonparse

# 2. Install Python dependencies
make install-dev

# 3. Install website dependencies
cd website
npm install

Verify Installation

# Python: Check formatting and run tests
make format-check
make test

# Website: Start dev server
cd website
npm run dev

🛠️ Development Workflow

Code Style and Formatting

We use a comprehensive formatting pipeline:

Tool	Purpose	Configuration
Black	Code formatting	pyproject.toml
usort	Import sorting	pyproject.toml
Ruff	Linting	Built-in rules

Essential Commands

# Format code
make format

# Check formatting
make format-check

# Run linting
make lint-check

# Run tests (CPU only)
make test

# Run all tests (including CUDA)
make test-cuda

# Website development
cd website && npm run dev

Development Quality Checks

Before committing, ensure:

1. Code is formatted: make format
2. Linting passes: make lint-check
3. Tests pass: make test
4. Website builds: cd website && npm run build

🏗️ Backend Development

Core Components

Structured Logging (structured_logging.py) - Captures Triton compilation and launch events. Main functions: init() and init_with_env() for initialization.

Log Processing (utils.py) - Transforms raw logs into analyzable format. Entry point: unified_parse() for parsing NDJSON logs, extracting source mappings, and compressing data.

Source Mapping (extract_source_mappings.py) - Correlates lines between different IR stages (TTIR, TTGIR, LLIR, PTX, AMDGCN).

💡 See inline code documentation for detailed function signatures and parameters.

Adding New Features

1. Define the new data: Determine what new information needs to be captured.
2. Update structured_logging.py: Add logic to capture the new data within the appropriate hooks (e.g., pre-compilation, post-compilation).
3. Modify trace_processor.py: If the new data requires special processing or aggregation (like the launch analysis), add the logic here.
4. Update unified_parse(): Ensure the new data is handled correctly during the main parsing routine.
5. Write tests: Add unit and integration tests to tests/ to validate the new feature.

CLI Architecture

The CLI uses argparse with two main subcommands: parse and reproduce.

# Parse subcommand
tritonparseoss parse ./logs/ --out ./parsed_output

# Reproduce subcommand
tritonparseoss reproduce trace.ndjson --line 1 --out-dir repro_output

Implementation Files:

• run.py - Main CLI with argparse subparsers
• __main__.py - Entry point that calls run.main()
• utils.py - _add_parse_args() function
• reproducer/cli.py - _add_reproducer_args() function

💡 See Usage Guide for complete CLI reference and parameters.

Reproducer System

Generates standalone Python scripts from trace files.

Core Components:

• Orchestrator - Main reproduce() function, loads events and generates scripts
• Ingestion - Extracts kernel context, arguments, and metadata from NDJSON
• Templates - Customizable code templates (default: example.py)
• Placeholder Replacement - Substitutes template placeholders for imports, invocations, paths
• Utils - Rebuilds kernel arguments from JSON, handles tensor creation

Custom Templates Example:

# my_template.py
import torch
# {{KERNEL_IMPORT_PLACEHOLDER}}

if __name__ == "__main__":
    # {{KERNEL_INVOCATION_PLACEHOLDER}}
    print("Custom execution!")

Use with: tritonparseoss reproduce trace.ndjson --line 1 --template my_template.py

💡 See Reproducer Guide for comprehensive documentation on templates, tensor reconstruction, and advanced usage.

🎨 Frontend Development

Technology Stack

• React 19 - UI framework
• TypeScript - Type safety
• Vite - Build tool and dev server
• Tailwind CSS - Styling
• Monaco Editor - IDE-quality code display

Key Components

Data Loading (utils/dataLoader.ts) - Loads and processes trace files from URLs or local files.

Code Viewer (components/CodeViewer.tsx) - Displays IR code with syntax highlighting, line numbers, and source mapping.

IR Code View (components/CodeComparisonView.tsx) - Side-by-side IR viewing with synchronized scrolling and interactive line mapping.

File Diff View (pages/FileDiffView.tsx) - Cross-trace kernel comparison with customizable diff options.

Adding Frontend Features

1. Update data loader - Modify dataLoader.ts for new data fields
2. Create components - Add React components in website/src/components/
3. Integrate - Add components to pages (e.g., KernelOverview.tsx)
4. Style - Use Tailwind CSS for consistent styling
5. Test - Verify with npm run build and manual testing

Testing Frontend Changes

cd website

# Development server
npm run dev

# Type checking
npm run build

# Linting
npm run lint

# Test with sample data
# Load ./public/f0_fc0_a0_cai-.ndjson in browser

📊 Data Flow

End-to-End Data Flow

Python Code
     │
     ▼
Triton Compilation
(triggers Hook Events)
     │
     ▼
Structured Logging
     │
     ▼
Raw NDJSON Logs
     │
     ▼
Log Processing
  - Source Mapping
  - Launch Analysis
     │
     ▼
Compressed Data (.gz)
     │
     ▼
Web Interface
     │
     ▼
Interactive Visualization

💡 Data Format: Events are logged as NDJSON, processed into structured format with IR files and source mappings, then compressed as .ndjson.gz for the web interface.

🔍 Debugging and Development Tools

Debug Logging

# Enable debug logging
export TRITONPARSE_DEBUG=1

# Run with debug output
python your_script.py

Development Utilities

# Check log file contents
head -n 10 ./logs/*.ndjson

# Inspect compressed data
zcat ./parsed_output/*.gz | head -n 20

# Test parsing pipeline
python -c "
import tritonparse.parse.utils
tritonparse.parse.utils.unified_parse('./logs/', './test_output/', verbose=True)
"

Browser Developer Tools

// Enable frontend debug logging
localStorage.setItem('tritonparse-debug', 'true');

// Inspect loaded data
console.log(window.tritonparseData);

// Test data processing
import { processKernelData } from './utils/dataLoader';
console.log(processKernelData(rawData));

🧪 Testing

Test Structure

tests/
├── cpu/                        # CPU-only tests (no GPU required)
├── gpu/                        # GPU tests (require CUDA)
├── ai/                         # AI module tests
├── test_add.py                 # Manual test example
└── example_output/             # Sample data

Running Tests

# CPU-only tests
make test

# All tests including CUDA (requires GPU)
make test-cuda

# Manual test
cd tests
TORCHINDUCTOR_FX_GRAPH_CACHE=0 python test_add.py

Writing Tests

Follow this workflow when adding end-to-end tests in TestTritonparseCUDA:

1. Define test method - Create method starting with test_
2. Define Triton kernel - Write kernel to test feature
3. Setup environment - Create temp directories with tempfile.mkdtemp()
4. Initialize logging - Call tritonparse.structured_logging.init()
5. Run kernel - Execute to generate compilation/launch events
6. Parse logs - Call tritonparse.parse.utils.unified_parse()
7. Assert results - Verify output files and contents
8. Cleanup - Use try...finally to remove temp directories

Example Structure:

@unittest.skipUnless(torch.cuda.is_available(), "CUDA not available")
def test_new_feature(self):
    temp_dir = tempfile.mkdtemp()
    tritonparse.structured_logging.init(temp_dir + "/logs", enable_trace_launch=True)
    try:
        # Run kernel and generate logs
        kernel[(grid,)](args, BLOCK_SIZE=128)

        # Parse and verify
        tritonparse.parse.utils.unified_parse(source=log_path, out=parsed_path)
        self.assertGreater(len(os.listdir(parsed_path)), 0)
    finally:
        shutil.rmtree(temp_dir)
        tritonparse.structured_logging.clear_logging_config()

💡 See existing tests in tests/test_tritonparse.py for complete examples.

📦 Release Process

Version Management

Versions are managed in:

• pyproject.toml - Python package version
• website/package.json - Frontend version

Release Steps

1. Update version numbers
2. Update CHANGELOG.md
3. Run full test suite
4. Build and test website
5. Create GitHub release
6. Deploy to GitHub Pages

GitHub Actions

CI/CD pipeline includes:

• Format checking - Code style validation
• Linting - Code quality checks
• Testing - Python and frontend tests
• Website deployment - Automatic GitHub Pages deployment

🤝 Contributing Guidelines

Pull Request Process

1. Fork the repository
2. Create feature branch: git checkout -b feature-name
3. Make changes following coding standards
4. Add tests for new functionality
5. Run formatting: make format
6. Run tests: make format-check && make test
7. Submit pull request

Code Review Process

• All PRs require review by core maintainers
• CI checks must pass before merge
• Documentation updates required for new features
• Tests required for new functionality

Issue Reporting

When reporting issues:

1. Use issue templates provided
2. Include system information
3. Provide reproduction steps
4. Include error messages and logs

📚 Additional Resources

Documentation

• Code Formatting Guide - Detailed formatting standards
• Python API Reference - Complete API documentation
• Environment Variables Reference - Configuration options
• Reproducer Guide - Reproducer system documentation

Community

• GitHub Discussions - Community Q&A
• GitHub Issues - Bug reports and feature requests

External Resources

• Triton Documentation - Official Triton docs
• React Documentation - React development guide
• TypeScript Documentation - TypeScript reference

🔗 Next Steps

For new developers:

1. Complete the Installation Guide
2. Read the Usage Guide to understand the tool
3. Explore the codebase starting with simple components
4. Run the test suite to verify your setup
5. Join GitHub Discussions for community support

For experienced contributors:

1. Check GitHub Issues for open tasks
2. Review the Architecture Deep Dive for advanced topics
3. Contribute to documentation improvements
4. Propose new features through GitHub Discussions