Custom Annotation Queue, Feedback, and Trace Viewer

A custom platform for reviewing and scoring slide decks generated by AI agents built with LangSmith observability and feedback as a backend. Built on LangSmith, this tool enables structured feedback collection with automatic trace attachment for continuous improvement of presentation generation workflows on a custom frontend UI.

Overview

This application demos LangSmith trace monitoring and human evaluation of generated artifacts on a custom frontend. It automatically fetches traces containing PowerPoint outputs, converts them to viewable formats, and provides a structured interface for collecting quantitative scores (1-5) and qualitative feedback that's directly attached to the source traces in LangSmith.

Key Capabilities

Automated Trace Ingestion: Pulls recent traces from LangSmith projects that contain PPTX outputs
Dual-Level Feedback: Collects both trace-level (overall PPTX quality) and slide-level (individual data communication) annotations
Quantitative Scoring: 1-5 rating system for systematic evaluation and analytics
LangSmith Integration: Feedback automatically attached to traces via LangSmith API for downstream analysis
Trace Visualization: Deep inspection of agent runs, including inputs/outputs, timing, and execution flow
Agent Playground: Chat view of the entire trace to-date with preserved thread to continue the conversation with the agent
Artifact Management: PPTX-to-PDF conversion for reliable cross-platform viewing and downloadable artifacts

The Agent: Financial Slide Deck Generator

This project includes financial_slide_agent.py, a LangGraph-based AI agent that generates polished financial PowerPoint presentations from structured data. The agent uses GPT-4o-mini with granular tool control to create professional slides:

Tools Available to the Agent:

create_presentation: Initialize a new PPTX with 16:9 aspect ratio
add_slide: Insert blank slides with custom layouts
add_title, add_text: Add formatted text with positioning and styling
add_chart: Create bar, line, or pie charts from data
add_table: Insert formatted tables with headers and styling
finalize_presentation: Export the completed PPTX as base64 (stored in LangSmith trace)

Example Use Cases: The agent generates three types of financial decks:

Q4 2024 Business Review: Revenue, profit, and growth metrics with trend comparisons
SaaS Investor Update: MRR, ARR, unit economics, and churn analysis
E-commerce Category Review: Top-performing categories with YoY growth

The agent's traces (including the generated PPTX artifacts) are automatically ingested by this annotation platform for human review and scoring.

Prerequisites

LibreOffice Required for PPTX to PDF conversion:

macOS: brew install --cask libreoffice
Ubuntu/Debian: sudo apt-get install libreoffice
Windows: Download from libreoffice.org

Setup

1. Environment Variables

Create backend/.env:

LANGSMITH_API_KEY=your-api-key
LANGSMITH_PROJECT=your-project-name
LANGSMITH_ORG=your-org-id
LANGSMITH_PROJECT_ID=your-project-uuid


- **ORG_ID**: Copy the UUID after `/o/`
- **PROJECT_ID**: Copy the UUID after `/projects/p/`

### 2. Start Backend

```bash
cd backend
pip install -r requirements.txt
python -m uvicorn app:app --reload

Backend runs on http://localhost:8000

3. Start Frontend

cd frontend
npm install
npm run dev

Frontend runs on http://localhost:5173

Usage Workflow

1. Trace Ingestion

The backend automatically fetches recent traces from your LangSmith project that contain PowerPoint outputs (specifically, traces with finalize_presentation tool calls). Traces are listed with status indicators and metadata.

2. Trace Inspection

Select a trace to view:

Full execution tree: All runs (LLM calls, tool executions, etc.)
Input/Output data: Syntax-highlighted JSON for each step
Timing information: Duration and status of each run
Direct LangSmith link: One-click access to full trace details

3. Slide Review

Navigate through the converted PDF slides to review the generated presentation. The current slide number is tracked for contextual feedback.

4. Feedback Annotation

Trace-Level Feedback (Overall PPTX Quality)

Score (1-5): Rate the entire presentation's layout, design, and structure
Notes: Provide qualitative feedback on overall quality
Submitted as key: pptx_layout_quality in LangSmith

Slide-Level Feedback (Individual Slide Quality)

Score (1-5): Rate how effectively the slide communicates its data
Notes: Comment on visualization clarity, data accuracy, design
Submitted as key: slide_{N}_communication in LangSmith

5. LangSmith Integration

All feedback is automatically attached to the source trace via the LangSmith API:

Scores normalized to 0-1 for LangSmith analytics
Comments stored as feedback text
Feedback appears in the "Feedback" tab of the trace
Enables downstream analysis, score distributions, and trend tracking

Agent Playground

Click "Open in Chat" on any trace to enter an interactive playground where you can regenerate slides with different requirements:

Conversational Interface: Chat with the agent to modify presentations
Data Preservation: Uses the original data from the trace
Live Generation: Creates new slides based on your instructions
Instant Downloads: Get PPTX and PDF links immediately after generation

Example prompts:

"Create a 2-slide version instead of 3"
"Make the metrics cards larger and use green colors"
"Show only revenue metrics on the first slide"

The playground provides a custom agent experience tailored to your workflow, separate from the standard LangSmith prompt playground.

API Endpoints

GET /api/traces

Fetches recent traces from LangSmith that contain PowerPoint outputs.

Response:

{
  "traces": [
    {
      "trace_id": "uuid",
      "trace_name": "string",
      "created_at": "ISO8601 timestamp",
      "pptx_base64": "base64 encoded PPTX",
      "has_pdf": true,
      "langsmith_url": "https://smith.langchain.com/o/{org}/projects/p/{project}?peek={trace_id}",
      "runs": [
        {
          "run_id": "uuid",
          "name": "string",
          "run_type": "llm | tool | chain",
          "status": "success | error",
          "inputs_summary": {...},
          "outputs_summary": {...},
          "duration_ms": 1234
        }
      ]
    }
  ],
  "project_name": "Your LangSmith Project"
}

GET /api/traces/{trace_id}/slides.pdf

Returns the PDF-converted version of the PowerPoint presentation for the specified trace.

Response: application/pdf

POST /api/feedback

Submits scored feedback and automatically attaches it to the LangSmith trace.

Request Body:

{
  "trace_id": "uuid",
  "feedback_type": "trace" | "slide",
  "score": 1-5,
  "content": "Free-form annotation text",
  "slide_number": 1  // Required for slide feedback, optional for trace
}

Validation:

score: Must be integer between 1-5
feedback_type: Either "trace" or "slide"
slide_number: Required when feedback_type is "slide"

LangSmith Integration:

Trace feedback → Key: pptx_layout_quality
Slide feedback → Key: slide_{N}_communication
Score normalized to 0-1 for LangSmith (e.g., 4/5 → 0.8)
Comment includes score and user notes

Response:

{
  "success": true,
  "message": "Feedback submitted and attached to trace"
}

GET /api/feedback/{trace_id}

Retrieves all feedback submitted for a specific trace (locally stored).

Response:

[
  {
    "trace_id": "uuid",
    "feedback_type": "trace",
    "score": 4,
    "content": "Good layout but spacing could be improved",
    "timestamp": "ISO8601 timestamp"
  }
]

POST /api/chat

Interact with the agent to regenerate slides based on trace data.

Request Body:

{
  "trace_id": "uuid",
  "message": "Create a 2-slide version",
  "history": []  // Previous chat messages
}

Response:

{
  "response": "✅ Slides Generated Successfully!\n\n📥 Download Links:\n- [Download PPTX](/api/chat/download/...)",
  "trace_id": "uuid"
}

GET /api/chat/download/{cache_key}

Download a chat-generated PPTX or PDF file.

Response: application/vnd.openxmlformats-officedocument.presentationml.presentation or application/pdf

GET /api/health

Health check endpoint for monitoring.

Response:

{
  "status": "healthy"
}

Architecture

System Overview

┌─────────────────────────────────────────────────────────────────┐
│                         LangSmith Cloud                         │
│  • Project: Stores traces from AI agent runs                    │
│  • Traces: Contains run metadata, I/O, timing, artifacts        │
│  • Feedback API: Receives annotations and scores                │
└────────────────────────────┬────────────────────────────────────┘
                             │
                             ▼
┌─────────────────────────────────────────────────────────────────┐
│                      FastAPI Backend                            │
│                                                                  │
│  LangSmith Client                                               │
│  ├─ list_runs() → Fetch traces with PPTX outputs                │
│  ├─ get_run_tree() → Retrieve full trace execution tree         │
│  └─ create_feedback() → Attach annotations to traces            │
│                                                                  │
│  PPTX Processing                                                │
│  ├─ Extract base64 PPTX from trace outputs                      │
│  ├─ Convert to PDF via LibreOffice subprocess                   │
│  └─ Serve PDF and downloadable PPTX                             │
│                                                                  │
│  Feedback Management                                            │
│  ├─ Validate score (1-5) and metadata                           │
│  ├─ Normalize score to 0-1 for LangSmith                        │
│  └─ Store locally + sync to LangSmith                           │
│                                                                  │
│  Agent Playground                                               │
│  ├─ Extract original data from trace                            │
│  ├─ Invoke agent with user's chat instructions                  │
│  ├─ Cache generated PPTX and converted PDF                      │
│  └─ Serve download links                                        │
└────────────────────────────┬────────────────────────────────────┘
                             │
                             ▼
┌─────────────────────────────────────────────────────────────────┐
│                       React Frontend                            │
│                                                                  │
│  Trace Browser                                                  │
│  ├─ List recent traces from project                             │
│  ├─ Display trace metadata (ID, name, timestamp)                │
│  └─ Link directly to LangSmith for deep inspection              │
│                                                                  │
│  Execution Viewer                                               │
│  ├─ Expandable trace steps (all runs in tree)                   │
│  ├─ Syntax-highlighted JSON for inputs/outputs                  │
│  └─ Run timing, status, and error details                       │
│                                                                  │
│  Slide Viewer                                                   │
│  ├─ PDF rendering via react-pdf                                 │
│  ├─ Keyboard navigation (arrow keys)                            │
│  └─ Current slide tracking for feedback context                 │
│                                                                  │
│  Annotation Interface                                           │
│  ├─ Trace-level: PPTX layout quality (1-5)                      │
│  ├─ Slide-level: Data communication effectiveness (1-5)         │
│  ├─ Free-form notes for qualitative feedback                    │
│  └─ One-click submission to LangSmith                           │
│                                                                  │
│  Agent Playground (Chat)                                        │
│  ├─ Conversational UI for slide regeneration                    │
│  ├─ Preserves original trace data                               │
│  ├─ Real-time agent interaction                                 │
│  └─ Download links for generated PPTX/PDF                       │
└─────────────────────────────────────────────────────────────────┘

Data Flow

1. Agent Run → LangSmith
   └─ AI agent generates PPTX → Stored as base64 in trace output

2. Backend Ingestion
   └─ Poll LangSmith → Filter traces with PPTX → Convert to PDF

3. Frontend Display
   └─ Load traces → Display PDF → Navigate slides

4. Feedback Collection
   └─ User scores/annotates → Submit to backend → Attach to LangSmith

5. Analytics
   └─ LangSmith aggregates feedback → Score distributions → Trend analysis

Technology Stack

Backend

FastAPI: REST API server
LangSmith Client: Trace ingestion and feedback submission
Pydantic: Request/response validation
LibreOffice: Headless PPTX-to-PDF conversion

Frontend

React + Vite: UI framework and build tool
Chakra UI v3: Component library and design system
react-pdf: PDF rendering with navigation
react-syntax-highlighter: JSON formatting for trace I/O

Required Dependencies:

Python 3.9+, FastAPI, LangSmith Client, Pydantic, LangChain, pandas
Node.js 18+, React, Vite, Chakra UI v3, react-pdf
LibreOffice (system-level dependency for PDF conversion)

Key Design Decisions:

PPTX stored as base64 in LangSmith trace outputs
PDF conversion for cross-platform compatibility
Local feedback storage + LangSmith sync for reliability
Feedback scores normalized to 0-1 for LangSmith analytics
Separate trace-level and slide-level annotation workflows

Name		Name	Last commit message	Last commit date
Latest commit History 18 Commits
backend		backend
frontend		frontend
.env.example		.env.example
.gitignore		.gitignore
README.md		README.md
financial_slide_agent.py		financial_slide_agent.py
package-lock.json		package-lock.json
pyproject.toml		pyproject.toml
uv.lock		uv.lock

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Repository files navigation

Custom Annotation Queue, Feedback, and Trace Viewer

Overview

Key Capabilities

The Agent: Financial Slide Deck Generator

Prerequisites

Setup

1. Environment Variables

3. Start Frontend

Usage Workflow

1. Trace Ingestion

2. Trace Inspection

3. Slide Review

4. Feedback Annotation

5. LangSmith Integration

Agent Playground

API Endpoints

GET /api/traces

GET /api/traces/{trace_id}/slides.pdf

POST /api/feedback

GET /api/feedback/{trace_id}

POST /api/chat

GET /api/chat/download/{cache_key}

GET /api/health

Architecture

System Overview

Data Flow

Technology Stack

About

Uh oh!

Releases

Packages

Uh oh!

Contributors

Uh oh!

Languages

Folders and files

Latest commit

History

Repository files navigation

Custom Annotation Queue, Feedback, and Trace Viewer

Overview

Key Capabilities

The Agent: Financial Slide Deck Generator

Prerequisites

Setup

1. Environment Variables

3. Start Frontend

Usage Workflow

1. Trace Ingestion

2. Trace Inspection

3. Slide Review

4. Feedback Annotation

5. LangSmith Integration

Agent Playground

API Endpoints

GET /api/traces

GET /api/traces/{trace_id}/slides.pdf

POST /api/feedback

GET /api/feedback/{trace_id}

POST /api/chat

GET /api/chat/download/{cache_key}

GET /api/health

Architecture

System Overview

Data Flow

Technology Stack

About

Topics

Resources

Contributing

Uh oh!

Stars

Watchers

Forks

Releases

Packages 0

Uh oh!

Contributors

Uh oh!

Languages

Packages