Skip to content

STORYBOARD_REFACTORING_PLAN.md

Latest commit

 

History

History
1009 lines (814 loc) · 27.5 KB

STORYBOARD_REFACTORING_PLAN.md

File metadata and controls

1009 lines (814 loc) · 27.5 KB

Storyboard Refactoring Plan - TIP Paradigm + World Pattern

Branch: toon-storyboard Date: 2026-01-28 Goal: Refactor storyboard usando il paradigma TIP migliorato (come PDD) + World Pattern

🎯 Obiettivi Principali

1. Architettura TIP Lite (come PDD/Fan Pivot)

  • Separazione in 3 layer: archetipi → molecole → organismi
  • Pure functions per timing, velocity tracking, event processing
  • Domain logic decoupled da React/Zustand

2. World Paradigm (Enter/Exit)

  • StoryboardWorld.enter() / exit() invece di toggle booleano
  • Lifecycle esplicito con cleanup garantito
  • Phase management: idle → recording → settling → exporting

3. Hook System con Element Highlighting

  • Markers clickabili nel transcript che triggerano highlight degli elementi
  • Links bidirezionali: voice → elements, elements → timeline events
  • Sincronizzazione timing tra speech e operazioni GUI

4. Speech-to-Operation Synchronization

  • User parla + clicca operazione GUI → evento linkato
  • Timeline mostra: "utente dice 'ruota' [marker audio] + click rotate button"
  • Migliorie su Whisper integration (o switch ad alternativa)

5. TOON Export Enhancement

  • Clickable element references nel TOON export
  • Ottimizzazione ulteriore per token saving
  • Metadata per replay deterministico

📐 Architettura Target

src/canvas/storyboard/
├── archetipi.ts               # Pure functions (NEW)
│   ├── timing.ts              # Delta time, velocity, interpolation
│   ├── events.ts              # Event processing utils
│   └── markers.ts             # Marker injection, positioning
│
├── molecole.ts                # Domain rules (NEW)
│   ├── StoryboardPhysics      # Settling, momentum tracking
│   ├── EventLinker            # Link voice → elements → operations
│   ├── TimelineCalculator     # Event timing, duration calculation
│   └── MarkerProcessor        # Process voice markers + element clicks
│
├── organismi/                 # Orchestration (NEW)
│   ├── StoryboardWorld.ts     # Main world class (enter/exit/commit)
│   └── types.ts               # Phase types, interfaces
│
├── hooks/                     # React integration (NEW)
│   ├── useStoryboardWorld.ts  # World lifecycle hook
│   └── useElementHighlight.ts # Highlight on marker click
│
└── services/
    ├── toon/
    │   ├── encoder.ts         # TOON format (EXISTING - enhance)
    │   └── decoder.ts         # TOON parser (NEW - for replay)
    └── speech/
        ├── whisper.ts         # Whisper integration (EXISTING - improve)
        └── speechSync.ts      # Speech-to-operation sync (NEW)

🔄 Refactoring Phases

Phase 1: Archetipi Layer (Pure Functions)

File: src/canvas/storyboard/archetipi.ts

// Pure math functions - zero dependencies

// Timing utilities
export const deltaTime = (current: number, previous: number): number => {
  return Math.min((current - previous) / 1000, 0.1); // Cap at 100ms
};

export const lerp = (a: number, b: number, t: number): number => {
  return a + (b - a) * t;
};

export const damp = (
  current: number,
  target: number,
  damping: number,
  dt: number
): number => {
  return lerp(current, target, 1 - Math.pow(damping, dt));
};

// Velocity tracking
export interface Velocity2D {
  x: number;
  y: number;
}

export const calculateVelocity = (
  currentPos: { x: number; y: number },
  previousPos: { x: number; y: number },
  dt: number
): Velocity2D => {
  return {
    x: (currentPos.x - previousPos.x) / dt,
    y: (currentPos.y - previousPos.y) / dt,
  };
};

export const velocityMagnitude = (vel: Velocity2D): number => {
  return Math.hypot(vel.x, vel.y);
};

// Marker injection utilities
export interface VoiceMarker {
  id: string;
  elementId: string;
  elementName: string;
  timestamp: number; // seconds from recording start
  transcriptPosition: number; // character index
  operation?: string; // e.g., 'rotate', 'move', 'resize'
}

export const createMarker = (
  elementId: string,
  elementName: string,
  timestamp: number,
  transcriptLength: number,
  operation?: string
): VoiceMarker => {
  return {
    id: `marker_${Date.now()}_${Math.random().toString(36).slice(2, 9)}`,
    elementId,
    elementName,
    timestamp,
    transcriptPosition: transcriptLength,
    operation,
  };
};

export const injectMarkerText = (
  transcript: string,
  marker: VoiceMarker
): string => {
  const markerText = marker.operation
    ? ` @${marker.elementName}:${marker.operation} `
    : ` @${marker.elementName} `;
  return transcript + markerText;
};

Phase 2: Molecole Layer (Domain Rules)

File: src/canvas/storyboard/molecole.ts

import {
  VoiceMarker,
  Velocity2D,
  calculateVelocity,
  velocityMagnitude,
  damp,
} from './archetipi';
import { CanvasElement, StoryboardEvent } from '../../shared/types';

// Physics for drag settling
export interface StoryboardPhysics {
  velocity: Velocity2D;
  damping: number; // 0.92 = 8% energy loss per frame
}

export const dampPhysics = (physics: StoryboardPhysics): StoryboardPhysics => {
  return {
    ...physics,
    velocity: {
      x: physics.velocity.x * physics.damping,
      y: physics.velocity.y * physics.damping,
    },
  };
};

export const isSettled = (
  physics: StoryboardPhysics,
  threshold = 0.5 // pixels/second
): boolean => {
  return velocityMagnitude(physics.velocity) < threshold;
};

// Event linking - associates voice markers with operations
export interface LinkedEvent {
  event: StoryboardEvent;
  voiceMarkers: VoiceMarker[]; // Markers within event timeframe
}

export const linkVoiceToEvent = (
  event: StoryboardEvent,
  markers: VoiceMarker[],
  timeWindow = 2000 // 2 seconds before/after
): LinkedEvent => {
  const linkedMarkers = markers.filter(marker => {
    const markerTime = marker.timestamp * 1000; // Convert to ms
    const eventTime = event.timestamp;
    return Math.abs(markerTime - eventTime) <= timeWindow;
  });

  return {
    event,
    voiceMarkers: linkedMarkers,
  };
};

// Timeline calculation
export interface TimelineSnapshot {
  timestamp: number;
  elementStates: Map<string, CanvasElement>;
}

export const calculateEventDuration = (
  start: TimelineSnapshot,
  end: TimelineSnapshot
): number => {
  return end.timestamp - start.timestamp;
};

// Marker processing - sync voice with GUI operations
export interface OperationMarker extends VoiceMarker {
  operation: string; // 'rotate' | 'move' | 'resize' | 'scale' | 'delete' | etc.
  operationTimestamp: number; // When GUI button was clicked
}

export const syncVoiceWithOperation = (
  voiceMarker: VoiceMarker,
  operationType: string,
  operationTimestamp: number
): OperationMarker => {
  return {
    ...voiceMarker,
    operation: operationType,
    operationTimestamp,
  };
};

Phase 3: Organismi Layer (World Orchestration)

File: src/canvas/storyboard/organismi/StoryboardWorld.ts

import { v4 as uuidv4 } from 'uuid';
import {
  Storyboard,
  StoryboardEvent,
  CanvasElement,
} from '../../../shared/types';
import {
  VoiceMarker,
  createMarker,
  injectMarkerText,
  deltaTime,
  Velocity2D,
  calculateVelocity,
} from '../archetipi';
import {
  StoryboardPhysics,
  dampPhysics,
  isSettled,
  linkedEvent,
  syncVoiceWithOperation,
  OperationMarker,
} from '../molecole';

// Phases for storyboard lifecycle
export type StoryboardPhase =
  | 'idle'        // No storyboard active
  | 'ready'       // Storyboard created, not recording
  | 'recording'   // Actively recording events
  | 'settling'    // Drag momentum decay
  | 'reviewing'   // Playback/review mode
  | 'exporting';  // Export in progress

// Callbacks for world events
export interface StoryboardCallbacks {
  onPhaseChange?: (phase: StoryboardPhase) => void;
  onEventLogged?: (event: StoryboardEvent) => void;
  onMarkerAdded?: (marker: VoiceMarker) => void;
  onHighlightElement?: (elementId: string) => void;
  onUpdate?: () => void;
}

// World listener pattern (like PDD)
type WorldEvent =
  | { type: 'phase_change'; phase: StoryboardPhase }
  | { type: 'event_logged'; event: StoryboardEvent }
  | { type: 'marker_added'; marker: VoiceMarker }
  | { type: 'highlight_element'; elementId: string }
  | { type: 'commit'; storyboard: Storyboard }
  | { type: 'update' };

type WorldListener = (event: WorldEvent) => void;

export class StoryboardWorld {
  private phase: StoryboardPhase = 'idle';
  private storyboard: Storyboard | null = null;
  private callbacks: StoryboardCallbacks;
  private listeners: Set<WorldListener> = new Set();

  // Voice recording state
  private voiceMarkers: VoiceMarker[] = [];
  private operationMarkers: OperationMarker[] = [];
  private voiceStartTime: number = 0;
  private lastSelectedElementId: string | null = null;
  private currentTranscript: string = '';

  // Drag tracking with velocity
  private dragTrackers: Map<string, {
    startTime: number;
    startPos: { x: number; y: number };
    currentPos: { x: number; y: number };
    velocity: Velocity2D;
  }> = new Map();

  // Physics for settling phase
  private physics: StoryboardPhysics = {
    velocity: { x: 0, y: 0 },
    damping: 0.92,
  };

  // Animation loop
  private animationFrameId: number | null = null;
  private lastTime: number = 0;

  constructor(callbacks: StoryboardCallbacks = {}) {
    this.callbacks = callbacks;
  }

  // === LIFECYCLE ===

  enter(projectId: string, name?: string): void {
    if (this.phase !== 'idle') {
      console.warn('[StoryboardWorld] Already in world, exiting first');
      this.exit();
    }

    this.storyboard = {
      id: uuidv4(),
      projectId,
      name: name || `Storyboard ${new Date().toLocaleDateString()}`,
      createdAt: new Date().toISOString(),
      events: [],
      isRecording: false,
    };

    this.setPhase('ready');
    console.log('[StoryboardWorld] Entered world:', this.storyboard.name);
  }

  exit(): void {
    this.stopRecording();
    this.stopAnimation();
    this.reset();
    this.setPhase('idle');
    console.log('[StoryboardWorld] Exited world (cancelled)');
  }

  commit(): Storyboard | null {
    if (!this.storyboard) {
      console.warn('[StoryboardWorld] No storyboard to commit');
      return null;
    }

    const finalStoryboard = { ...this.storyboard };
    this.emit({ type: 'commit', storyboard: finalStoryboard });
    this.exit();
    console.log('[StoryboardWorld] Committed storyboard');
    return finalStoryboard;
  }

  // === RECORDING ===

  startRecording(): void {
    if (this.phase !== 'ready') {
      console.warn('[StoryboardWorld] Cannot start recording from phase:', this.phase);
      return;
    }

    if (!this.storyboard) {
      console.error('[StoryboardWorld] No storyboard exists');
      return;
    }

    this.storyboard.isRecording = true;
    this.setPhase('recording');
    console.log('[StoryboardWorld] Recording started');
  }

  stopRecording(): void {
    if (this.phase !== 'recording' && this.phase !== 'settling') return;

    if (this.storyboard) {
      this.storyboard.isRecording = false;
    }

    this.setPhase('ready');
    console.log('[StoryboardWorld] Recording stopped');
  }

  // === VOICE RECORDING WITH MARKERS ===

  startVoiceRecording(): void {
    if (this.phase !== 'recording') {
      console.warn('[StoryboardWorld] Must be recording to start voice');
      return;
    }

    this.voiceMarkers = [];
    this.operationMarkers = [];
    this.voiceStartTime = Date.now();
    this.lastSelectedElementId = null;
    this.currentTranscript = '';

    console.log('[StoryboardWorld] Voice recording started - click elements to inject markers');
  }

  // Inject marker when user clicks element during voice recording
  injectMarker(elementId: string, elementName: string, operation?: string): void {
    // Prevent duplicate markers for same element
    if (elementId === this.lastSelectedElementId) return;
    this.lastSelectedElementId = elementId;

    const timestamp = (Date.now() - this.voiceStartTime) / 1000;
    const marker = createMarker(
      elementId,
      elementName,
      timestamp,
      this.currentTranscript.length,
      operation
    );

    this.voiceMarkers.push(marker);

    // Update transcript with marker
    this.currentTranscript = injectMarkerText(this.currentTranscript, marker);

    this.emit({ type: 'marker_added', marker });
    console.log(`[StoryboardWorld] Marker injected: @${elementName}${operation ? ':' + operation : ''}`);
  }

  // Sync voice marker with GUI operation (e.g., user clicks rotate button)
  syncMarkerWithOperation(operation: string): void {
    if (this.voiceMarkers.length === 0) return;

    const lastMarker = this.voiceMarkers[this.voiceMarkers.length - 1];
    const operationMarker = syncVoiceWithOperation(
      lastMarker,
      operation,
      Date.now()
    );

    this.operationMarkers.push(operationMarker);

    console.log(`[StoryboardWorld] Marker synced with operation: ${operation}`);
  }

  updateTranscript(transcript: string): void {
    this.currentTranscript = transcript;
  }

  stopVoiceRecording(finalTranscript: string): void {
    this.currentTranscript = finalTranscript;
    this.lastSelectedElementId = null;

    // Log voice event with all markers
    if (finalTranscript.trim() && this.storyboard) {
      const allElementIds = this.voiceMarkers.map(m => m.elementId);
      const allElementNames = this.voiceMarkers.map(m => m.elementName);

      const event: StoryboardEvent = {
        id: uuidv4(),
        timestamp: Date.now(),
        type: 'voice_note',
        elementIds: allElementIds,
        elementNames: allElementNames,
        comment: finalTranscript,
        isVoice: true,
        description: `Voice: "${finalTranscript.slice(0, 50)}..."`,
        metadata: {
          markers: this.voiceMarkers,
          operationMarkers: this.operationMarkers,
        },
      };

      this.logEvent(event);
    }

    console.log('[StoryboardWorld] Voice recording stopped');
  }

  // === DRAG TRACKING WITH VELOCITY ===

  startDragTracking(elements: CanvasElement[]): void {
    if (this.phase !== 'recording') return;

    const now = performance.now();
    this.dragTrackers.clear();

    elements.forEach(el => {
      this.dragTrackers.set(el.id, {
        startTime: now,
        startPos: { x: el.x, y: el.y },
        currentPos: { x: el.x, y: el.y },
        velocity: { x: 0, y: 0 },
      });
    });
  }

  updateDrag(elements: CanvasElement[]): void {
    if (this.phase !== 'recording') return;

    const now = performance.now();

    elements.forEach(el => {
      const tracker = this.dragTrackers.get(el.id);
      if (!tracker) return;

      const dt = deltaTime(now, tracker.startTime);
      const velocity = calculateVelocity(
        { x: el.x, y: el.y },
        tracker.currentPos,
        dt
      );

      tracker.velocity = velocity;
      tracker.currentPos = { x: el.x, y: el.y };
    });
  }

  endDragTracking(elements: CanvasElement[]): void {
    if (this.phase !== 'recording') return;

    // Check if we need settling phase
    let maxVelocity = 0;
    this.dragTrackers.forEach(tracker => {
      const velMag = velocityMagnitude(tracker.velocity);
      if (velMag > maxVelocity) {
        maxVelocity = velMag;
        this.physics.velocity = tracker.velocity;
      }
    });

    // Log movement event
    // ... (implement event logging)

    // Enter settling if velocity is high
    if (maxVelocity > 50) { // pixels/second threshold
      this.setPhase('settling');
      this.startAnimation();
    }

    this.dragTrackers.clear();
  }

  // === EVENT LOGGING ===

  private logEvent(event: StoryboardEvent): void {
    if (!this.storyboard) return;

    this.storyboard.events.push(event);
    this.emit({ type: 'event_logged', event });
    this.callbacks.onEventLogged?.(event);
  }

  // === ELEMENT HIGHLIGHTING (for clickable markers) ===

  highlightElement(elementId: string): void {
    this.emit({ type: 'highlight_element', elementId });
    this.callbacks.onHighlightElement?.(elementId);
  }

  // === ANIMATION LOOP (for settling phase) ===

  private startAnimation(): void {
    if (this.animationFrameId !== null) return;
    this.lastTime = performance.now();
    this.tick();
  }

  private stopAnimation(): void {
    if (this.animationFrameId !== null) {
      cancelAnimationFrame(this.animationFrameId);
      this.animationFrameId = null;
    }
  }

  private tick = (): void => {
    const now = performance.now();
    const dt = deltaTime(now, this.lastTime);

    if (this.phase === 'settling') {
      this.physics = dampPhysics(this.physics);

      // Check if settled
      if (isSettled(this.physics)) {
        this.setPhase('recording');
        this.stopAnimation();
      }

      this.emit({ type: 'update' });
    }

    this.lastTime = now;
    this.animationFrameId = requestAnimationFrame(this.tick);
  };

  // === PHASE MANAGEMENT ===

  private setPhase(phase: StoryboardPhase): void {
    if (this.phase === phase) return;

    this.phase = phase;
    this.emit({ type: 'phase_change', phase });
    this.callbacks.onPhaseChange?.(phase);
  }

  getPhase(): StoryboardPhase {
    return this.phase;
  }

  // === EVENT LISTENER PATTERN ===

  listen(callback: WorldListener): () => void {
    this.listeners.add(callback);
    return () => this.listeners.delete(callback);
  }

  private emit(event: WorldEvent): void {
    this.listeners.forEach(listener => listener(event));
    this.callbacks.onUpdate?.();
  }

  // === CLEANUP ===

  private reset(): void {
    this.storyboard = null;
    this.voiceMarkers = [];
    this.operationMarkers = [];
    this.dragTrackers.clear();
    this.voiceStartTime = 0;
    this.lastSelectedElementId = null;
    this.currentTranscript = '';
    this.physics = { velocity: { x: 0, y: 0 }, damping: 0.92 };
  }

  // === GETTERS ===

  getStoryboard(): Storyboard | null {
    return this.storyboard;
  }

  getVoiceMarkers(): VoiceMarker[] {
    return [...this.voiceMarkers];
  }

  getOperationMarkers(): OperationMarker[] {
    return [...this.operationMarkers];
  }
}

Phase 4: React Hook Integration

File: src/canvas/storyboard/hooks/useStoryboardWorld.ts

import { useEffect, useRef, useState, useCallback } from 'react';
import { StoryboardWorld, StoryboardPhase, StoryboardCallbacks } from '../organismi/StoryboardWorld';
import { Storyboard, StoryboardEvent } from '../../../shared/types';
import { VoiceMarker } from '../archetipi';

export const useStoryboardWorld = (callbacks?: StoryboardCallbacks) => {
  const worldRef = useRef<StoryboardWorld | null>(null);
  const [phase, setPhase] = useState<StoryboardPhase>('idle');
  const [storyboard, setStoryboard] = useState<Storyboard | null>(null);
  const [events, setEvents] = useState<StoryboardEvent[]>([]);

  // Initialize world
  useEffect(() => {
    worldRef.current = new StoryboardWorld({
      ...callbacks,
      onPhaseChange: (newPhase) => {
        setPhase(newPhase);
        callbacks?.onPhaseChange?.(newPhase);
      },
      onEventLogged: (event) => {
        setEvents(prev => [...prev, event]);
        callbacks?.onEventLogged?.(event);
      },
      onUpdate: () => {
        if (worldRef.current) {
          setStoryboard(worldRef.current.getStoryboard());
        }
        callbacks?.onUpdate?.();
      },
    });

    return () => {
      if (worldRef.current) {
        worldRef.current.exit();
      }
    };
  }, []);

  // World actions
  const enter = useCallback((projectId: string, name?: string) => {
    worldRef.current?.enter(projectId, name);
  }, []);

  const exit = useCallback(() => {
    worldRef.current?.exit();
  }, []);

  const commit = useCallback(() => {
    return worldRef.current?.commit() || null;
  }, []);

  const startRecording = useCallback(() => {
    worldRef.current?.startRecording();
  }, []);

  const stopRecording = useCallback(() => {
    worldRef.current?.stopRecording();
  }, []);

  const startVoiceRecording = useCallback(() => {
    worldRef.current?.startVoiceRecording();
  }, []);

  const stopVoiceRecording = useCallback((transcript: string) => {
    worldRef.current?.stopVoiceRecording(transcript);
  }, []);

  const injectMarker = useCallback((elementId: string, elementName: string, operation?: string) => {
    worldRef.current?.injectMarker(elementId, elementName, operation);
  }, []);

  const syncMarkerWithOperation = useCallback((operation: string) => {
    worldRef.current?.syncMarkerWithOperation(operation);
  }, []);

  const highlightElement = useCallback((elementId: string) => {
    worldRef.current?.highlightElement(elementId);
  }, []);

  return {
    world: worldRef.current,
    phase,
    storyboard,
    events,
    // Actions
    enter,
    exit,
    commit,
    startRecording,
    stopRecording,
    startVoiceRecording,
    stopVoiceRecording,
    injectMarker,
    syncMarkerWithOperation,
    highlightElement,
  };
};

Phase 5: Element Highlighting Hook

File: src/canvas/storyboard/hooks/useElementHighlight.ts

import { useEffect, useState } from 'react';
import { useStoryboardWorld } from './useStoryboardWorld';

export const useElementHighlight = () => {
  const [highlightedElementId, setHighlightedElementId] = useState<string | null>(null);
  const { world } = useStoryboardWorld();

  useEffect(() => {
    if (!world) return;

    const unsubscribe = world.listen((event) => {
      if (event.type === 'highlight_element') {
        setHighlightedElementId(event.elementId);

        // Auto-clear after 2 seconds
        setTimeout(() => {
          setHighlightedElementId(null);
        }, 2000);
      }
    });

    return unsubscribe;
  }, [world]);

  return highlightedElementId;
};

Phase 6: Enhanced TOON Export with Clickable Markers

File: src/canvas/services/toon/encoder.ts (enhance existing)

// Add after existing toTOON function

/**
 * Enhanced TOON export with clickable element markers
 */
export function toTOONWithMarkers(data: StoryboardExport): string {
  const lines: string[] = [];

  // ... (existing meta, elements sections)

  // Voice notes with clickable markers
  const voiceNotes = (data.timeline?.events || []).filter(
    ev => ev.type === 'voice_note' && ev.comment
  );

  if (voiceNotes.length > 0) {
    lines.push(`voiceNotes[${voiceNotes.length}]{id,time,transcript,markers}:`);
    voiceNotes.forEach(vn => {
      const relativeTime = ((vn.timestamp - (events[0]?.timestamp || 0)) / 1000).toFixed(1);

      // Extract markers from metadata
      const markers = vn.metadata?.markers || [];
      const markerRefs = markers
        .map((m: VoiceMarker) => {
          const op = m.operation ? `:${m.operation}` : '';
          return `@${m.elementName.replace(/\s+/g, '_')}${op}[${m.timestamp.toFixed(1)}s]`;
        })
        .join(' ');

      const row = [
        vn.id.slice(0, 8),
        relativeTime,
        escapeValue(vn.comment || ''),
        markerRefs,
      ].join(',');

      lines.push(`  ${row}`);
    });
    lines.push('');
  }

  // Marker index (for clickable references)
  const allMarkers = voiceNotes.flatMap(vn => vn.metadata?.markers || []);
  if (allMarkers.length > 0) {
    lines.push(`markerIndex[${allMarkers.length}]{markerId,elementId,elementName,timestamp,operation}:`);
    allMarkers.forEach((marker: VoiceMarker) => {
      const row = [
        marker.id.slice(0, 8),
        marker.elementId.slice(0, 8),
        escapeValue(marker.elementName),
        marker.timestamp.toFixed(2),
        marker.operation || '',
      ].join(',');
      lines.push(`  ${row}`);
    });
  }

  return lines.join('\n');
}

🎯 Implementation Tasks (TodoList)

Immediate Tasks

  1. ✅ Analyze current storyboard architecture
  2. ⏳ Create archetipi.ts with pure functions
  3. ⏳ Create molecole.ts with domain logic
  4. ⏳ Implement StoryboardWorld class
  5. ⏳ Create React hooks (useStoryboardWorld, useElementHighlight)
  6. ⏳ Enhance TOON encoder with markers
  7. ⏳ Refactor StoryboardPanel to use world pattern
  8. ⏳ Add speech-to-operation sync
  9. ⏳ Add clickable markers in UI
  10. ⏳ Test integration

🚨 Critical Issues to Fix

1. Speech Timing Sync

Problem: Markers injection timing non sincronizzato con speech recognition Solution: Use performance.now() per timing preciso, track markers con timestamps relativi

2. Whisper Model Issues

Problem: "Whisper fa un po' cagare" Options:

  • Switch to better speech recognition (Google Cloud Speech-to-Text, Assembly AI)
  • Improve Whisper integration con better prompts/temperature settings
  • Add post-processing per migliorare transcript quality

3. Inline Hooks

Problem: Hooks non hanno links che evidenziano elementi Solution:

  • Clickable markers in StoryboardPanel
  • onMarkerClick → world.highlightElement(elementId)
  • Visual feedback su canvas (border highlight, glow effect)

4. Operation Linking

Problem: User parla + clicca operazione, ma link non viene tracciato Solution:

  • syncMarkerWithOperation(operation) chiamato quando user clicca toolbar button
  • Store operation timestamp in OperationMarker
  • Link voice transcript con GUI action

📊 Success Metrics

  • World lifecycle completamente funzionante (enter/exit/commit)
  • Phase transitions smooth (idle → recording → settling → exporting)
  • Voice markers clickabili che evidenziano elementi
  • Speech-to-operation sync con max 200ms latency
  • TOON export con token saving >40%
  • Marker links funzionanti in entrambe le direzioni
  • Zero memory leaks (cleanup su exit)
  • Unit tests per archetipi e molecole layers

🔧 Testing Strategy

Unit Tests (Vitest)

// Test archetipi (pure functions)
describe('archetipi', () => {
  it('should calculate velocity correctly', () => {
    const vel = calculateVelocity({ x: 100, y: 50 }, { x: 90, y: 40 }, 0.1);
    expect(vel.x).toBe(100); // (100-90)/0.1
    expect(vel.y).toBe(100);
  });

  it('should inject marker text correctly', () => {
    const marker = createMarker('el1', 'Button', 1.5, 100, 'rotate');
    const transcript = injectMarkerText('Hello ', marker);
    expect(transcript).toBe('Hello  @Button:rotate ');
  });
});

// Test molecole (domain logic)
describe('molecole', () => {
  it('should detect settled physics', () => {
    const physics = { velocity: { x: 0.3, y: 0.2 }, damping: 0.92 };
    expect(isSettled(physics, 0.5)).toBe(true);
    expect(isSettled(physics, 0.1)).toBe(false);
  });
});

Integration Tests

  • Test world lifecycle (enter → record → commit)
  • Test voice recording con marker injection
  • Test drag tracking con velocity calculation
  • Test phase transitions

📅 Timeline Estimate

Total: 3-5 giorni di sviluppo (assumendo 6-8 ore/giorno)

  • Day 1: Archetipi + Molecole layers (pure functions + domain logic)
  • Day 2: StoryboardWorld class + lifecycle management
  • Day 3: React hooks + StoryboardPanel refactoring
  • Day 4: Speech sync + clickable markers + TOON enhancement
  • Day 5: Testing, debugging, integration

🤝 Questions for Clarification

Prima di iniziare l'implementazione, conferma:

  1. Speech Recognition: Vuoi sostituire Whisper con un altro provider, o migliorare l'integrazione esistente?
  2. Marker UI: Come vuoi visualizzare i markers clickabili? (badges, highlights, tooltip?)
  3. Operation Types: Quali operazioni GUI vuoi tracciare? (rotate, move, resize, scale, delete, ...?)
  4. TOON Parsing: Serve anche un decoder TOON per replay? (per ricostruire storyboard da file TOON)
  5. Backward Compatibility: Dobbiamo supportare vecchi storyboard JSON o possiamo fare breaking change?

Ready to start implementation? 🚀