Skip to content

TESTING.md

Latest commit

 

History

History
724 lines (541 loc) · 17.5 KB

TESTING.md

File metadata and controls

724 lines (541 loc) · 17.5 KB

Full Library Sync - Testing Guide

This document outlines comprehensive testing procedures for the Full Library Sync feature.

Overview

The Full Library Sync feature automatically detects if your entire music library fits on your iPod and offers to sync everything, eliminating the need for manual playlist selection.

Key Behaviors:

  • Library < 90% of storage → User chooses Full Library or Playlist mode
  • Library > 90% of storage → Automatically uses Playlist mode
  • Choice saved on-device (survives computer changes)
  • Growth detection prevents storage overflow

Prerequisites

Before testing, ensure you have:

  • Plex server running with music library
  • iPod/device connected and mounted
  • config.yaml configured correctly
  • Python environment activated
  • Recent backup of device state (optional but recommended)

Test Scenarios

Scenario 1: Fresh Device - Library Fits ✅

Objective: Verify Full Library Sync mode selection when library fits.

Setup:

  • Fresh iPod (no previous state file) OR delete .plex_sync_state.yaml
  • Library size < 90% of iPod storage

Steps:

  1. Run: python -m plex_ipod_sync
  2. Observe device detection
  3. Wait for "Analyzing library and storage..." message
  4. Verify mode selection prompt appears

Expected Results:

  • Prompt shows:
    • Music library size
    • Library track count
    • iPod storage capacity
    • Percentage usage (should be < 90%)
    • Two options: [1] Full Library Sync, [2] Playlist Sync
  • Selection [1] chosen
  • Message: "✓ Full Library Sync mode selected"
  • Message: "This choice will be saved for this device"
  • Section header: "FULL LIBRARY SYNC MODE"
  • "Fetching library tracks from Plex..."
  • "Fetching playlists from Plex..."
  • Success: "Library: [X] tracks, [Y] playlists"
  • Progress bar: "Copying tracks"
  • All tracks synced to iPod
  • All playlists created as .m3u files

Validation:

# Check state file on device
cat /Volumes/IPOD/.plex_sync_state.yaml | grep sync_mode
# Should show: sync_mode: full_library

# Check tracks on device
ls /Volumes/IPOD/Music
# Should show all artists/albums

# Check playlists
ls /Volumes/IPOD/Playlists
# Should show all .m3u files

Scenario 2: Fresh Device - Library Too Big ⚠️

Objective: Verify automatic Playlist mode when library doesn't fit.

Setup:

  • Fresh iPod OR delete .plex_sync_state.yaml
  • Library size > 90% of iPod storage
  • (Alternatively, temporarily reduce iPod mount point size for testing)

Steps:

  1. Run: python -m plex_ipod_sync
  2. Observe device detection
  3. Wait for "Analyzing library and storage..." message

Expected Results:

  • Warning message: 
    ⚠ Library size ([X] GB) exceeds iPod capacity ([Y] GB)
Using Playlist Sync mode
  • NO mode selection prompt
  • Continues with standard playlist selection
  • "PLAYLIST STATUS" section appears
  • Interactive: "Change playlist selection? (y/n)"

Validation:

# Check state file
cat /Volumes/IPOD/.plex_sync_state.yaml | grep sync_mode
# Should show: sync_mode: playlist

Scenario 3: Existing Device - Full Library Mode 🔄

Objective: Verify mode persistence across syncs.

Setup:

  • iPod already synced in Full Library mode
  • Add 5-10 new tracks to Plex library
  • Remove 2-3 tracks from Plex library

Steps:

  1. Run: python -m plex_ipod_sync
  2. Observe device detection

Expected Results:

  • Message: "Using saved sync mode: Full Library"
  • NO mode selection prompt
  • Section: "FULL LIBRARY SYNC MODE"
  • Tracks analyzed
  • Summary shows:
    • Tracks to add: [new tracks count]
    • Tracks to remove: [deleted tracks count]
  • Progress bar: "Copying tracks"
  • New tracks copied
  • Deleted tracks removed
  • Success message

Validation:

# Verify new tracks exist on iPod
find /Volumes/IPOD/Music -name "*[new track name]*"

# Verify removed tracks are gone
find /Volumes/IPOD/Music -name "*[deleted track name]*"
# Should return nothing

Scenario 4: Library Growth Detection 📈

Objective: Verify automatic transition when library grows beyond capacity.

Setup:

  • iPod in Full Library mode
  • Add enough music to Plex library to exceed 90% of iPod storage
  • (Alternatively, simulate by reducing available storage)

Steps:

  1. Run: python -m plex_ipod_sync
  2. Observe device detection
  3. Wait for library size check

Expected Results:

  • Warning panel: 
    ⚠ LIBRARY GROWTH DETECTED
  Library size: [X] GB
  iPod storage: [Y] GB
  Library now uses: [Z]% of storage

Your music library has grown beyond iPod capacity.
You need to switch to Playlist Sync mode to select specific playlists.
  • Prompt: "Switch to Playlist Sync mode now? (Y/n)"
  • Answer: Yes
  • Message: "✓ Switched to Playlist Sync mode"
  • Message: "Restarting sync with playlist selection..."
  • Sync restarts
  • Playlist selection interface appears

Validation:

# Check state file - mode should be changed
cat /Volumes/IPOD/.plex_sync_state.yaml | grep sync_mode
# Should show: sync_mode: playlist

Scenario 5: Multi-Computer Sync 💻💻

Objective: Verify mode persistence across different computers.

Setup:

  • Computer A: Sync device in Full Library mode
  • Computer B: Different computer with same tool installed

Steps:

  1. On Computer A:

    • Run sync, choose Full Library mode
    • Wait for completion
    • Safely eject iPod

  2. On Computer B:

    • Connect same iPod
    • Run: python -m plex_ipod_sync
    • Observe behavior

Expected Results:

  • Computer B detects device
  • Message: "Using saved sync mode: Full Library"
  • Multi-computer detection warning (from Week 4): 
    ⚠ MULTI-COMPUTER SYNC DETECTED
  Last synced from: [user@computerA]
  Current computer: [user@computerB]

Device settings are shared across computers.
Last sync wins - your changes will overwrite previous sync.

Continue with sync? (y/n):
  • Answer: Yes
  • Sync proceeds in Full Library mode
  • No re-prompting for mode selection

Validation:

# On both computers, verify state shows same mode
cat /Volumes/IPOD/.plex_sync_state.yaml | grep sync_mode
# Should show: sync_mode: full_library

# Verify last_sync_client reflects Computer B
cat /Volumes/IPOD/.plex_sync_state.yaml | grep -A 3 last_sync_client

Scenario 6: State Migration v2 → v3 🔄

Objective: Verify automatic state migration for existing devices.

Setup:

  • Device with v2 state file (pre-Full Library Sync)
  • Can simulate by manually editing .plex_sync_state.yaml: 
    version: 2
# (remove sync_mode field if present)

Steps:

  1. Run: python -m plex_ipod_sync
  2. Observe state loading

Expected Results:

  • Message: "Migrating sync state from v2 to v3..."
  • Migration completes silently
  • Sync proceeds normally
  • If library fits: Mode selection prompt appears
  • If library doesn't fit: Playlist mode used

Validation:

# Check state version
cat /Volumes/IPOD/.plex_sync_state.yaml | grep version
# Should show: version: 3

# Check sync_mode was added
cat /Volumes/IPOD/.plex_sync_state.yaml | grep sync_mode
# Should exist

Scenario 7: Dry Run Mode 🧪

Objective: Verify both modes work in dry run without making changes.

Setup:

  • Fresh iPod OR existing device

Steps:

  1. Run: python -m plex_ipod_sync --dry-run
  2. Select Full Library mode (if prompted)
  3. Review output

Expected Results:

  • Mode selection works (if fresh device)
  • Message: "[DRY RUN MODE - No changes will be made]"
  • Shows what would be synced:
    • Playlists that will be synced
    • Tracks that would be added (sample list)
    • Tracks that would be removed (sample list)
  • NO actual copying occurs
  • NO state file modifications
  • Exit cleanly

Validation:

# Verify no tracks were copied
# Count should be same before and after
find /Volumes/IPOD/Music -type f | wc -l

Scenario 8: Verbose Mode 🗣️

Objective: Verify verbose output shows detailed operations.

Setup:

  • Any device in any mode

Steps:

  1. Run: python -m plex_ipod_sync --verbose
  2. Observe output during sync

Expected Results:

  • Clean UI progress bars still appear
  • PLUS detailed file-by-file output:
    • "Copying: [Artist] - [Track]"
    • "✓ Copied artwork: [Artist] - [Album]"
    • "⬇ Downloaded artwork from Plex: [Artist] - [Album]"
  • Summary shows detailed breakdown:
    • File-based artwork: X copied, Y existed
    • Downloaded from Plex: Z
    • Albums without artwork: N

Validation:

  • Output should be significantly more detailed than normal mode
  • Progress dialogs still appear (not replaced by verbose output)

Scenario 9: Playlist Creation in Full Library Mode 📝

Objective: Verify all Plex playlists are created as .m3u files.

Setup:

  • Device in Full Library mode
  • Multiple playlists in Plex (at least 5)

Steps:

  1. Run sync in Full Library mode
  2. Wait for completion
  3. Check playlist directory

Expected Results:

  • All Plex playlists created as .m3u files
  • Playlist files contain correct paths
  • Playlists are playable in Rockbox

Validation:

# Check playlist count matches Plex
ls /Volumes/IPOD/Playlists/*.m3u | wc -l

# Verify playlist content
cat "/Volumes/IPOD/Playlists/[PlaylistName].m3u"
# Should show relative paths to tracks

# Check playlist in state
cat /Volumes/IPOD/.plex_sync_state.yaml | grep -A 2 playlists:

Scenario 10: No Interaction Mode 🤖

Objective: Verify --no-interaction works with saved mode.

Setup:

  • Device with saved sync mode (either full_library or playlist)

Steps:

  1. Run: python -m plex_ipod_sync --no-interaction

Expected Results:

  • No prompts
  • Uses saved sync mode
  • Syncs automatically
  • Completes without user input

Edge Case:

  • If device is fresh (no saved mode): Should error or use default

Scenario 11: Source Field Tracking 🏷️

Objective: Verify tracks are labeled with correct source.

Setup:

  • Device with tracks from different sources

Test Cases:

11a: Full Library Mode

# Sync in full library mode
python -m plex_ipod_sync

# Check state file
cat /Volumes/IPOD/.plex_sync_state.yaml | grep -A 5 "tracks:"
# All tracks should show: source: library

11b: Playlist Mode

# Sync in playlist mode with AC disabled
python -m plex_ipod_sync

# Check state file
cat /Volumes/IPOD/.plex_sync_state.yaml | grep "source:"
# Should show: source: playlist

11c: Auto-Complete Mode

# Sync in playlist mode with AC enabled

# Check state file
cat /Volumes/IPOD/.plex_sync_state.yaml | grep "source:"
# Should show mix of: source: playlist, source: auto_complete

Scenario 12: Album Artwork Sync 🖼️

Objective: Verify artwork syncs correctly in both modes.

Setup:

  • Albums with embedded artwork files
  • Albums without local artwork (require Plex API download)

Steps:

  1. Run sync in Full Library mode
  2. Check artwork copying

Expected Results:

  • Section: "SYNCING ALBUM ARTWORK"
  • Progress dialog: "Syncing artwork [X/Y]"
  • Recent albums shown in progress (5 most recent)
  • Summary: "Synced artwork for X albums"
  • Verbose mode shows:
    • File-based artwork: X copied, Y existed
    • Downloaded from Plex: Z
    • Albums without artwork: N

Validation:

# Check for cover.jpg files
find /Volumes/IPOD/Music -name "cover.jpg" | wc -l

# Verify state tracks artwork
cat /Volumes/IPOD/.plex_sync_state.yaml | grep -A 3 "albums:"
# Should show artwork: true for albums with art

Scenario 13: Space Checking ⚙️

Objective: Verify space checking works correctly in both modes.

Setup:

  • Device with limited space

Test Cases:

13a: Full Library - Sufficient Space

  • Library < 90% storage
  • Sync proceeds normally

13b: Full Library - Insufficient Space

  • Library > 90% storage
  • Warning appears, switches to playlist mode

13c: Playlist Mode - Insufficient Space

  • Selected playlists + AC > available space
  • Warning appears with option to disable AC

Expected Results:

  • Space report shows:
    • Available storage
    • Required space
    • Usage percentage
    • Buffer calculations

Scenario 14: Error Recovery 🔧

Objective: Verify graceful error handling.

Test Cases:

14a: Corrupted State File

  1. Manually corrupt .plex_sync_state.yaml
  2. Run sync
  3. Should recover from backup or prompt for empty state

14b: Network Interruption

  1. Start sync
  2. Disconnect network during Plex fetch
  3. Should error gracefully with clear message

14c: Device Disconnected Mid-Sync

  1. Start sync
  2. Eject device during copy
  3. Should error and not corrupt state (lock file prevents re-entry)

Scenario 15: State Restoration 📦

Objective: Verify --restore-state works with new sync_mode field.

Setup:

  • Device with backups (from previous syncs)

Steps:

  1. Run: python -m plex_ipod_sync --restore-state
  2. Select a backup
  3. Confirm restoration

Expected Results:

  • Lists available backups (newest first)
  • Shows timestamp and size
  • Prompt for selection
  • Warning about overwriting current state
  • Restoration succeeds
  • State includes sync_mode field

Validation:

# Verify restored state has sync_mode
cat /Volumes/IPOD/.plex_sync_state.yaml | grep sync_mode

# Verify restored state version
cat /Volumes/IPOD/.plex_sync_state.yaml | grep version
# Should be v3

Edge Cases & Special Scenarios

Edge Case 1: Empty Library

  • Library has 0 tracks
  • Should handle gracefully, show "No tracks to sync"

Edge Case 2: Extremely Large Library (10,000+ tracks)

  • Performance test
  • Progress indicators should work smoothly
  • May take significant time to fetch/analyze

Edge Case 3: Special Characters in Filenames

  • Tracks with Unicode, emojis, special chars
  • Should sanitize and copy correctly

Edge Case 4: Missing Track Files

  • Plex shows track but file doesn't exist
  • Should skip with error message (verbose mode)
  • Should not abort entire sync

Edge Case 5: Read-Only Device

  • iPod mounted read-only
  • Should error early with clear message

Edge Case 6: Multiple Devices

  • Switch between different iPods
  • Each should maintain its own sync mode
  • No cross-contamination

Performance Benchmarks

Track these metrics during testing:

Operation Small Library (500 tracks) Medium Library (2,000 tracks) Large Library (10,000 tracks)
Library size calculation < 5s < 10s < 30s
Mode selection prompt < 1s < 1s < 1s
Track fetching < 10s < 30s < 2m
Playlist fetching < 5s < 5s < 10s
Change calculation < 5s < 15s < 1m
First full sync ~ 30m ~ 2h ~ 6h
Incremental sync (10 new tracks) < 2m < 3m < 5m

Regression Testing

Verify existing features still work:

  • Playlist-based sync (original behavior)
  • Auto-complete albums
  • Album artwork downloading
  • State backup to client
  • Multi-computer sync warnings
  • Sync lock prevents concurrent syncs
  • State recovery from corruption
  • Orphaned AC track cleanup
  • Verbose mode detailed output
  • Clean UI (no timers, simplified progress)
  • Dry run mode

Test Data Setup

Recommended Test Library

For comprehensive testing, ensure your Plex library has:

  • 100-500 tracks (small test set)
  • Multiple artists (20+)
  • Multiple albums (50+)
  • 5-10 playlists
  • Some albums with artwork files
  • Some albums without artwork (will download from Plex)
  • Mix of audio formats (MP3, FLAC, M4A, etc.)
  • Tracks with special characters in names

Test Device Setup

Ideally test with:

  • 32GB device (small)
  • 64GB device (medium)
  • 128GB+ device (large)

OR use loopback devices with varying sizes.

Reporting Issues

When reporting bugs, include:

  1. Scenario: Which test scenario failed
  2. Device Info: Storage size, device ID, current state
  3. Library Info: Track count, total size
  4. Command: Exact command run
  5. Expected: What should have happened
  6. Actual: What actually happened
  7. Logs: Terminal output (verbose mode helpful)
  8. State File: Contents of .plex_sync_state.yaml (sanitized)

Success Criteria

Full Library Sync feature is ready for production when:

  • ✅ All 15 scenarios pass
  • ✅ All edge cases handled gracefully
  • ✅ Performance within acceptable ranges
  • ✅ No regressions in existing features
  • ✅ State migration works flawlessly
  • ✅ Multi-computer sync verified
  • ✅ Growth detection transitions smoothly
  • ✅ Documentation complete
  • ✅ User experience smooth and intuitive

Quick Test Script

For rapid testing, use this checklist:

# 1. Fresh device, library fits
rm /Volumes/IPOD/.plex_sync_state.yaml
python -m plex_ipod_sync
# Select [1] Full Library
# Verify: All tracks sync

# 2. Reconnect, verify mode persists
python -m plex_ipod_sync
# Verify: No prompt, uses saved mode

# 3. Test verbose mode
python -m plex_ipod_sync --verbose
# Verify: Detailed output shown

# 4. Test dry run
python -m plex_ipod_sync --dry-run
# Verify: No changes made

# 5. Verify state file
cat /Volumes/IPOD/.plex_sync_state.yaml | head -20

# 6. Verify playlists created
ls /Volumes/IPOD/Playlists/

# 7. Verify tracks synced
find /Volumes/IPOD/Music -type f | wc -l

# 8. Check artwork
find /Volumes/IPOD/Music -name "cover.jpg" | wc -l

Notes

  • Keep backups before testing destructive scenarios
  • Use --dry-run frequently to preview changes
  • Test on non-production devices when possible
  • Document any unexpected behavior
  • Consider edge cases specific to your setup

Testing Status: ⏳ Ready for QA

Last Updated: 2025-10-30