Skip to content

Conversation

@sridhs21
Copy link
Contributor

This PR adds performance tracking and evaluation metrics.

Changes

Performance Benchmarking (new module: benchmark.py)

  • Tracks hardware specifications (CPU, GPU, RAM, CUDA version)
  • Monitors training performance: epoch timing, batch throughput, samples/second
  • Records GPU memory usage (current and peak allocation)
  • Exports benchmark data to JSON via --benchmark and --benchmark-output flags

Model Evaluation Metrics (new module: eval_metrics.py)

  • Evaluates model on both training and validation sets post-training
  • Computes standard metrics: Precision, Recall, F1 Score, IoU, Accuracy
  • Provides confusion matrix breakdown (TP, FP, FN, TN) per-frame and globally
  • Automatic overfitting detection by comparing train vs validation F1 scores
  • Exports evaluation results to JSON via --eval-output flag

Integration in XPointMLTest.py

  • Import benchmark and evaluation modules
  • Pass benchmark tracker through training loop when enabled
  • Run evaluation after training completes
  • Generate formatted console output and JSON exports for both systems

Benefits

  • Monitor hardware utilization and training efficiency
  • Quantify model accuracy with standard ML metrics
  • Detect overfitting automatically
  • Enable reproducible comparisons across runs via JSON exports

Comment on lines +17 to +31
Evaluates model performance on X-point detection task.

Computes metrics including:
- True Positives (TP): X-point pixels correctly identified
- False Positives (FP): Background pixels incorrectly labeled as X-points
- False Negatives (FN): X-point pixels that were missed
- True Negatives (TN): Background pixels correctly identified

Metrics calculated:
- Accuracy: (TP + TN) / (TP + TN + FP + FN)
- Precision: TP / (TP + FP)
- Recall: TP / (TP + FN)
- F1 Score: 2 * (Precision * Recall) / (Precision + Recall)
- IoU: TP / (TP + FP + FN)
"""
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Great. Thank you.

@cwsmith
Copy link
Contributor

cwsmith commented Oct 20, 2025

@sridhs21 Thank you. Would you please rebase on main or merge main into this branch?

@sridhs21 sridhs21 force-pushed the performance-profiling branch from 0d96a23 to 426184e Compare October 21, 2025 11:17
Copy link
Contributor

@cwsmith cwsmith left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Looks great. Thank you.

@cwsmith cwsmith merged commit 3635d7b into main Oct 21, 2025
1 check passed
@cwsmith cwsmith deleted the performance-profiling branch October 21, 2025 16:59
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

3 participants