feat(T4MetricV2): add object distributions to T4MetricV2 dataframe

[KSeangTan]

Summary

This PR adds distributions of objects when creating info pickles files and save them to a parquet file based on their split. It also support to load metadata for test set from the parquet file instead of generating them when running evaluation.

Change point

• Add testing_statistics_parquet_path to T4MetricV2 [1].
• Update _process_evaluator_results to remove sample_id_to_prefix_mapping [2].
• Update _process_metrics_for_aggregation to remove sample_id_to_prefix_mapping [3].
• Update T4MetricV2DataFrame to include testing_statistics_parquet_path [4].
• Add parquet data from a test split to a T4MetricV2 dataframe [5].
• Predefine polars column schema when creating a dataframe [6].

Test performed

• Example for the dataframe

location	vehicle_type	suffix_name	train_num_frame_distribution_keys	train_num_frame_distribution_values	train_total_num_frames	train_ego_pose_translation_x_keys	train_ego_pose_translation_x_values	train_ego_pose_translation_y_keys	train_ego_pose_translation_y_values	train_ego_pose_translation_z_keys	train_ego_pose_translation_z_values	train_car_frequency_distribution_keys	train_car_frequency_distribution_values	train_car_area_distribution_keys	train_car_area_distribution_values	train_car_volume_distribution_keys	train_car_volume_distribution_values	train_car_orientation_distribution_keys	train_car_orientation_distribution_values	train_car_translation_x_distribution_keys	train_car_translation_x_distribution_values	train_car_translation_y_distribution_keys	train_car_translation_y_distribution_values	train_car_translation_z_distribution_keys	train_car_translation_z_distribution_values	train_car_bev_radial_distance_distribution_keys	train_car_bev_radial_distance_distribution_values	train_car_velocity_y_distribution_keys	train_car_velocity_y_distribution_values	train_car_velocity_x_distribution_keys	train_car_velocity_x_distribution_values	train_car_num_lidar_pts_distribution_keys	train_car_num_lidar_pts_distribution_values	train_truck_frequency_distribution_keys	train_truck_frequency_distribution_values	train_bicycle_orientation_distribution_keys	train_bicycle_orientation_distribution_values	train_bicycle_translation_x_distribution_keys	train_bicycle_translation_x_distribution_values	train_bicycle_translation_y_distribution_keys	train_bicycle_translation_y_distribution_values	train_bicycle_translation_z_distribution_keys	train_bicycle_translation_z_distribution_values	train_bicycle_bev_radial_distance_distribution_keys	train_bicycle_bev_radial_distance_distribution_values	train_bicycle_velocity_y_distribution_keys	train_bicycle_velocity_y_distribution_values	train_bicycle_velocity_x_distribution_keys	train_bicycle_velocity_x_distribution_values	train_bicycle_num_lidar_pts_distribution_keys	train_bicycle_num_lidar_pts_distribution_values	train_pedestrian_frequency_distribution_keys	train_pedestrian_frequency_distribution_values	train_pedestrian_area_distribution_keys	train_pedestrian_area_distribution_values	train_pedestrian_volume_distribution_keys	train_pedestrian_volume_distribution_values	train_pedestrian_orientation_distribution_keys	train_pedestrian_orientation_distribution_values	train_pedestrian_translation_x_distribution_keys	train_pedestrian_translation_x_distribution_values	train_pedestrian_translation_y_distribution_keys	train_pedestrian_translation_y_distribution_values	train_pedestrian_translation_z_distribution_keys	train_pedestrian_translation_z_distribution_values	train_pedestrian_bev_radial_distance_distribution_keys	train_pedestrian_bev_radial_distance_distribution_values	train_pedestrian_velocity_y_distribution_keys	train_pedestrian_velocity_y_distribution_values	train_pedestrian_velocity_x_distribution_keys	train_pedestrian_velocity_x_distribution_values	train_pedestrian_num_lidar_pts_distribution_keys	train_pedestrian_num_lidar_pts_distribution_values
str	str	str	list[str]	list[i64]	i64	list[str]	list[list[f64]]	list[str]	list[list[f64]]	list[str]	list[list[f64]]	list[str]	list[i64]	list[str]	list[list[f64]]	list[str]	list[list[f64]]	list[str]	list[list[f64]]	list[str]	list[list[f64]]	list[str]	list[list[f64]]	list[str]	list[list[f64]]	list[str]	list[list[f64]]	list[str]	list[list[f64]]	list[str]	list[list[f64]]	list[str]	list[list[i64]]	list[str]	list[i64]	list[str]	list[list[f64]]	list[str]	list[list[f64]]	list[str]	list[list[f64]]	list[str]	list[list[f64]]	list[str]	list[list[f64]]	list[str]	list[list[f64]]	list[str]	list[list[f64]]	list[str]	list[list[i64]]	list[str]	list[i64]	list[str]	list[list[f64]]	list[str]	list[list[f64]]	list[str]	list[list[f64]]	list[str]	list[list[f64]]	list[str]	list[list[f64]]	list[str]	list[list[f64]]	list[str]	list[list[f64]]	list[str]	list[list[f64]]	list[str]	list[list[f64]]	list[str]	list[list[i64]]

[Copilot]

Pull request overview

This PR enables test set object distribution tracking by adding object distribution metadata to T4MetricV2 dataframes. The changes allow loading pre-computed test set statistics from parquet files instead of regenerating them during evaluation.

Changes:

• Added testing_statistics_parquet_path parameter to T4MetricV2 and T4MetricV2DataFrame for loading test set statistics
• Extended T4DatasetStatistics to capture object distributions (area, volume, orientation, translation, velocity, etc.) per class
• Removed sample_id_to_prefix_mapping from evaluation processing as test distributions are now loaded from parquet files

Reviewed changes

Copilot reviewed 7 out of 7 changed files in this pull request and generated 3 comments.

Show a summary per file

File	Description
tools/detection3d/t4dataset_converters/t4dataset_statistics.py	Implements object distribution collection for all classes with predefined schema and extraction logic
tools/detection3d/create_data_t4dataset.py	Updates statistic collection to pass infos and bev_distance_range to add_samples
projects/CenterPoint/configs/t4dataset/Centerpoint/second_secfpn_4xb16_121m_j6gen2_base_amp_t4metric_v2.py	Adds testing_statistics_parquet_path to configuration
projects/CenterPoint/configs/t4dataset/Centerpoint/second_secfpn_4xb16_121m_base_amp_t4metric_v2.py	Adds testing_statistics_parquet_path to configuration
autoware_ml/detection3d/evaluation/t4metric/t4metric_v2_dataframe.py	Loads and joins test statistics parquet to evaluation dataframe
autoware_ml/detection3d/evaluation/t4metric/t4metric_v2.py	Removes sample_id_to_prefix_mapping logic and accepts testing_statistics_parquet_path
autoware_ml/configs/detection3d/dataset/t4dataset/j6gen2_base.py	Simplifies dataset_test_groups to single entry

---

💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.