Improved pixel to meter conversion, OpenSim scaling, person sorting

Pose2Sim v0.10.43

• Sorting: If a person is not seen for more than 100 frames (tunable), they are considered stale and they won't be confused with a person passing by
• Pixel to meter conversion: Better estimation of the factor in case of noisy data:
  • exclude nans before finding the best frames for calculation;
  • do not exclude fastest and slowest frames anymore: fragile, it is replaced with trimmed mean on height;
  • large_hip_knee_angles should be 135°, not 45
• Scaling and IK:
  • Large improvements on upper-body scaling of the marker augmented model
  • You can now filter IK (mot files) instead of marker trajectories (trc files) with filter_ik = true
• Others:
  • Automatically set project_dir in config_dict (also writes logs in project_dir instead of current dir)
  • Added subject mass in logs

Full Changelog: v0.10.42...v0.10.43

Filter out ghost detections, multiple video format support, parallelized kinematics

Pose2Sim v0.10.42

• Filter out ghost detections (tripods, for example) by adding average_likelihood_threshold_pose parameter
• Parallelized kinematics across persons. Also attempted to parallelize person association and triangulation, but it was needlessly complex and not usually faster.
• Automatic video discovery: supports multiple video formats in the same session, removed vid_img_extension parameter
• Ensured backward compatibility by assigning default value in all configuration parameters
• Fixed "set_always_on_top" for manual calibration, which now uses PySide6 instead of PyQt5
• Fixed unresponsive Ctrl+C interrupt in pose estimation (with LLM help)

Note:

Python only allows for one thread at a time: it uses a GIL (Global Interpreter Lock). So in theory, it is not possible to parallelize anything. There are two workarounds:

• Multiprocessing: Creating independent processes instead of threads. A thread is a lightweight worker inside the main program, while a process is a fully independent Python instance--which takes time to initialize. It is valuable for long tasks (one process per person for inverse kinematics, for example), but not at all for smaller ones (one process per triangulated frame).
• Multithreading: Some C++ libraries were created with multithreading in mind and have bindings for Python, such as numpy, opencv, onnx runtime, file writing. In such cases, the Python GIL is bypassed, which is why pose estimation becomes so much faster. However, some libraries, even written in C++, do not support it (for example, OpenSim). And anything purely python, such as loops, list operations, json parsing, can only run one thread at a time. For person association and triangulation, the cost of the thread overhead is similar to the gains or parallelization, so it is not worth parallelizing.

For future reference:

• Further speeding up person association, triangulation (and even pose estimation) would primarily be done by saving the pose estimation results as a single csv or h5 file instead of multiple per-frame json files, but that's another consideration and it would require some additional work and testing.
• Another opportunity to speed up triangulation is to use the same association logic as in the multi-person mode of personAssociation.
• In fact, we could get rid of the single person mode altogether. The multi-person mode should be enhanced to order the triangulated persons with the person_ordering_method parameter (by greatest displacement, least displacement, most successfully triangulated frames, least excluded cameras, or on click, similarly to what's done in Sports2D), and only a subset of person would be selected with nb_persons_to_detect (all, 1, 2...).

Full Changelog: v0.10.41...v0.10.42

Parallel pose estimation + other fixes

Pose2Sim v0.10.41

• Pose estimation: Enable parallel pose estimation of multiple videos, which makes it much faster. Only available if if display_detection=False. Thanks @f-fraysse!
  See this part of the documentation to drop your pose estimation times from 1 min 23 s to 5 s!

• Calibration: Removed res from read_qca, as resolution is computed later

• Synchronization: Fixed synchronization issue when GUI is false

• Person association: In single person person association, solved n_cams_off > min_cams_triangulation
• Person association: Filtered out persons with all NaN keypoints in JSON pose file, preventing unnecessary combinations in person_combinations()

• Filtering: Implemented zero-phase oneeuro filter, and removed some unused condition in the other filters

• Kinematics: Clamped l5_S1 axial rotation to prevent 360° twists
• Kinematics: Harmonized mediapipe keypoint names with other pose models

• Others: Switched from PyQt5 to PySide6 (may need a clean reinstall)
• Others: Height measurement: excludes low speeds based on percentage rather than absolute value
• Others: Interpolation: Deactivated extrapolation, which created unnatural trajectories at the beginning and end
• Others: Fixed angle computation issues: flips, mirrored, wrong wrapping, etc. In particular, used interior angles
• Others: trc_rotate: Dropped "Unnamed" columns and handled pathlib paths
• Others: Silenced some warnings
• Others: Fixed continuous integration for MacOS

• Instructions: Clarifications in Config.toml and Readme.md

What's Changed

• Filter out persons with all NaN keypoints during personAssication by @ErwanBeurienne in #210
• Fix n_cams_off > min_cams_for_triangulation in personAssociation by @leadroletroy in #203
• Switch from PyQt5 to PySide6 by @mprib in #223
• Add parallel_pose_detection option for multi-video threading by @f-fraysse in #224

New Contributors

• @ErwanBeurienne made their first contribution in #210
• @leadroletroy made their first contribution in #203
• @mprib made their first contribution in #223
• @f-fraysse made their first contribution in #224

Full Changelog: v0.10.40...v0.10.41

Improved calibration, new Utility scripts, ...

Improved calibration procedure:

Calibration can be a hassle when you have numerous cameras. If you make a mistake, you need to restart the process from scratch, and painfully click on all the image points on all videos again.

I now save the position of the clicked points and their corresponding object points, so that if anything happens, whether you make a mistake or the code throws an error, you don't have to label again. Extrinsic parameters are also recalculated on the spot if you manually edited the intrinsic parameters.

For each camera, if points have already been clicked, Pose2Sim shows you the image points and their reprojected object points, and asks you if you are satisfied. If so, it saves the newly calculated extrinsic parameters, and moves on to the next image. If not, it lets you reclick your points (or tries to automatically find them in case of board calibration).

Other changes:

• Added trc_rotate.py Utility script
• Added trc_scale.py Utility script
• Fixed calibration conversion for new Qualisys camera models
• extrinsic images or videos can be put in the "extrinsic" folder instead of in the "extrinsic/cam_name" one
• Person height calculation: if the top_head point is not available, the head segment length is calculated as (Midshoulder to Nose) * 1.5
• Filtered some annoying numpy warnings
• Factored the "setup_model_class_mode". If the future, Sports2D will import them from Pose2Sim
• removed deepsort dependency
• slightly reorganized Config.toml (no breaking changes)

Minor edits for conda-forge acceptance

• Smoke test on entry points
• Removed deep-sort-realtime for conda-forge acceptance

Full Changelog: v0.10.38...v0.10.39

Improve sorting and fixed bad estimates in the triangulation recap

• Rewrote the sorting algorithm to handle some swaps (more information in the latest Sports2D release: https://github.com/davidpagnon/Sports2D/releases/tag/v0.8.22):

  • Added a distance constraint, so that if the best association between a frame and the next one is too far, it creates a new person instead
  • Switched to frame-by-frame median keypoint distance (instead of mean), in order not to ignore outliers
  • Ran non-maximum suppression (NMS) for bounding boxes recomputed from keypoints instead of straight from the person detector, which is more accurate and prevents having 2 boxes for the same person
  • Added a likelihood threshold in the keypoints used to recompute the bounding boxes to ignore points that were probably wrongly estimated
  • This made me rewrite the algorithm from scratch, but with the same logic. Among other edits: I used the Hungarian algorithm from scipy.optimize.linear_sum_assignment, so my custom greedy min_with_single_indices function is not required anymore. This is very slightly slower with 2-3 people in the scene, but faster in crowded scenes.

• Rewrote the triangulation module:

  • Fixed the IDs and count of excluded cameras, which were sometimes not accurate.
  • Computed the recap indicators person by person.
  • First participant is now #0 for better consistency with the rest of the library
  • Better excluded badly triangulated estimations, based on min_chunk_size instead of a threshold of 4 valid frames

• Updated marker locations for nose, left_eye, and right_eye for BlazePose markerset.

• Fixed pose estimation with save_video = 'to_images'

• Clarified some of the docstrings

Full Changelog: v0.10.37...v0.10.38

More robust triangulation and various other improvements

Installation:

• Python 3.12 by default
• OpenCV: removed <4.12 constraint since numpy>=2.0 is now supported, and added !=4.11 constraint due to displaymatrix being ignored
• constrained numpy version to python version
• pip install pose2sim is now done before conda install opensim

Calibration:

• Fixed wrong orientation with vertical checkerboard
• Calibration points window stays always on top

Config.toml:

• reorganized triangulation arguments for more clarity
• increased interp_if_gap_smaller_than to 20 px instead of 10
• added remove_incomplete_frames parameter to remove a full frame if any keypoint is missing
• added save_sync_plots and save_filt_plots` parameters

Triangulation:

• make_trc: removed f_range argument. This is automatically and more accurately retrieved from Q.index
• recap_triangulate: Added warning message when some time frames had to be trimmed. Also removed f_range argument.
• changed order of operations:
  1. Interpolate small missing sections (uses interp_if_gap_smaller_than)
  2. Trim sections where person is out of the frame (uses sections_to_keep, remove_incomplete_frames, and ignores small sections)
  3. Remove persons with fewer than 4 correct frames
  4. Fill non-interpolated values (uses fill_large_gaps_with)
  5. Replace missing keypoints with zeros (otherwise, marker augmentation fails)

  . gaps, _ valid values
  
  0) ....._.....____._....._____.....
  1) ....._.....______....._____.....
  2)            ______....._____     
  3) 
  4)            ________________     
  5) 
  

Miscellaneous:

• all: np.set_printoptions(legacy='1.21') in order to print 3.0 instead of np.float64(3.0) np>=2.0
• Pose2Sim.py: provided default value in logging parameter
• poseEstimation: clearer error message and logs when no video is found
• synchronization.py: clearer logging messages and slightly better design
• markerAugmentation.py: replaced isnan by isfinite to account for np.inf, too
• kinematics.py: typo in logging
• common.py: interpolate_zero_nans now returns pandas series instead of of numpy array in order not to remove the column name
• edits to tests.py

Full Changelog: v0.10.36...v0.10.37

Fixed ID swaps

• Corrected LPinky in marker files (position to fixed=true, like all others)

• Fixed ID swaps:
  Non-maximum suppression (NMS) consists in ignoring all bounding boxes that overlap by more than a 0.45 threshold except the one with highest confidence. RTMlib natively runs it.
  However, this is done at the person detection level, and was not always satisfactory. Pose estimation does not exclusively look into the detected bounding box, and sometimes finds points outside. In practice, 2 bounding boxes that do not overlap much (large border) can lead to the same detected skeleton.
  I recalculated bounding boxes at the pose estimation level, ie based on skeleton detection (thin border), and ran NMS from there. This fixes ID swaps in most cases.
  

Full Changelog: v0.10.35...v0.10.36

Various edits, eg to solve numpy version issues

• Limits cv2 version, otherwise forces numpy>=2 which is incompatible with some OpenSim versions

• Fixed eye position for some marker sets, and added hat_spine to Geometries

• Back to the previous way for Vicon calibration, since the new one missed some cameras sometimes

• The py-c3d library on which Pose2Sim was dependent used not to support numpy>=2.0. This has started to cause problems for some users. I worked on the py-c3d library to make it compatible with numpy>=2, made a pull request that was accepted, so this problem is solved now: EmbodiedCognition/py-c3d#54

Full Changelog: v0.10.34...v0.10.35

Fixed GCV filtering + Expired OpenMMLab certificate

• Fixed gcv filtering (careful if series too short: noise can be considered as signal -> no filtering. See this conversation: https://stackoverflow.com/a/79740481/12196632, and this issue: scipy/scipy#23472
• Temporarily ignore SSL certificate verification to handle OpenMMLab's expired certificate
• Handle RTMO if pose_model is not 'body'

Full Changelog: v0.10.33...v0.10.34