A set of Python scripts that capture video input, recognize body landmarks, and send their coordinates via Open Sound Control (OSC) protocol, so that they can be used in software like Max/MSP, TouchDesigner, Pure Data or any OSC-capable DAW or plugin. Google MediaPipe library and models are used for computer vision tasks.
Download the latest version:
git clone https://github.com/valeriorlandini/visiosc.git (or cd visiosc && git pull to update an existing download)
Once inside the visiosc folder, install the necessary dependencies with:
pip install -r requirements.txt
Then, refer to the following guides for using the scripts.
Coordinates for x and y are always inside [0.0, 1.0] range, with the origin being the left bottom corner (camera image is automatically mirrored).
For each script, the following options can be specified:
| Option | Description |
|---|---|
-a ADDRESS, --address ADDRESS |
Address to send OSC messages (default: 127.0.0.1) |
-d DEVICE, --device DEVICE |
Index of the video device to use (default: 0, if you have multiple video input devices you might have to try different values) |
-f FILE, --file FILE |
Path to a video file to process instead of a live camera feed |
-l, --loop |
Loop the video file when it ends (ignored when using a live camera feed) |
-m MODE, --mode MODE |
Send a simplified and named list of the landmarks (0, default) or send all the numbered landmarks (1) |
-p PORT, --port PORT |
Port to send OSC messages (default: 8000) |
-s WIDTH HEIGHT, --size WIDTH HEIGHT |
Width and height of the capture window (default: 640 480, ignored when --file is used) |
The script visiosc.py is a single entry point that can launch one or more trackers at once. Use the -t / --tracking option to specify which tracker(s) to run. Multiple trackers can be requested by repeating the flag or by passing a comma-separated list. If -t is omitted, all three trackers are launched. All other options are forwarded to the relevant tracker(s).
# Run all three trackers
python visiosc.py
# Run only face and hands tracking
python visiosc.py -t face -t hands
# or equivalently
python visiosc.py -t face,hands
# Run pose tracking on a looping video file
python visiosc.py -t pose -f video.mp4 -lWhen multiple trackers are requested, each one runs in its own independent process with its own window. Pressing any key in a tracker's window closes that tracker; pressing Ctrl+C in the terminal stops them all.
The script hand_track.py recognizes 21 hand landmarks for each hand. If two right or two left hands appear in the video stream, only one will be detected (you can easily modify this behavior by editing the code). The default mode does not pass to OSC stream the 21 landmarks, but the coordinates of the tips of the fingers, the wrist and the palm. This behavior can be changed with the appropriate option (see options table). Launch it with:
python hand_track.py
The script face_track.py recognizes 478 face landmarks on the face (one face is detected, this can optionally be changed in the code). The default mode does not pass to OSC stream the 478 landmarks, but the coordinates of the main parts of the face (chin, lips, mouth, cheeks, nose, eyes, eyebrows, forehead). This behavior can be changed with the appropriate option (see options table). Launch it with:
python face_track.py
The script pose_track.py recognizes 33 pose landmarks on whole body. The default mode does not pass to OSC stream the numbered 33 landmarks, but the named body parts minus some redundant ones but plus the torso and mouth centres. This behavior can be changed with the appropriate option (see options table). Launch it with:
python pose_track.py
Pose tracking works even if a whole figure is not detected, and tries to guess the coordinates of the out-of-screen body parts. For this reason, some coordinates may be outside [0.0, 1.0] range. The script has a -o / --out option that if set to 0 filters out all out-of-screen coordinates, if set to 1 clamps all the coordinates inside [0.0, 1.0] range and if set to 2 (which is the default) passes all the coordinates as they are. When using visiosc.py, this option is forwarded exclusively to the pose tracker.
Max for Live devices are available in the maxforlive folder. They can be used to receive the OSC messages sent by the scripts and use them to control parameters in Ableton Live. Each device is designed to work with a specific tracker, when launched in simplified mode. If you want to use the devices with the full list of landmarks instead, you can easily modify the code to match the OSC addresses and arguments expected by the devices, by opening the .amxd files in Max and editing the parsed path of the incoming OSC messages.
