camera_arm

Allows robot arms to be repurposed as camera arms.

Installation

Create virtual environment and install required packages. Using python package manager: uv

uv sync

or alternatively

mkdir -p .venv/py3.12
python3.12 -m venv .venv/py3.12
pip install -r requirements.txt

Optional Installation You can also opt in to install several other object detection models.

1. mediapipe
   • Run uv sync --extra mediapipe
   • This will add the mediapipe model options in the model dropdown menu
2. yolo
   • Run uv sync --extra yolo
   • This will add the yolo model option in the model dropdown menu
3. all
   • Run uv sync --all-extras
   • This will add all of the extra model options in the dropdown menu

Setting up the virtual camera

In order to stream video out of commander, you will need to set up a virtual camera on your computer. This will allow you to select the commander video stream as a camera input in other applications (e.g. zoom, obs, etc.). Please refer to the pyvirtualcam documentation for instructions on how to set up a virtual camera on your operating system: https://github.com/letmaik/pyvirtualcam. Note: If you are on MacOS or Windows the default backend for pyvirtualcam is obs while v4l2 is the default for Linux. If you want to use the obs backend on MacOS or Windows, you will need to have obs installed and running. Refer to pyvirtualcam's documentation for additional setup instructions. If you want to use the v4l2 backend on Linux, you will need to have v4l2loopback installed and set up.

Creating an executable

Pyinstaller can be used to create an executable (on any OS) for easier launching/distribution of the application. Follow the steps below to create an executable:

This is especially recommended if you want to use the talos executable on a Linux machine since the executable can be upwards of 3.9 GB, making it hard for us to easily create and distribute while using the free tier of GitHub Actions.

1. Install pyinstaller:

   If using uv, then the dev dependencies should be installed by default by running:

   uv sync --group dev

   If not using uv:

   pip install pyinstaller

2. Run the pyinstaller command to create an executable. This will create a dist folder with the executable talos inside.

   Using uv:

   uv run pyinstaller_runner.py

   Otherwise:

   python pyinstaller_runner.py

Known bugs with arm based macs

1. Installation fails to build pybullet If pybullet build fails set a flag during installation:

CFLAGS="-Dfdopen=fdopen" pip install -r requirements.txt

or with uv

CFLAGS="-Dfdopen=fdopen" uv sync

2. Tcl wasn't installed properly when running uv run talos.py, an error occurs: This probably means that Tcl wasn't installed properly.. Fix: brew install tcl-tk if this doesn't work, reinstall python for uv uv python uninstall 3.12 && uv python install 3.12

Setup

Activate your virtual environment (update 2025: no longer needed with uv)

source venv/py3.12/bin/activate

Running

Using uv, run the general app entry point:

uv run commander

Run the TUI interface:

uv run commander-terminal

or

uv run commander -t

Run the Tk GUI interface:

uv run commander-tk

Configurations

The config/example_default_config.yaml file contains default parameters for the application that will be used to fill in the new connections configs. You can override these parameters using config/default_config.local.yaml which will be prioritized over the example default config.

When you create a new connection in the commander application, it will automatically create a new config file, config/robot_configs.local.yaml. This file will contain the parameters for each connection you create. You can also edit this file directly to change the parameters for each connection.

App settings overrides

AppSettings are loaded from config/app_settings.local.yaml, and can be overridden by environment variables.

Priority order:

1. Environment variables (including .env)
2. config/app_settings.local.yaml
3. Hardcoded defaults in the schema

Supported environment variables:

• COMMANDER_LOG_LEVEL
• COMMANDER_BBOX_MAX_FPS
• COMMANDER_FRAME_PROCESS_FPS

Example:

export COMMANDER_BBOX_MAX_FPS=15
uv run commander

Troubleshooting

If you are having trouble connecting to the arm, try running the following command on the Pi:

ls /dev/tty*

and ensure that you see a device named /dev/ttyUSB0 If you don't see it, try unplugging and replugging the arm and running the command again. If it still doesn't show up, try restarting the Pi. If you see the device, but still can't connect, try running the following command on the Pi:

screen /dev/ttyUSB0 9600

Hit enter a few times to see if > shows up on each line. If it does, then the arm is connected properly. If not, try restarting the robot arm and running the command again. After testing, exit the screen session by typing Ctrl+A then K then Y.