A worker died or was killed while executing a task by an unexpected system error.

[ZhonghaoLu]

--actor_num_nodes 1
--actor_num_gpus_per_node 8
--vllm_num_engines 8
--vllm_tensor_parallel_size 1

Originally posted by @LeoDu0314 in #5

感谢，这个问题已经及解决，但是我遇到了新的问题，/(ㄒoㄒ)/~~

我搞了半天实在不知道哪里有问题，有大佬指点一下吗？
@LeoDu0314 @FanqingM @kkkaiaiai

[ZhonghaoLu]

命令脚本

#!/bin/bash
set -x

export RAY_MASTER_PORT=2298
export RAY_DASHBOARD_PORT=8080
export NCCL_TIMEOUT=7200

export NODE_RANK=0

export MASTER_ADDR='1.1.1.1'
export OUTPUT_DIR='/pfs/mt-hiEd6E/home/lzh/Code/vlm-z1/MM-EUREKA/output'
export REWARD_LOG_PATH="${OUTPUT_DIR}/reward.log"
export WORKING_DIR=$PWD


if [ ! -d "$OUTPUT_DIR" ]; then
  mkdir -p "$OUTPUT_DIR"
fi

if [ "$NODE_RANK" -eq 0 ]; then
    ray start --head  --port=$RAY_MASTER_PORT --dashboard-host=0.0.0.0 --dashboard-port=$RAY_DASHBOARD_PORT --num-gpus 8
else
    sleep 30
    ray start --address="$MASTER_ADDR:$RAY_MASTER_PORT" --num-gpus 8 --block
fi

sleep 30


if [ "$NODE_RANK" -eq 0 ]; then
  RAY_ADDRESS="http://127.0.0.1:$RAY_DASHBOARD_PORT" \
  ray job submit \
  --working-dir $WORKING_DIR \
  -- python3 -m openrlhf.cli.train_ppo_ray \
  --remote_rm_url examples/scripts/reward_func_qwen_instruct.py \
  --actor_num_nodes 1 \
  --actor_num_gpus_per_node 8 \
  --vllm_num_engines 8 \
  --vllm_tensor_parallel_size 1 \
  --vllm_gpu_memory_utilization 0.7 \
  --pretrain /pfs/mt-hiEd6E/home/llm_model/Qwen/Qwen2___5-VL-7B-Instruct \
  --save_path ${OUTPUT_DIR} \
  --micro_train_batch_size 2 \
  --train_batch_size 128 \
  --micro_rollout_batch_size 2 \
  --rollout_batch_size 128 \
  --temperature 1.0 \
  --n_samples_per_prompt 8 \
  --lambd 1.0 \
  --gamma 1.0 \
  --max_epochs 1 \
  --num_episodes 10 \
  --prompt_max_len 3000 \
  --max_samples 1000 \
  --generate_max_len 4096 \
  --advantage_estimator group_norm \
  --zero_stage 1 \
  --bf16 \
  --actor_learning_rate 1e-6 \
  --init_kl_coef 0.0 \
  --prompt_data /pfs/mt-hiEd6E/home/lzh/Code/vlm-z1/step4_train_format_qwen.json \
  --disable_fast_tokenizer \
  --input_key message \
  --adam_offload \
  --flash_attn \
  --gradient_checkpointing \
  --save_steps 50 \
  --ckpt_path "${OUTPUT_DIR}/ckpt" \
  --max_ckpt_num 1000000 \
  --save_hf_ckpt \
  --freeze_prefix visual \
  --enable_accuracy_filter \
  --accuracy_lower_bound 0.1 \
  --accuracy_upper_bound 0.9 \
  --use_tensorboard "${OUTPUT_DIR}/tensorboard" \
  --load_checkpoint | tee ${OUTPUT_DIR}/training.log\
  --enforce_eager False
fi

机器配置：
两台A10080G8卡

[LeoDu0314]

不同worker上的$NODE_RANK这个环境变量应该是不同的

但是你这里直接设置了export NODE_RANK=0, 先把这个问题解决一下?

[ZhonghaoLu]

不同worker上的$NODE_RANK这个环境变量应该是不同的

但是你这里直接设置了export NODE_RANK=0, 先把这个问题解决一下?

NODE_RANK的确应该设置不一样，我用了一个比较笨的方法，我在第一台机器上设置export NODE_RANK=0，随后修改为export NODE_RANK=1，再次在第二台机器上执行命令，因为我使用官方给的示例代码：

#!/bin/bash
set -x

export RAY_MASTER_PORT=2298
export RAY_DASHBOARD_PORT=8080
export NCCL_TIMEOUT=7200

export MASTER_ADDR='1.1.1.1'
export OUTPUT_DIR='/pfs/mt-hiEd6E/home/lzh/Code/vlm-z1/MM-EUREKA/output'
export REWARD_LOG_PATH="${OUTPUT_DIR}/reward.log"
export WORKING_DIR=$PWD


if [ ! -d "$OUTPUT_DIR" ]; then
  mkdir -p "$OUTPUT_DIR"
fi

if [ "$PET_NODE_RANK" -eq 0 ]; then
    ray start --head  --port=$RAY_MASTER_PORT --dashboard-host=0.0.0.0 --dashboard-port=$RAY_DASHBOARD_PORT --num-gpus 8
else
    sleep 30
    ray start --address="$MASTER_ADDR:$RAY_MASTER_PORT" --num-gpus 8 --block
fi

sleep 30


if [ "$NODE_RANK" -eq 0 ]; then
  RAY_ADDRESS="http://127.0.0.1:$RAY_DASHBOARD_PORT" \

会出现以下问题：

但是我很确定我的端口是可以部署服务被访问的

[LeoDu0314]

应该一次性在所有机器上都设置正确的NODE_RANK 然后在所有机器上都执行示例脚本

[ZhonghaoLu]

应该一次性在所有机器上都设置正确的NODE_RANK 然后在所有机器上都执行示例脚本

感谢，我已经正确设置，然后在所有机器上都执行示例脚本，but

依旧出现最初始的问题：

Traceback (most recent call last):
  File "/pfs/mt-hiEd6E/home/lzh/anaconda3/envs/eureka/lib/python3.10/runpy.py", line 196, in _run_module_as_main
    return _run_code(code, main_globals, None,
  File "/pfs/mt-hiEd6E/home/lzh/anaconda3/envs/eureka/lib/python3.10/runpy.py", line 86, in _run_code
    exec(code, run_globals)
  File "/tmp/ray/session_2025-04-02_17-09-41_809519_220794/runtime_resources/working_dir_files/_ray_pkg_2c1c27606df0b6cf/openrlhf/cli/train_ppo_ray.py", line 478, in <module>
    train(args)
  File "/tmp/ray/session_2025-04-02_17-09-41_809519_220794/runtime_resources/working_dir_files/_ray_pkg_2c1c27606df0b6cf/openrlhf/cli/train_ppo_ray.py", line 170, in train
    ray.get(refs)
  File "/pfs/mt-hiEd6E/home/lzh/anaconda3/envs/eureka/lib/python3.10/site-packages/ray/_private/auto_init_hook.py", line 21, in auto_init_wrapper
    return fn(*args, **kwargs)
  File "/pfs/mt-hiEd6E/home/lzh/anaconda3/envs/eureka/lib/python3.10/site-packages/ray/_private/client_mode_hook.py", line 103, in wrapper
    return func(*args, **kwargs)
  File "/pfs/mt-hiEd6E/home/lzh/anaconda3/envs/eureka/lib/python3.10/site-packages/ray/_private/worker.py", line 2755, in get
    values, debugger_breakpoint = worker.get_objects(object_refs, timeout=timeout)
  File "/pfs/mt-hiEd6E/home/lzh/anaconda3/envs/eureka/lib/python3.10/site-packages/ray/_private/worker.py", line 908, in get_objects
    raise value
ray.exceptions.ActorDiedError: The actor died unexpectedly before finishing this task.
        class_name: ActorModelRayActor
        actor_id: e1010b1d984dddbce728a14c02000000
        pid: 234166
        namespace: d0b51493-84d9-47ab-bc4c-2022cead77e1
        ip: 192.168.32.6
The actor is dead because its worker process has died. Worker exit type: SYSTEM_ERROR Worker exit detail: Worker unexpectedly exits with a connection error code 2. End of file. There are some potential root causes. (1) The process is killed by SIGKILL by OOM killer due to high memory usage. (2) ray stop --force is called. (3) The worker is crashed unexpectedly due to SIGSEGV or other unexpected errors.
(raylet) A worker died or was killed while executing a task by an unexpected system error. To troubleshoot the problem, check the logs for the dead worker. RayTask ID: ffffffffffffffffe1010b1d984dddbce728a14c02000000 Worker ID: 7a9e6656ba262dd71f42d6078442961f8b9ece0eb8f5339b28e87322 Node ID: cacd4b45df08ec5af58324a3b949d175a86db088be9e0694fe2ec33c Worker IP address: 192.168.32.6 Worker port: 10198 Worker PID: 234166 Worker exit type: SYSTEM_ERROR Worker exit detail: Worker unexpectedly exits with a connection error code 2. End of file. There are some potential root causes. (1) The process is killed by SIGKILL by OOM killer due to high memory usage. (2) ray stop --force is called. (3) The worker is crashed unexpectedly due to SIGSEGV or other unexpected errors.
(raylet) A worker died or was killed while executing a task by an unexpected system error. To troubleshoot the problem, check the logs for the dead worker. RayTask ID: ffffffffffffffff3c02f942099a3fc540240c2c02000000 Worker ID: 3b71cd3848874d82b7d6781d038f24da336696c8f7f351820561a327 Node ID: cacd4b45df08ec5af58324a3b949d175a86db088be9e0694fe2ec33c Worker IP address: 192.168.32.6 Worker port: 10200 Worker PID: 234170 Worker exit type: SYSTEM_ERROR Worker exit detail: Worker unexpectedly exits with a connection error code 2. End of file. There are some potential root causes. (1) The process is killed by SIGKILL by OOM killer due to high memory usage. (2) ray stop --force is called. (3) The worker is crashed unexpectedly due to SIGSEGV or other unexpected errors. [repeated 5x across cluster]
(LLMRayActor pid=2695473, ip=192.168.32.7)

😭

[Observer-Zhj]

请问问题解决了吗？我也碰到了相同的问题

[zihui33]

请问问题解决了吗？我也碰到了相同的问题