Skip to content

Segmentation Fault in Remote Compute Cluster #132

@nomadka

Description

@nomadka

Hello folks, I am looking forward towards the segmentation issue which persist even after trying various versions of carla and python in order to match the compatibility. Below I have added the full traceback of the error. And I am working on remote computing cluster without visual input (Currently logged in Nvidia L40s - 48 GB Graphics card). I appreciate any advice or resolution on similar cases.

`(lmdrive) [es_nokamt00@gpu103 lmdrive]$ nvidia-smi
Sat May 10 13:18:17 2025
+-----------------------------------------------------------------------------------------+
| NVIDIA-SMI 560.35.03 Driver Version: 560.35.03 CUDA Version: 12.6 |
|-----------------------------------------+------------------------+----------------------+
| GPU Name Persistence-M | Bus-Id Disp.A | Volatile Uncorr. ECC |
| Fan Temp Perf Pwr:Usage/Cap | Memory-Usage | GPU-Util Compute M. |
| | | MIG M. |
|=========================================+========================+======================|
| 0 NVIDIA L40S On | 00000000:01:00.0 Off | 0 |
| N/A 39C P8 37W / 350W | 1MiB / 46068MiB | 0% Default |
| | | N/A |
+-----------------------------------------+------------------------+----------------------+

+-----------------------------------------------------------------------------------------+
| Processes: |
| GPU GI CI PID Type Process name GPU Memory |
| ID ID Usage |
|=========================================================================================|
| No running processes found |
+-----------------------------------------------------------------------------------------+
(lmdrive) [es_nokamt00@gpu103 lmdrive]$ python --version
Python 3.9.21
(lmdrive) [es_nokamt00@gpu103 lmdrive]$ pip show carla
Name: carla
Version: 0.9.15
Summary: Python API for communicating with the CARLA server.
Home-page: https://github.com/carla-simulator/carla
Author: The CARLA team
Author-email: carla.simulator@gmail.com
License: MIT License
Location: /home/es/es_es/es_nokamt00/anaconda3/envs/lmdrive/lib/python3.9/site-packages
Requires:
Required-by:
(lmdrive) [es_nokamt00@gpu103 lmdrive]$ export SDL_AUDIODRIVER=dummy
export CARLA_ROOT=/home/es/es_es/es_nokamt00/master_thesis/lmdrive/LMDrive/carla
export TEAM_AGENT=/home/es/es_es/es_nokamt00/master_thesis/lmdrive/LMDrive/leaderboard/team_code/lmdriver_agent.py
export TEAM_CONFIG=/home/es/es_es/es_nokamt00/master_thesis/lmdrive/LMDrive/leaderboard/team_code/lmdriver_config.py
export CHECKPOINT_ENDPOINT=/home/es/es_es/es_nokamt00/master_thesis/lmdrive/LMDrive/results/lmdrive_result.json
export SCENARIOS=/home/es/es_es/es_nokamt00/master_thesis/lmdrive/LMDrive/leaderboard/data/official/all_towns_traffic_scenarios_public.json
export ROUTES=/home/es/es_es/es_nokamt00/master_thesis/lmdrive/LMDrive/langauto/benchmark_long.xml
export PYTHONPATH=/home/es/es_es/es_nokamt00/master_thesis/lmdrive/LMDrive/carla/PythonAPI/carla/dist/carla-0.9.10-py3.10-linux-x86_64.egg:$PYTHONPATH
(lmdrive) [es_nokamt00@gpu103 lmdrive]$ CUDA_VISIBLE_DEVICES=0 /home/es/es_es/es_nokamt00/master_thesis/lmdrive/LMDrive/leaderboard/scripts/run_evaluation.sh
4.24.3-0+++UE4+Release-4.24 518 0
Disabling core dumps.
sh: line 1: xdg-user-dir: command not found
/home/es/es_es/es_nokamt00/master_thesis/lmdrive/LMDrive/leaderboard/leaderboard/leaderboard_evaluator.py
/home/es/es_es/es_nokamt00/master_thesis/lmdrive/LMDrive/leaderboard/leaderboard/leaderboard_evaluator.py:24: DeprecationWarning: pkg_resources is deprecated as an API. See https://setuptools.pypa.io/en/latest/pkg_resources.html
import pkg_resources
/home/es/es_es/es_nokamt00/master_thesis/lmdrive/LMDrive/leaderboard/scripts/run_evaluation.sh: line 45: 2127391 Segmentation fault (core dumped) python3 -u ${LEADERBOARD_ROOT}/leaderboard/leaderboard_evaluator.py --scenarios=${SCENARIOS} --routes=${ROUTES} --repetitions=${REPETITIONS} --track=${CHALLENGE_TRACK_CODENAME} --checkpoint=${CHECKPOINT_ENDPOINT} --agent=${TEAM_AGENT} --agent-config=${TEAM_CONFIG} --debug=${DEBUG_CHALLENGE} --record=${RECORD_PATH} --resume=${RESUME} --port=${PORT} --trafficManagerPort=${TM_PORT}
(lmdrive) [es_nokamt00@gpu103 lmdrive]$ `

Metadata

Metadata

Assignees

No one assigned

    Labels

    No labels
    No labels

    Type

    No type

    Projects

    No projects

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions