Skip to content

Commit 86cb561

Browse files
lukasdumasiuslukasdumasius
authored
Merge pull request #223 from krishauser/sensor_fusion_tianyu
Working Sensor Fusion Code
2 parents 6d04639 + 7542f05 commit 86cb561

24 files changed

Lines changed: 3515 additions & 37 deletions

‎GEMstack/knowledge/detection/bevfusion_instructions.md‎

Lines changed: 146 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,146 @@
1+
# BEVFusion Set Up Instructions
2+
These instructions were tested on T4 g4dn.xlarge AWS instances with Arara Ubuntu 20.04 DCV images.
3+
4+
## Set Up Instructions for Cuda 11.3
5+
### Set Up your Nvida Driver
6+
```
7+
sudo apt-get update
8+
sudo apt-get install -y ubuntu-drivers-common
9+
ubuntu-drivers devices
10+
sudo apt-get install -y nvidia-driver-565
11+
sudo reboot
12+
```
13+
14+
### Check to make sure that your nvidia driver was set up correctly:
15+
```
16+
nvidia-smi
17+
```
18+
19+
### Install Cuda 11.3
20+
```
21+
wget https://developer.download.nvidia.com/compute/cuda/repos/ubuntu2004/x86_64/cuda-ubuntu2004.pin
22+
sudo mv cuda-ubuntu2004.pin /etc/apt/preferences.d/cuda-repository-pin-600
23+
wget https://developer.download.nvidia.com/compute/cuda/11.3.0/local_installers/cuda-repo-ubuntu2004-11-3-local_11.3.0-465.19.01-1_amd64.deb
24+
sudo dpkg -i cuda-repo-ubuntu2004-11-3-local_11.3.0-465.19.01-1_amd64.deb # This never copy pastes right. Just manually type
25+
sudo apt-key add /var/cuda-repo-ubuntu2004-11-3-local/7fa2af80.pub
26+
sudo apt-get update
27+
sudo apt-get -y install cuda-11-3
28+
```
29+
30+
### Manually modify your bashrc file to include Cuda 11.3
31+
```
32+
sudo nano ~/.bashrc
33+
```
34+
35+
Add the next 2 lines to the bottom of the file:
36+
```
37+
export PATH=/usr/local/cuda-11.3/bin:$PATH
38+
export LD_LIBRARY_PATH=/usr/local/cuda-11.3/lib64:$LD_LIBRARY_PATH
39+
```
40+
41+
Ensure you source your bashrc file:
42+
```
43+
source ~/.bashrc
44+
nvidia-smi
45+
```
46+
47+
### Set Up Miniconda
48+
```
49+
wget https://repo.anaconda.com/miniconda/Miniconda3-latest-Linux-x86_64.sh
50+
bash ~/Miniconda3-latest-Linux-x86_64.sh
51+
source ~/.bashrc
52+
```
53+
54+
### Set Up mmdetection3d:
55+
```
56+
conda create --name openmmlab python=3.8 -y
57+
conda activate openmmlab
58+
conda install pytorch=1.10.2 torchvision=0.11.3 cudatoolkit=11.3 -c pytorch
59+
pip install -U openmim
60+
mim install mmengine
61+
mim install 'mmcv>=2.0.0rc4, <2.2.0'
62+
mim install 'mmdet>=3.0.0,<3.3.0'
63+
pip install cumm-cu113
64+
pip install spconv-cu113
65+
git clone https://github.com/open-mmlab/mmdetection3d.git -b dev-1.x
66+
cd mmdetection3d
67+
pip install -v -e .
68+
python projects/BEVFusion/setup.py develop
69+
```
70+
71+
### Run this afterwards to verify BEVFusion has been set up correctly:
72+
```
73+
python projects/BEVFusion/demo/multi_modality_demo.py demo/data/nuscenes/n015-2018-07-24-11-22-45+0800__LIDAR_TOP__1532402927647951.pcd.bin demo/data/nuscenes/ demo/data/nuscenes/n015-2018-07-24-11-22-45+0800.pkl projects/BEVFusion/configs/bevfusion_lidar-cam_voxel0075_second_secfpn_8xb4-cyclic-20e_nus-3d.py ~/Downloads/bevfusion_lidar-cam_voxel0075_second_secfpn_8xb4-cyclic-20e_nus-3d-5239b1af.pth --cam-type all --score-thr 0.2 --show
74+
```
75+
76+
## Set Up Instructions for Cuda 11.1
77+
### Set Up your Nvida Driver
78+
```
79+
sudo apt-get update
80+
sudo apt-get install -y ubuntu-drivers-common
81+
ubuntu-drivers devices
82+
sudo apt-get install -y nvidia-driver-565
83+
sudo reboot
84+
```
85+
86+
### Check to make sure that your nvidia driver was set up correctly:
87+
```
88+
nvidia-smi
89+
```
90+
91+
### Install Cuda 11.3
92+
```
93+
wget https://developer.download.nvidia.com/compute/cuda/repos/ubuntu2004/x86_64/cuda-ubuntu2004.pin
94+
sudo mv cuda-ubuntu2004.pin /etc/apt/preferences.d/cuda-repository-pin-600
95+
wget https://developer.download.nvidia.com/compute/cuda/11.1.0/local_installers/cuda-repo-ubuntu2004-11-1-local_11.1.0-455.23.05-1_amd64.deb
96+
sudo dpkg -i cuda-repo-ubuntu2004-11-1-local_11.1.0-455.23.05-1_amd64.deb
97+
sudo apt-key add /var/cuda-repo-ubuntu2004-11-1-local/7fa2af80.pub
98+
sudo apt-get update
99+
sudo apt-get -y install cuda-11-1
100+
```
101+
102+
### Manually modify your bashrc file to include Cuda 11.3
103+
```
104+
sudo nano ~/.bashrc
105+
```
106+
107+
Add the next 2 lines to the bottom of the file:
108+
```
109+
export PATH=/usr/local/cuda-11.1/bin:$PATH
110+
export LD_LIBRARY_PATH=/usr/local/cuda-11.1/lib64:$LD_LIBRARY_PATH
111+
```
112+
113+
Ensure you source your bashrc file:
114+
```
115+
source ~/.bashrc
116+
nvidia-smi
117+
```
118+
119+
### Set Up Miniconda
120+
```
121+
wget https://repo.anaconda.com/miniconda/Miniconda3-latest-Linux-x86_64.sh
122+
bash ~/Miniconda3-latest-Linux-x86_64.sh
123+
source ~/.bashrc
124+
```
125+
126+
### Set Up mmdetection3d:
127+
```
128+
conda create --name openmmlab python=3.8 -y
129+
conda activate openmmlab
130+
pip install torch==1.10.0+cu111 torchvision==0.11.1+cu111 -f https://download.pytorch.org/whl/torch_stable.html
131+
pip install -U openmim
132+
mim install mmengine
133+
mim install 'mmcv>=2.0.0rc4, <2.2.0'
134+
mim install 'mmdet>=3.0.0,<3.3.0'
135+
pip install cumm-cu111
136+
pip install spconv-cu111
137+
git clone https://github.com/open-mmlab/mmdetection3d.git -b dev-1.x
138+
cd mmdetection3d
139+
pip install -v -e .
140+
python projects/BEVFusion/setup.py develop
141+
```
142+
143+
### Run this afterwards to verify BEVFusion has been set up correctly:
144+
```
145+
python projects/BEVFusion/demo/multi_modality_demo.py demo/data/nuscenes/n015-2018-07-24-11-22-45+0800__LIDAR_TOP__1532402927647951.pcd.bin demo/data/nuscenes/ demo/data/nuscenes/n015-2018-07-24-11-22-45+0800.pkl projects/BEVFusion/configs/bevfusion_lidar-cam_voxel0075_second_secfpn_8xb4-cyclic-20e_nus-3d.py ~/Downloads/bevfusion_lidar-cam_voxel0075_second_secfpn_8xb4-cyclic-20e_nus-3d-5239b1af.pth --cam-type all --score-thr 0.2 --show
146+
```

‎GEMstack/onboard/perception/README.md‎

Lines changed: 113 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,113 @@
1+
# Perception Team
2+
This folder contains code that is used to detect objects in 3D space and subsequently notify other GEMstack components of the detected objects. It is split up into 3 main areas: Pedestrian detection code created at the beginning-middle of the course, Cone Detection code that is used to detect cones (optionally with orientation), and Sensor Fusion code which fuses YOLO + painted lidar data and PointPillars 3D bounding boxes to detect pedestrians.
3+
4+
## Cone Detection
5+
A YOLO model was trained to detect the orientations of traffic cones in 3D space to support the Vertical Groups.
6+
7+
### Relevant Files
8+
- cone_detection.py
9+
- perception_utils.py
10+
11+
12+
## Sensor Fusion
13+
To improve the quality of the detected pedestrians, we decided to fuse detections from multiple modalities to take advantage of the strengths each sensor (camera and lidar in our case) provides. We accomplished this by fusing the 3D bounding box detections of pedestrians generated by YOLO (model which detects pedestrians with camera data) + painted lidar data and PointPillars (model which detects pedestrians with only lidar data).
14+
15+
### Relevant Files
16+
#### Setup files to create PointPillars Cuda 11.1.1 Docker container:
17+
- build_point_pillars.sh
18+
- setup/docker-compose.yaml
19+
- setup/Dockerfile.cuda111
20+
21+
#### Code used to detect pedestrians:
22+
- combined_detection.py
23+
- point_pillars_node.py
24+
- yolo_node.py
25+
26+
#### Code used to analyze the results of detections and extract data from rosbags for further analysis:
27+
- eval_3d_bbox_performance.py
28+
- rosbag_processor.py
29+
- test_eval_3d_bbox_performance.py
30+
31+
### Local Installation Steps for PointPillars Docker Container
32+
#### READ BEFOREHAND:
33+
- Before perfoming installation steps, please make sure you source ALL terminal windows (except for docker terminal window).
34+
```
35+
source /opt/ros/noetic/setup.bash
36+
source ~/catkin_ws/devel/setup.bash
37+
```
38+
- These instructions were written with the assumption that you are running them inside of the outermost GEMstack folder.
39+
- If you have set up issues please read the "Set Up Issues Known Fixes" section at the bottom.
40+
41+
#### Steps:
42+
1. Install Docker
43+
2. Install Docker Compose
44+
3. A bash script was created to handle docker permissions issues and make the set up process simpler:
45+
```
46+
cd GEMstack/onboard/perception
47+
bash build_point_pillars.sh
48+
```
49+
4. Start the container (use sudo if you run into permissions issues)
50+
```
51+
docker compose -f setup/docker-compose.yaml up
52+
```
53+
5. Run roscore on local machine (make sure you source first)
54+
```
55+
roscore
56+
```
57+
6. Start up YOLO node (make sure you source first):
58+
```
59+
python3 GEMstack/onboard/perception/yolo_node.py
60+
```
61+
7. Run yaml file to start up the CombinedDetector3D GEMstack Component (make sure you source first):
62+
```
63+
python3 main.py --variant=detector_only launch/combined_detection.yaml
64+
```
65+
8. Run a rosbag on a loop (make sure you source first):
66+
```
67+
rosbag play -l yourRosbagNameGoesHere.bag
68+
```
69+
70+
### Vehicle Installation Steps for PointPillars Docker Container
71+
Perform the same setup steps as the above section with the below exceptions:
72+
1. Ensure you source instead with the following command:
73+
```
74+
source ~/demo_ws/devel/setup.bash
75+
```
76+
2. Initialize the sensors:
77+
```
78+
roslaunch basic_launch sensor_init.launch
79+
```
80+
3. Initialize GNSS (if you need it)
81+
```
82+
roslaunch basic_launch visualization.launch
83+
```
84+
4. Do not run a rosbag in Step 8 above (it's not needed since you'll be getting live data from the vehicle)
85+
86+
#### Known Fixes for Set Up Issues
87+
1. If you get a shape error when creating the "results_normal" variable in yolo_node.py, please downgrade your Ultralytics version to 8.1.5 (this is the version used on the car at the time of writing this):
88+
```
89+
pip install 'ultralytics==8.1.5'
90+
```
91+
2. If you run into communication issues with ROS, please make sure you have sourced EVERY terminal window (except for docker window there's no need to):
92+
```
93+
source /opt/ros/noetic/setup.bash
94+
source ~/catkin_ws/devel/setup.bash
95+
```
96+
97+
### Visualization Steps:
98+
Please make sure you source each new terminal window after creating it (local source commands are below):
99+
```
100+
source /opt/ros/noetic/setup.bash
101+
source ~/catkin_ws/devel/setup.bash
102+
```
103+
104+
1. Start rviz:
105+
```
106+
rviz
107+
```
108+
2. Publish a static transform from the map to visualize the published bounding box data:
109+
```
110+
rosrun tf2_ros static_transform_publisher 0 0 0 0 0 0 map currentVehicleFrame
111+
```
112+
3. In Rviz, click "add" in the bottom left corner. In "By display type", under "jsk_rviz_plugins" select BoundingBoxArray.
113+
4. Expand BoundingBoxArray on the left. Under it you will see "Topic" with a blank space to the right of it. Click the blank space (it's a hidden drop down box) and select the BoundingBoxArray topic to visualize

‎GEMstack/onboard/perception/build_point_pillars.sh‎

Lines changed: 49 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,49 @@
1+
#!/bin/bash
2+
3+
# Check if the point_pillars_node.py, the helper functions file, and model weights exist
4+
if [ ! -f "point_pillars_node.py" ]; then
5+
echo "ERROR: point_pillars_node.py not found in the current directory!"
6+
echo "Please place your point_pillars_node.py file in the same directory as this script."
7+
exit 1
8+
fi
9+
10+
if [ ! -f "combined_detection_utils.py" ]; then
11+
echo "ERROR: combined_detection_utils.py not found in the current directory!"
12+
echo "Please place your combined_detection_utils.py file in the same directory as this script."
13+
exit 1
14+
fi
15+
16+
if [ ! -f "epoch_160.pth" ]; then
17+
echo "WARNING: epoch_160.pth model weights not found in the current directory!"
18+
echo "Please place your model weights file in the same directory as this script."
19+
echo "Continue anyway? (y/n)"
20+
read -p ">" choice
21+
if [ "$choice" != "y" ] && [ "$choice" != "Y" ]; then
22+
exit 1
23+
fi
24+
fi
25+
26+
echo "Building Point Pillars Docker container..."
27+
export DOCKERFILE=setup/Dockerfile.cuda111
28+
29+
# Using sudo to handle permissions
30+
MY_UID=$(id -u)
31+
MY_GID=$(id -g)
32+
33+
# Attempt to use docker-compose directly, then with sudo if needed (if you uncomment it)
34+
if ! docker compose -f setup/docker-compose.yaml build; then
35+
echo "Uncomment these lines if you wish to use sudo to build container as backup"
36+
# echo "Using sudo to build the container..."
37+
# sudo -E docker compose -f setup/docker-compose.yaml build
38+
fi
39+
40+
# Notify user of how to run the container
41+
echo "Build complete. To start the container, run:"
42+
echo "docker compose -f setup/docker-compose.yaml up"
43+
echo ""
44+
echo "Or with sudo if you have permission issues:"
45+
echo "sudo docker compose -f setup/docker-compose.yaml up"
46+
echo ""
47+
echo "To run in detached mode (background):"
48+
echo "docker compose -f setup/docker-compose.yaml up -d"
49+
echo "Or: sudo docker compose -f setup/docker-compose.yaml up -d"

0 commit comments

Comments
 (0)