update LICENSE. Update README with our results. We set a all_suites task to evaluate on all benchmarks. We had a link to the freezing issue when importing carla and torch. We removed the hack to destuck agent. We replace all print/exit with raise ValueError

Author: marintoro

LICENSE

@@ -1,6 +1,6 @@
 MIT License
 
-Copyright (c) 2019 Dian Chen
+Copyright (c) 2020 Marin Toromanoff
 
 Permission is hereby granted, free of charge, to any person obtaining a copy
 of this software and associated documentation files (the "Software"), to deal

README.md

@@ -7,7 +7,7 @@ from which we just kept all the code related to the evaluation on the standard C
 benchmark and on the new released No-Crash benchmark.
 
 ### Installation
-We provide a script here to install and download our weights:
+We provide a script to install every dependencies needed and download our weights.
 
 ```bash
 # Download CARLA 0.9.6
@@ -29,9 +29,9 @@ mv Town*.bin CarlaUE4/Content/Carla/Maps/Nav/
 conda env create -f environment.yml
 conda activate carla_RL_IAs
 # BE CAREFUL: you need to install pytorch according to your cuda version
-#conda install pytorch torchvision cudatoolkit=10.1 -c pytorch 
+conda install pytorch torchvision cudatoolkit=10.1 -c pytorch 
 #conda install pytorch torchvision cudatoolkit=10.0 -c pytorch
-conda install pytorch==1.1.0 torchvision==0.3.0 cudatoolkit=9.0 -c pytorch
+#conda install pytorch==1.1.0 torchvision==0.3.0 cudatoolkit=9.0 -c pytorch
 pip install --upgrade pip
 pip install pygame
 
@@ -41,15 +41,15 @@ rm carla-0.9.6-py3.5-linux-x86_64.egg
 wget http://www.cs.utexas.edu/~dchen/lbc_release/egg/carla-0.9.6-py3.5-linux-x86_64.egg
 easy_install carla-0.9.6-py3.5-linux-x86_64.egg
 
-# Download model checkpoints trained only on Town01
+# Download model checkpoints trained only on Town01/training weathers
 wget https://github.com/marintoro/LearningByCheating/releases/download/v1.0/model_RL_IAs_only_town01_train_weather.zip
 unzip model_RL_IAs_only_town01_train_weather.zip
 
-
 # Download model checkpoints used for CARLA challenge
 cd ../../..
 wget https://github.com/marintoro/LearningByCheating/releases/download/v1.0/model_RL_IAs_CARLA_Challenge.zip
 unzip model_RL_IAs_CARLA_Challenge.zip
+
 ```
 
 Then, open up a terminal, inside the carla directory run `./CarlaUE4.sh -fps=10 -benchmark`. 
@@ -61,49 +61,63 @@ If you want to see our model used for the CARLA challenge you need to run instea
 `python benchmark_agent.py --suite=town2 
 --max-run 100 --path-folder-model model_RL_IAs_CARLA_Challenge/ --render` 
 
+Note that the model we used for the CARLA challenge was trained on a way harder task and on another 
+version of CARLA so the results on the benchmark are lower. On the other hand it handles all towns,
+including Town03, Town04 and Town05 with US traffic lights!
+
 
 ## Detailed Benchmark Results
 ### Only town01/train weather
+
 ```
-╔Performance of autopilot════════════╦═════════╦═══════╗
-║ Suite Name          ║ Success Rate ║ Total   ║ Seeds ║
-╠═════════════════════╬══════════════╬═════════╬═══════╣
-║ FullTown01-v1       ║ 100          ║ 100/100 ║ 2019  ║
-║ FullTown01-v2       ║ 100          ║ 50/50   ║ 2019  ║
-║ FullTown01-v3       ║ 100          ║ 100/100 ║ 2019  ║
-║ FullTown01-v4       ║ 100          ║ 50/50   ║ 2019  ║
-║ FullTown02-v1       ║ 99           ║ 99/100  ║ 2019  ║
-║ FullTown02-v2       ║ 96           ║ 48/50   ║ 2019  ║
-║ FullTown02-v3       ║ 98           ║ 98/100  ║ 2019  ║
-║ FullTown02-v4       ║ 92           ║ 46/50   ║ 2019  ║
-║ NoCrashTown01-v1    ║ 100          ║ 100/100 ║ 2019  ║
-║ NoCrashTown01-v2    ║ 24           ║ 12/50   ║ 2019  ║
-║ NoCrashTown01-v3    ║ 99           ║ 99/100  ║ 2019  ║
-║ NoCrashTown01-v4    ║ 26           ║ 13/50   ║ 2019  ║
-║ NoCrashTown01-v5    ║ 67           ║ 67/100  ║ 2019  ║
-║ NoCrashTown01-v6    ║ 14           ║ 7/50    ║ 2019  ║
-║ NoCrashTown02-v1    ║ 97           ║ 97/100  ║ 2019  ║
-║ NoCrashTown02-v2    ║ 18           ║ 9/50    ║ 2019  ║
-║ NoCrashTown02-v3    ║ 86           ║ 86/100  ║ 2019  ║
-║ NoCrashTown02-v4    ║ 14           ║ 7/50    ║ 2019  ║
-║ NoCrashTown02-v5    ║ 44           ║ 44/100  ║ 2019  ║
-║ NoCrashTown02-v6    ║ 14           ║ 7/50    ║ 2019  ║
-║ StraightTown01-v1   ║ 100          ║ 100/100 ║ 2019  ║
-║ StraightTown01-v2   ║ 100          ║ 50/50   ║ 2019  ║
-║ StraightTown02-v1   ║ 100          ║ 100/100 ║ 2019  ║
-║ StraightTown02-v2   ║ 100          ║ 50/50   ║ 2019  ║
-║ TurnTown01-v1       ║ 100          ║ 100/100 ║ 2019  ║
-║ TurnTown01-v2       ║ 100          ║ 50/50   ║ 2019  ║
-║ TurnTown02-v1       ║ 99           ║ 99/100  ║ 2019  ║
-║ TurnTown02-v2       ║ 100          ║ 50/50   ║ 2019  ║
-╚═════════════════════╩══════════════╩═════════╩═══════╝
+╔═══════════════════╦══════════════╦═════════╦═══════╗
+║ Suite Name        ║ Success Rate ║ Total   ║ Seeds ║
+╠═══════════════════╬══════════════╬═════════╬═══════╣
+║ FullTown01-v1     ║ 100          ║ 100/100 ║ 2020  ║
+║ FullTown01-v2     ║ 100          ║ 50/50   ║ 2020  ║
+║ FullTown01-v3     ║ 100          ║ 100/100 ║ 2020  ║
+║ FullTown01-v4     ║ 100          ║ 50/50   ║ 2020  ║
+║ FullTown02-v1     ║ 100          ║ 100/100 ║ 2020  ║
+║ FullTown02-v2     ║ 100          ║ 50/50   ║ 2020  ║
+║ FullTown02-v3     ║ 98           ║ 98/100  ║ 2020  ║
+║ FullTown02-v4     ║ 100          ║ 50/50   ║ 2020  ║
+║ NoCrashTown01-v1  ║ 100          ║ 100/100 ║ 2020  ║
+║ NoCrashTown01-v2  ║ 36           ║ 18/50   ║ 2020  ║
+║ NoCrashTown01-v3  ║ 96           ║ 96/100  ║ 2020  ║
+║ NoCrashTown01-v4  ║ 34           ║ 17/50   ║ 2020  ║
+║ NoCrashTown01-v5  ║ 70           ║ 70/100  ║ 2020  ║
+║ NoCrashTown01-v6  ║ 26           ║ 13/50   ║ 2020  ║
+║ NoCrashTown02-v1  ║ 99           ║ 99/100  ║ 2020  ║
+║ NoCrashTown02-v2  ║ 24           ║ 12/50   ║ 2020  ║
+║ NoCrashTown02-v3  ║ 87           ║ 87/100  ║ 2020  ║
+║ NoCrashTown02-v4  ║ 34           ║ 17/50   ║ 2020  ║
+║ NoCrashTown02-v5  ║ 42           ║ 42/100  ║ 2020  ║
+║ NoCrashTown02-v6  ║ 18           ║ 9/50    ║ 2020  ║
+║ StraightTown01-v1 ║ 100          ║ 100/100 ║ 2020  ║
+║ StraightTown01-v2 ║ 100          ║ 50/50   ║ 2020  ║
+║ StraightTown02-v1 ║ 100          ║ 100/100 ║ 2020  ║
+║ StraightTown02-v2 ║ 100          ║ 50/50   ║ 2020  ║
+║ TurnTown01-v1     ║ 100          ║ 100/100 ║ 2020  ║
+║ TurnTown01-v2     ║ 100          ║ 50/50   ║ 2020  ║
+║ TurnTown02-v1     ║ 100          ║ 100/100 ║ 2020  ║
+║ TurnTown02-v2     ║ 100          ║ 50/50   ║ 2020  ║
+╚═══════════════════╩══════════════╩═════════╩═══════╝
+
 ```
 
 ### CARLA Challenge (town02/04/05 with dynamic weather) (ONGOING)
 
+The results there are way below the one above, but note that it was trained on much harder tasks and that 
+it works also on US towns, i.e. Town03, Town04 and Town05.
+
 ```
 COMING SOON
 ```
 
 ## License
-This repo is released under the MIT License (please refer to the LICENSE file for details). Part of the PythonAPI and the map rendering code is borrowed from the official [CARLA](https://github.com/carla-simulator/carla) repo, which is under MIT license. The image augmentation code is borrowed from [Coiltraine](https://github.com/felipecode/coiltraine) which is released under MIT license.
+This repo is released under the MIT License (please refer to the LICENSE file for details).
+Most of the code come from the repository 
+[**Learning by Cheating**](https://github.com/dianchen96/LearningByCheating)
+which is under MIT license.
+Part of the PythonAPI and the map rendering code is borrowed from the official 
+[CARLA](https://github.com/carla-simulator/carla) repo, which is under MIT license.

benchmark/__init__.py

@@ -244,6 +244,30 @@ def _add(suite_name, *args, **kwargs):
 
 
 _aliases = {
+    "all_suite": [
+        "NoCrashTown02-v5", "NoCrashTown02-v6",
+        "NoCrashTown01-v5", "NoCrashTown01-v6",
+        "NoCrashTown02-v3", "NoCrashTown02-v4",
+        "NoCrashTown01-v3", "NoCrashTown01-v4",
+        "NoCrashTown02-v1", "NoCrashTown02-v2",
+        "NoCrashTown01-v1", "NoCrashTown01-v2",
+        "FullTown02-v1",
+        "FullTown02-v2",
+        "FullTown02-v3",
+        "FullTown02-v4",
+        "StraightTown02-v1",
+        "StraightTown02-v2",
+        "TurnTown02-v1",
+        "TurnTown02-v2",
+        "FullTown01-v1",
+        "FullTown01-v2",
+        "FullTown01-v3",
+        "FullTown01-v4",
+        "StraightTown01-v1",
+        "StraightTown01-v2",
+        "TurnTown01-v1",
+        "TurnTown01-v2",
+    ],
     "town1": [
         "FullTown01-v1",
         "FullTown01-v2",

benchmark_agent.py

@@ -1,9 +1,14 @@
 import argparse
 import time
 
-# print("test torchvision bug before!")
-# import torchvision.models as models
-# print("test torchvision bug after!")
+# There is a known issue with carla and torch (or torchvision)
+# https://www.google.com/url?q=https://github.com/carla-simulator
+# /carla/issues/2132&sa=D&source=hangouts&ust=1584205287893000&u
+# sg=AFQjCNHYKEjQVb0NQYvsQd3BNMw0smQbyg
+# That's why we import torchvision there before anything else
+# Note that we found this was freezing when using cuda9.0
+# Probably if you encounter freeze, you should comment next line
+import torchvision.models as models
 from pathlib import Path
 
 from benchmark import make_suite, get_suites, ALL_SUITES
@@ -50,7 +55,7 @@ def run(args, model_path, port, suite, seed, resume, max_run):
     )
     parser.add_argument("--port", type=int, default=2000)
     parser.add_argument("--suite", choices=ALL_SUITES, default="town1")
-    parser.add_argument("--seed", type=int, default=2019)
+    parser.add_argument("--seed", type=int, default=2020)
     parser.add_argument("--resume", action="store_true")
     parser.add_argument("--max-run", type=int, default=100)
 

bird_view/models/agent_IAs_RL.py

@@ -21,16 +21,15 @@ def __init__(self, args=None, **kwargs):
         for file in os.listdir(path_to_model_supervised):
             if ".pth" in file:
                 if path_model_supervised is not None:
-                    print(
-                        "There is multiple model supervised in folder ",
-                        path_to_model_supervised,
+                    raise ValueError(
+                        "There is multiple model supervised in folder " +
+                        path_to_model_supervised +
                         " you must keep only one!",
                     )
-                    exit(1)
                 path_model_supervised = os.path.join(path_to_model_supervised, file)
         if path_model_supervised is None:
-            print("We didn't find any model supervised in folder ", path_to_model_supervised)
-            exit(1)
+            raise ValueError("We didn't find any model supervised in folder " +
+                             path_to_model_supervised)
 
         # All this magic number should match the one used when training supervised...
         model_supervised = Model_Segmentation_Traffic_Light_Supervised(
@@ -54,8 +53,7 @@ def __init__(self, args=None, **kwargs):
                 tab_model.append(os.path.join(path_to_model_RL, file))
 
         if len(tab_model) == 0:
-            print("We didn't find any RL model in folder ", path_to_model_RL)
-            exit(1)
+            raise ValueError("We didn't find any RL model in folder "+ path_to_model_RL)
 
         self.tab_RL_model = []
         for current_model in tab_model:
@@ -109,9 +107,6 @@ def __init__(self, args=None, **kwargs):
 
         self.current_timestep = 0
 
-        self.nb_frame_stuck = 0
-        self.speed_stuck = 5 / 3.6  # 5km/h
-
     def act(self, state_buffer, RL_model):
         speeds = []
         order = state_buffer[-1].order
@@ -225,16 +220,6 @@ def run_step(self, observations):
         else:
             brake = brake / len(tab_action)
 
-        # This is a little hack to "destuck" agent, sometimes the image is
-        # always the same and the agent never ever move...
-        if speed < self.speed_stuck:
-            self.nb_frame_stuck += 1
-        else:
-            self.nb_frame_stuck = 0
-        if self.current_timestep < 15 or self.nb_frame_stuck > 1000:
-            brake = 0
-            throttle = 0.5
-
         control = carla.VehicleControl()
         control.steer = np.clip(steer, -1.0, 1.0)
         control.throttle = np.clip(throttle, 0.0, 1.0)