dm_control cheetah run training stops suddenly #151
Description
Hi, I'm now trying to execute dm_control walker walk, walker run, and cheetah run.
Two walker walk, walker run work fine, however cheetah run fails during training like below...
Failure message
Number of errored trials: 1
+--------------------------+--------------+---------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+
| Trial name | # failures | error file |
|--------------------------+--------------+---------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------|
| id=43fbb_00000-seed=8373 | 4 | /home/acc12468eh/ray_results/dm_control/cheetah/run/2020-09-14T19-51-36-sl-sac/id=43fbb_00000-seed=8373_0_hidden_layer_sizes=(256, 256),preprocessors=({'pixels': {'class_name': 'convnet_preprocessor', 'config'_2020-09-14_19-51-38hsvhe5yt/error.txt |
+--------------------------+--------------+---------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+
Traceback (most recent call last):
File "/home/acc12468eh/miniconda3/envs/softlearning/bin/softlearning", line 11, in <module>
load_entry_point('softlearning', 'console_scripts', 'softlearning')()
File "/home/acc12468eh/softlearning/softlearning/scripts/console_scripts.py", line 207, in main
return cli()
File "/home/acc12468eh/miniconda3/envs/softlearning/lib/python3.7/site-packages/click/core.py", line 764, in __call__
return self.main(*args, **kwargs)
File "/home/acc12468eh/miniconda3/envs/softlearning/lib/python3.7/site-packages/click/core.py", line 717, in main
rv = self.invoke(ctx)
File "/home/acc12468eh/miniconda3/envs/softlearning/lib/python3.7/site-packages/click/core.py", line 1137, in invoke
return _process_result(sub_ctx.command.invoke(sub_ctx))
File "/home/acc12468eh/miniconda3/envs/softlearning/lib/python3.7/site-packages/click/core.py", line 956, in invoke
return ctx.invoke(self.callback, **ctx.params)
File "/home/acc12468eh/miniconda3/envs/softlearning/lib/python3.7/site-packages/click/core.py", line 555, in invoke
return callback(*args, **kwargs)
File "/home/acc12468eh/softlearning/softlearning/scripts/console_scripts.py", line 73, in run_example_local_cmd
return run_example_local(example_module_name, example_argv)
File "/home/acc12468eh/softlearning/examples/instrument.py", line 244, in run_example_local
reuse_actors=True)
File "/home/acc12468eh/miniconda3/envs/softlearning/lib/python3.7/site-packages/ray/tune/tune.py", line 356, in run
raise TuneError("Trials did not complete", incomplete_trials)
ray.tune.error.TuneError: ('Trials did not complete', [id=43fbb_00000-seed=8373])
And I cat the error.txt something like that I found.
(base) [acc12468eh@es2 ~]$ cat /home/acc12468eh/ray_results/dm_control/cheetah/run/2020-09-14T19-51-36-sl-sac/id=43fbb_00000-seed=8373_0_hidden_layer_sizes=(256, 256),preprocessors=({'pixels': {'class_name': 'convnet_preprocessor', 'config'_2020-09-14_19-51-38hsvhe5yt/error.txt
Content of error.txt
-bash: unexpected token `('Thank you.