Running WTF in Cheyenne

[Fergui]

Running WTF in Cheyenne.

TYPE: bug fix, new feature.

KEYWORDS: Run WTF using batch for GNU and Intel, and using screen process for PGI.

SOURCE: Angel Farguell Caus (University of Colorado Denver).

DESCRIPTION OF CHANGES: The wrf-model/WTF repository with branch master was taken and tried to run in Cheyenne. The code had various problems while running on Cheyenne. Maybe this pull request would be suitable for a separate branch like cheyenne branch.

Among other changes:

1. run_from_github.py is updated with the new features in master branch.
2. Updating the scratch path in Cheyenne.
3. Changing the PGI version in order to be able to compile WRF.
4. Adding memory limits for batch tests.
5. New scripts to run batch for each compiler separately.
6. Scripts to run any compiler using screen. This is in case the compiler doesn't work in the batch nodes, like PGI in Cheyenne right now.
7. Update README description with the new changes.
8. Fixing typos when running without batch process.

LIST OF MODIFIED FILES:
M README
A qsub_gnu.pbs
A qsub_intel.pbs
A qsub_pgi.pbs
M qsub_this_to_run_all_compilers.pbs.csh
M regTest_gnu_Cheyenne.wtf
M regTest_gnu_Cheyenne_WRFDA.wtf
M regTest_intel_Cheyenne.wtf
M regTest_intel_Cheyenne_WRFDA.wtf
M regTest_pgi_Cheyenne.wtf
M regTest_pgi_Cheyenne_WRFDA.wtf
A run_all_Cheyenne.csh
M run_from_github.py
M scripts/allCheck.ksh
M scripts/buildWrf.ksh
M scripts/run_WRF_Tests.ksh
M scripts/run_all_for_qsub1.csh
M scripts/run_all_for_qsub2.csh
A scripts/run_gnu.csh
A scripts/run_intel.csh
A scripts/run_pgi.csh
M scripts/testWrf.ksh

TESTS CONDUCTED: Run ./run_from_github.py in Cheyenne, using github repository wrf-model/WRF with branch develop. This runs properly batch jobs for GNU and Intel, and screen process for PGI. The resulting files are:
github_wrf-model_develop_gnu_32_33_34.txt
github_wrf-model_develop_intel_13_14_15.txt
github_wrf-model_develop_pgi_52_53_54.txt

RELEASE NOTE: Once the problem with PGI in Cheyenne is solved, one can change the file run_all_Cheyenne.csh to run qsub qsub_pgi.pbs instead of screen -d -m -S pgi_WTF_v04.08 bash -c 'scripts/run_pgi.csh >& regtest_pgi.out'. Otherwise, if Intel or GNU fail on batch nodes, one could change the qsub instructions to follow the screen command.

This gives the opportunity to decide for each compiler where and how to run the whole process properly. One can as well create a file called run_all_*.csh specifying how to run each compiler, and call it from run_from_github.py.

[Fergui]

In the new version of the code, one can compile and run WRF using all the compilers always using batch jobs (not using screen processes). In order to solve the previous problem, the login-node environment is used in the PGI batch job. However, one can only run one job in such a node at the same time. Therefore, the PGI compilations still have to be done in serial mode because of a library problem in the batch nodes.

The new version is tested again using ./run_from_github.py on Cheyenne with the following results:
github_wrf-model_develop_gnu_32_33_34.2019-02-28_11:31:35.txt
github_wrf-model_develop_intel_13_14_15.2019-02-28_11:31:43.txt
github_wrf-model_develop_pgi_52_53_54.2019-02-28_15:27:32.txt