CMAS 5.2beta Repository Testing

git clone -b 5.2Beta https://github.com/CMASCenter/EPA-CMAQ.git

Clone this repo into a new working area and run through the user's guide installation and testing

[ ] Add a link to the input and output reference data.

[lizadams]

Tasks for Quickstart Guide: https://github.com/CMASCenter/EPA-CMAQ/blob/5.2Beta/CCTM/docs/User_Manual/CMAQ_OGD_quick_start.md

current instructions indicate to use the branch 5.2, while the branch is currently called 5.2beta. Are we releasing a new branch 5.2, or should we be using 5.2beta in the documentation?
This is the current error:

    git clone -b 5.2 https://github.com/USEPA/CMAQ
    Cloning into 'CMAQ'...
    fatal: Remote branch 5.2 not found in upstream origin

Zac: Yes, use the branch 5.2beta and update the documentation accordingly.

commands on linking the libraries under CMAQ_LIB need to be changed.
cd $CMAQ_LIB

ln –s /usr/local/netcdf netcdf

Really needs to be:

ln –s /usr/local/netcdf/netcdf-4.1.1/Linux86-64-ifort-icc_160/lib lib
ln -s /usr/local/netcdf/netcdf-4.1.1/Linux86-64-ifort-icc_160/include include

@CMASCenter - I am having an issue building icon on killdevil. I am using the same netcdf and ioapi libraries as was being used here: /proj/ie/apps/CMAQ/CMAQ_v5.2Beta_UNC_Testing/lib/x86_64/intel
What module should I be loading?
I've tried
mvapich_intel/11.1
and
mvapich2_intel/11.1

I am getting the following error

/usr/mpi/intel/mvapich-1.2.0/bin/mpif90 -openmp UTILIO_DEFN.o HGRD_DEFN.o IC_PARMS.o CGRID_SPCS.o VGRD_DEFN.o findex.o gc_spc_map.o get_envlist.o icon.o lat_lon.o lr_interp.o lst_spc_map.o ngc_spc_map.o opn_ic_file.o setup_logdev.o subhdomain.o prof_driver.o prof_icout.o prof_vinterp.o -L/nas02/home/l/i/lizadams/CMAQ_TESTING/EPA-CMAQ/lib/x86_64/intel/ioapi/lib -lioapi -L/nas02/home/l/i/lizadams/CMAQ_TESTING/EPA-CMAQ/lib/x86_64/intel/netcdf/lib -lnetcdf -lcurl -o ICON_v52_profile_Linux2_x86_64intel
/nas02/apps/intel-11.1/fortran/lib/intel64/libimf.so: warning: warning: feupdateenv is not implemented and will always fail
/nas02/home/l/i/lizadams/CMAQ_TESTING/EPA-CMAQ/lib/x86_64/intel/ioapi/lib/libioapi.a(desc3.o): In function desc3_': /nas01/depts/ie/cempd/apps/ioapi_31/20151106/ioapi/desc3.F:(.text+0x275): undefined reference to _intel_ssse3_rep_memcpy'
/nas02/home/l/i/lizadams/CMAQ_TESTING/EPA-CMAQ/lib/x86_64/intel/ioapi/lib/libioapi.a(bufint3.o): In function `bufget3':

tried using pgi compiler to build icon using openmpi_pgi/14.1
Makefile generated

/usr/mpi/pgi/openmpi-1.4.2/bin/mpif90 -c -Mfixed -O3 -Mextend -I /nas02/home/l/i/lizadams/CMAQ_TESTING/EPA-CMAQ/lib/x86_64/pgi/ioapi/lib -I /nas02/home/l/i/lizadams/CMAQ_TESTING/EPA-CMAQ/lib/x86_64/pgi/ioapi/src UTILIO_DEFN.F
PGF90-F-0004-Corrupt or Old Module file /nas02/home/l/i/lizadams/CMAQ_TESTING/EPA-CMAQ/lib/x86_64/pgi/ioapi/lib/m3utilio.mod (UTILIO_DEFN.F: 74)
PGF90/x86-64 Linux 14.1-0: compilation aborted
make: *** [UTILIO_DEFN.o] Error 2
ERROR while running make command

Figured out how to successfully run the bldit scripts after finding the correct openmpi pgi module version to load: openmpi_pgi/15.7

run.cctm, getting an undefined LU_Path

#> Windblown dust emissions configuration
if ( $CTM_WB_DUST == 'Y' ) then
   # Input variables for BELD3 Landuse option
   setenv DUST_LU_1 $LUpath/beld3_12CalnexBench_output_a.ncf
   setenv DUST_LU_2 $LUpath/beld4_12CalnexBench_output_tot.ncf
   setenv MODIS_FPAR $LUPath/modis_fpar_lai_12km_20101201_20111231_daily_ioapi.ncf

@CMASCenter - Should we have the default option be CTM_WB_DUST = N for the script?? N
Found the problem, LUPath isn't defined but LUpath is.. fixed it in script.

2nd problem running CCTM, the script is set up to use a CGRID file, rather than the file prepared by the default icon run which is a profile input file:
ICON_v52_profile_CMAQ-BENCHMARK_profile
So either the icon run script needs to be changed, or the input file needs to be provided:
CCTM_D51a_Linux2_x86_64intel_CGRID.CMAQ51-BENCHMARK_20110630

If they are going to use the CGRID input file, then they don't need to run ICON??

-[ ] 3rd problem running CCTM

Value for PROMPTFLAG: F returning FALSE
NLDN_STRIKES :/nas02/home/l/i/lizadams/CMAQ_TESTING/EPA-CMAQ/data/lightning/NLDN.Calnex.20110701.ioapi

 >>--->> WARNING in subroutine OPEN3
 File not available.
 
 Could not open file "NLDN_STRIKES".
 
 *** ERROR ABORT in subroutine LTNG_INIT
 Could not open file "NLDN_STRIKES".

Command exited with non-zero status 2

This may be due to my shortcut of trying to get data from where Zac had files under
/proj/ie/apps/CMAQ/CMAQ_v5.2Beta_UNC_Testing/data
I need to get the tar.gz file by following these instructions:
Dwnloaded the CMAQ reference data from the CMAS Center Software Clearinghouse and copy to $CMAQ_HOME

Note, there doesn't seem to be a way to get a link to the google drive that I can then use the wget command to transfer the DATA.CMAQv5.1.Nov2015.tar.gz file directly to killdevil, rather than downloading to my local machine, and then transferring it to the server. Is there a way to do this???

still getting errors with missing files (resolved them by linking over to Zac's directory
LTNG_Parms_Calnex.ncf
NLDN.Calnex.20110701.ioapi

getting missing dust file
modis_fpar_lai_12km_20101201_20111231_daily_ioapi.ncf
Can't find this under Zac's directories either. (we either need to provide this file or turn the dust option off)
It looks like for the following script in Zac's directory:
/proj/ie/apps/CMAQ/CMAQ_v5.2Beta_UNC_Testing/CCTM/scriptsrun.pgi157.mpi.dust
he used the following file:
CALNEX12_MODIS_GVF_LAI_daily.ioapi
fixing this in the script:
setenv MODIS_FPAR $LUpath/CALNEX12_MODIS_GVF_LAI_daily.ioapi

Issue regarding *.o and log files in the git repository
need to provide instructions to use .gitignore file to ignore all .o and .log files
for more information see: https://git-scm.com/docs/gitignore
added the following to the .gitignore file

**/*.log
**/*.o
**/BLD*
**/*.mod
data
lib

• need to mention that if you are using the option to run on multiple processors, and running on a compute server that uses a batch queue, then you need to submit the job using
  Note, I blindly followed the instructions and submitted the job using
  ./run.cctm
  This seemed to result in it being run on 1 processor, as it was using the interactive mode on killdevil.

I tried using a bsub command from Zac's directory, but am getting the following error:

A daemon (pid 6194) died unexpectedly with status 127 while attempting
to launch so we are aborting.
There may be more information reported by the environment (see above).
This may be because the daemon was unable to find all the needed shared
libraries on the remote node. You may set your LD_LIBRARY_PATH to have the
location of the shared libraries on the remote nodes and this will
automatically be forwarded to the remote nodes.
orterun noticed that the job aborted, but has no info as to the process
that caused that situation.
orterun was unable to cleanly terminate the daemons on the nodes shown
below. Additional manual cleanup may be required - please refer to
the "orte-clean" tool for assistance.
        c-191-51 - daemon did not report back when launched
        c-183-31 - daemon did not report back when launched
        c-194-57 - daemon did not report back when launched
/usr/mpi/pgi/openmpi-1.4.2/bin/orted: error while loading shared libraries: libpgc.so: cannot open shared object file: No such file or directory
Job  /opt/lsf/9.1/linux2.6-glibc2.3-x86_64/bin/openmpi_wrapper /nas02/home/l/i/lizadams/CMAQ_TESTING/EPA-CMAQ/CCTM/scripts/run.cctm

• I tried adding a module load command to the run.cctm script, but it didn't make a difference

Liz, these distribution scripts need to be somewhat generic. There are tweaks that are needed to get them to run on the UNC servers. I intentionally made the scripts and the quick start instructions "vanilla" because explicit instructions for how to get these scripts to work at UNC will certainly not apply everywhere (or anywhere else). I would like to redact most of this content that you've posted about running on the UNC system and the recent edits that you made to the quick start guide because they're specific to UNC and not necessarily applicable to the release package.

Here are are my notes/comments on your items above.

re: "commands on linking the libraries under CMAQ_LIB need to be changed."

Do not change. The generic /usr/local/netcdf is fine. I say in the text that this is just an example directory path. Of course it will need to be changed in the port of the scripts to any actual system.

re: "I am having an issue building icon on killdevil."

See the testing manifest (Table 1) for the list of compilers and UNC modules that I used.

re: the issues with the dust, lightning, and IC data.

I need to pull together the testing input data archive still. All of these files will be in there.

re: the Gitignore command.

I'll check this out. I didn't mean to include any object files or logs in the distribution. I thought the repo that I pushed up there was clean. I will fix this.

re: instructions on using multi-processor and batch scripts.

I don't believe that this is a CMAS responsibility. It's up to the user to figure out how to get the software to run on their system. Instructions that we provide for UNC will not really be applicable to anyone else.

[lizadams]

Hi Zac, Perhaps we can put a link to more explicit instructions somewhere in the OGD.

Chapter 5 in the OGD would be the more detailed instructions. Note that I'm working on an update to the config.cmaq script that will automate/simplify the library installation/linking. More info to come soon.

Input/reference data archives for testing (UNC module: openmpi_pgi/15.7):

/proj/ie/apps/CMAQ/CMAQ_v5.2Beta_UNC_Testing/CMAQv5.2.DATA.tar.gz
/proj/ie/apps/CMAQ/CMAQ_v5.2Beta_UNC_Testing/CMAQv5.2.REF_DATA.tar.gz

Will load to Gdrive when the testing is complete.

[lizadams]

CMAQv5.2.DATA.tar.gz contains everything that is needed to run the benchmark case.
CMAQv5.2.REF_DATA.tar.gz contains all of the output data that can be used to verify a successful run of the CMAQ modeling system for the benchmark case. Can we change the directory name in the CMAQv5.2.REF_DATA.tar.gz from data to data_ref or data_answers, so that if the user untar both to the same location, then they will not overwrite the answers when they do their modeling run.

[lizadams]

Quickstart guide: https://github.com/CMASCenter/EPA-CMAQ/blob/5.2Beta/CCTM/docs/User_Manual/CMAQ_OGD_quick_start.md

The quickstart guide uses the $CMAQ_LIB environment variable before it has been defined by sourcing the config.cmaq.
I think we need to use $CMAQ_HOME/lib rather than $CMAQ_LIB in the documentation where the users are given instructions on where the netcdf and ioapi libraries should be located.

I am not sure what you are asking folks to do here:
4. Check the names of the I/O API and netCDF libraries using the ioapi_lib and netcdf_lib script variables.

Do you want them to use echo to display the environment variable? Here is what I obtained:
echo $ioapi_lib
–lioapi
echo $netcdf_lib
–lnetcdf
echo $mpi
mpi:Undefined variable

Or do you want them to check by using the ls command to list the contents of the directory?

ls $IOAPI/libioapi.a
/nas01/depts/ie/cempd/apps/ioapi_31/20151106/Linux2_x86pg_pgcc_nomp15.7/libioapi.a
ls $NETCDF
bin include lib share
ls $CMAQ_LIB/mpi
bin etc include lib64 openmpi_pgi-1.4.2 share tests

Note, the bldmake command make clean gives an error.
make clean
rm *.o *.mod XEC
rm: cannot remove `XEC': No such file or directory
make: *** [clean] Error 1

Regarding the commands syntax used here, this seems to background the job and write the log to a file.
./run.icon >& icon.log

I like to both see the output from the log as it is being generated, and to save it to a log file, so I prefer to use:
./run.icon |& tee icon.log

Note: I am getting the following error:
ERROR: input file not found: GRIDDESC
/proj/ie/apps/CMAQ/CMAQ_TESTING_Liz/EPA-CMAQ/data/mcip/GRIDDESC

 >>--->> WARNING in subroutine DSCGRID
 Could not open GRIDDESC file
 
 Path name "/proj/ie/apps/CMAQ/CMAQ_TESTING_Liz/EPA-CMAQ/data/mcip/GRIDDESC"

In checking the library files, I'm suggesting that they confirm that the names of the libraries are correct for their system. In particular, the netCDF and MPI libs needs to be changed depending on what library they're using. There are notes in the config.cmaq file in reference to this. I can expand this explanation a bit into the QS guide.

There was a syntax error in the bldmake Makefile. I've corrected and will upload a new file to the repo.

Will update the comments to use "tee" as suggested.

I missed the GRIDDESC file in the DATA tar ball. I'm fixing that now. Also updated the REF_DATa tar ball to use the data_ref directory, per your suggestion.

[lizadams]

Consider adding a QA step in the Quickstart Guide
cd $CMAQ_HOME
setenv AFILE data/cctm/CCTM_v52b_Linux2_x86_64pgi.CONC.CMAQ52b_mpi_20110701
setenv BFILE data_ref/cctm/CCTM_v52b_Linux2_x86_64pgi.CONC.BENCHMARK_20110701
m3diff
grep 'A:B' REPORT

Note, I am seeing differences for the same compiler against the reference data.
Here is the O3 result for the first and second time step.
Date and time 2011182:000000 (0:00:00 July 1, 2011)
A:AFILE/O3 vs B:BFILE/O3 vs (A - B)
MAX @( C, R, L) Min @( C, R, L) Mean Sigma
A 1.30700E+00@(45,100,35) 1.13553E-02@(16, 72, 1) 8.31050E-02 1.33267E-01
B 1.30700E+00@(45,100,35) 1.13553E-02@(16, 72, 1) 8.31050E-02 1.33267E-01
A:B 0.00000E+00@( 1, 0, 0) 0.00000E+00@( 1, 0, 0) 0.00000E+00 0.00000E+00

Date and time 2011182:010000 (1:00:00 July 1, 2011)
A:AFILE/O3 vs B:BFILE/O3 vs (A - B)
MAX @( C, R, L) Min @( C, R, L) Mean Sigma
A 1.29306E+00@(45,100,35) 1.11435E-02@(16, 72, 1) 8.31179E-02 1.32214E-01
B 1.29306E+00@(45,100,35) 1.11434E-02@(16, 72, 1) 8.31179E-02 1.32214E-01
A:B 2.63657E-05@(16, 60, 1) -5.12600E-06@(12, 80,35) 3.90791E-09 1.81359E-07