Compiling ICAR
Edit the makefile to set the path to your compiled NetCDF and FFTW libraries
Also to set the compiler for your machine if necessary (defaults to gfortran)
make clean
# remove build by-products
make
# default (relatively high) optimization compile
make install
# compile if necessary, then install in the install directory [~/bin]
Options
MODE=fast # more optimization, slower compile, WARNING:not safe optimizations
MODE=profile # set profiling options for gnu or intel compilers
MODE=debug # debug compile with optimizations
MODE=debugslow # debug compile w/o optimizations
MODE=debugomp # debug compile with optimizations and OpenMP parallelization
MODE=debugompslow # debug compile w/o optimizations but with OpenMP parallelization
make doc
# build doxygen documentation in docs/html
make test
# compiles various test programs (mpdata_test, fftshift_test, and calendar_test)
add -jn to parallelize the compile over n processors
Example
make install MODE=debug -j4 # uses 4 processes to compile in debug mode
Derecho System Specifics
Instructions for building ICAR on Derecho, a supercomputer at NCAR. See instructions for running ICAR on Derecho.
Cray Compiler
module load ncarenv cce cray-mpich netcdf fftw
make -j 4
GNU Compiler
module load gcc ncarenv/23.09 cray-mpich netcdf fftw opencoarrays
COMPILER=gnu make -j 4
Derecho Debugging
When running ICAR on larger domains there is a chance the program will run out of memory available for coarrays. This may be difficult to diagnose because the program will stop without outputting any information but the following steps might help. NOTE: The following debugging information is only for Cray compilers, GNU's OpenCoarrays is not setup to use the SHMEM library.
Set Helpful Debug Variables
Set the following variables, further detail and more variables can be found in the Cray OpenSHMEMX documentation.
* SHMEM_MEMINFO_DISPLAY=1
to display information about the job's memory allocation during initialization
* SHMEM_ABORT_ON_ERROR=1
and MPICH_ABORT_ON_ERROR=1
, these are set when loading the ATP module
Increase Memory Available for Coarrays
The default symmetric heap size on Derecho is 64MB per process.
To increase the size set the variable XT_SYMMETRIC_HEAP_SIZE
to an integer value with the suffix M
for megabyte or G
for gigabyte.
export XT_SYMMETRIC_HEAP_SIZE=128M
Further Tools
- Derecho debugging and profiling documentation
- Use gdb (with cheatsheet) or gdb4hpc
- Linaro Forge Tools such as DTT or MAP.
- If the program returned
died from signal XYZ
, check the signal error against list shown by$ kill -L
or$ kill -l