Compiling ICAR

Edit the makefile to set the path to your compiled NetCDF and FFTW libraries

Also to set the compiler for your machine if necessary (defaults to gfortran)

make clean
     # remove build by-products

make
     # default (relatively high) optimization compile

make install
    # compile if necessary, then install in the install directory [~/bin]

Options

MODE=fast           # more optimization, slower compile, WARNING:not safe optimizations
MODE=profile        # set profiling options for gnu or intel compilers
MODE=debug          # debug compile with optimizations
MODE=debugslow      # debug compile w/o optimizations
MODE=debugomp       # debug compile with optimizations and OpenMP parallelization
MODE=debugompslow   # debug compile w/o optimizations but with OpenMP parallelization

make doc
# build doxygen documentation in docs/html

make test
    # compiles various test programs (mpdata_test, fftshift_test, and calendar_test)

add -jn to parallelize the compile over n processors

Example

make install MODE=debug -j4  # uses 4 processes to compile in debug mode

Derecho System Specifics

Instructions for building ICAR on Derecho, a supercomputer at NCAR. See instructions for running ICAR on Derecho.

Cray Compiler

module load ncarenv cce cray-mpich netcdf fftw
make -j 4

GNU Compiler

module load gcc ncarenv/23.09 cray-mpich netcdf fftw opencoarrays
COMPILER=gnu make -j 4

Derecho Debugging

When running ICAR on larger domains there is a chance the program will run out of memory available for coarrays. This may be difficult to diagnose because the program will stop without outputting any information but the following steps might help. NOTE: The following debugging information is only for Cray compilers, GNU's OpenCoarrays is not setup to use the SHMEM library.

Set Helpful Debug Variables

Set the following variables, further detail and more variables can be found in the Cray OpenSHMEMX documentation. * SHMEM_MEMINFO_DISPLAY=1 to display information about the job's memory allocation during initialization * SHMEM_ABORT_ON_ERROR=1 and MPICH_ABORT_ON_ERROR=1, these are set when loading the ATP module

Increase Memory Available for Coarrays

The default symmetric heap size on Derecho is 64MB per process. To increase the size set the variable XT_SYMMETRIC_HEAP_SIZE to an integer value with the suffix M for megabyte or G for gigabyte.

export XT_SYMMETRIC_HEAP_SIZE=128M

Further Tools