Christopher Bull

Building and Running XIOS and NEMO4 on PSMN (ENS de Lyon HPC) with partition E5.

2023-11-23T00:30:00+00:00

We take the first step of downloading, compiling and testing XIOS and NEMO4 code on PSMN (ENS de Lyon; Lyon, France). This is a post based on an installation made by Cerasela Calugaru. Thank you Cerasela ! This post is written by Louis Saddier (louis.saddier@ens-lyon.fr), thanks Louis!

Pre-requisites:

PSMN account

Suggested reading:

Installing XIOS

Log-into PSMN into a E5 partition node. Set-up your environment with:

module use /applis/PSMN/debian11/E5/modules/all/
module load netCDF-Fortran/4.6.1-gompi-2023a

Get XIOS :

svn co http://forge.ipsl.jussieu.fr/ioserver/svn/XIOS/trunk XIOS

Now we need to create arch files to indicate the options we want when we will compile XIOS. First we will create a .path file:

cd XIOS/arch/
vi arch-X64_PSMN_E5.path

and write:

NETCDF_INCDIR="-I$NETCDFF_INCDIR -I$NETCDF_INC_DIR"
NETCDF_LIBDIR="-L$NECDFF_LIBDIR -L$NETCDF_LIB_DIR"
NETCDF_LIB="-lnetcdff -lnetcdf"

MPI_INCDIR=""
MPI_LIBDIR=""
MPI_LIB=""

HDF5_INCDIR="-I$HDF5_INC_DIR"
HDF5_LIBDIR="-L$HDF5_LIB_DIR"
HDF5_LIB="-lhdf5_hl -lhdf5 -lhdf5 -lz"

OASIS_INCDIR="-I$PWD/../../oasis3-mct/BLD/build/lib/psmile.MPI1"
OASIS_LIBDIR="-L$PWD/../../oasis3-mct/BLD/lib"
OASIS_LIB="-lpsmile.MPI1 -lscrip -lmct -lmpeu"

Then, we will create a .fcm file:

vi arch-X64_PSMN_E5.fcm

and write:

%CCOMPILER      mpicc
%FCOMPILER      mpif90
%LINKER         mpif90

%BASE_CFLAGS    -w -std=c++11 -D__XIOS_EXCEPTION
%PROD_CFLAGS    -O3 -D BOOST_DISABLE_ASSERTS
%DEV_CFLAGS     -g -traceback
%DEBUG_CFLAGS   -DBZ_DEBUG -g -traceback -fno-inline

%BASE_FFLAGS    -D__NONE__
%PROD_FFLAGS    -O3
%DEV_FFLAGS     -g -O2 -traceback
%DEBUG_FFLAGS   -g -traceback

%BASE_INC       -D__NONE__
%BASE_LD        -lstdc++

%CPP            mpicc -EP
%FPP            cpp -P
%MAKE           gmake

Finally, we will create a .env file:

vi arch-X64_PSMN_E5.env

and then write:

module purge
module use /applis/PSMN/debian11/E5/modules/all/
module load netCDF-Fortran/4.6.1-gompi-2023a
export HDF5_INC_DIR=$HDF5_DIR/include
export HDF5_LIB_DIR=$HDF5_DIR/lib
export NETCDFF_INCDIR=/applis/PSMN/debian11/E5/software/netCDF-Fortran/4.6.1-gompi-2023a/include
export NETCDFF_LIBDIR=/applis/PSMN/debian11/E5/software/netCDF-Fortran/4.6.1-gompi-2023a/lib
export NETCDF_INC_DIR=/applis/PSMN/debian11/E5/software/netCDF/4.9.2-gompi-2023a/include
export NETCDF_LIB_DIR=/applis/PSMN/debian11/E5/software/netCDF/4.9.2-gompi-2023a/lib

Now we will compile XIOS with the options we just created:

cd ..
./make_xios --arch X64_PSMN_E5

To test XIOS:

cd generic_testcase
mpirun -np 4 ../bin/generic_testcase.exe

If no errors appear, XIOS should be well installed.

Installing NEMO

Get NEMO4, in my case I’m using the 4.2.1 version:

git clone --branch 4.2.1 https://forge.nemo-ocean.eu/nemo/nemo.git nemo_4.2.1
cd nemo_4.2.1/

If not already done do:

module use /applis/PSMN/debian11/E5/modules/all/
module load netCDF-Fortran/4.6.1-gompi-2023a

Then:

source XIOS/bin/fcm_env.sh

As in the installation of XIOS, we need to modifiate some arch files that contain Fortran options for the compilation of NEMO:

cd arch
cp arch-linux_gfortran.fcm arch-PSMN_E5.fcm
vi arch-PSMN_E5.fcm

Uncomment and modify these 4 lines:

Note that you need to specify in the last line the path where you installed XIOS. In my case it was in my home folder /home/lsaddier

%HDF5_HOME           $HDF5_DIR
%NCDF_C_HOME         /applis/PSMN/debian11/E5/software/netCDF/4.9.2-gompi-2023a
%NCDF_F_HOME         /applis/PSMN/debian11/E5/software/netCDF-Fortran/4.6.1-gompi-2023a
%XIOS_HOME           /home/lsaddier/XIOS

On the %FCFLAGS line, add a last argument:

%FCFLAGS             -fdefault-real-8 -O3 -funroll-all-loops -fcray-pointer -ffree-line-length-none -fallow-argument-mismatch

Then we are ready to compile NEMO. Let’s try with the GYRE_PISCES configuration:

cd ..
./makenemo -m 'PSMN_E5' -r 'GYRE_PISCES' -n 'MY_GYRE_PISCES'

Finally you can test the execution of NEMO (here with 4 processors):

cd /nemo_4.2.1/cfgs/MY_GYRE_PISCES/EXP00
mpirun -n 4 ./nemo

Building and Running XIOS and NEMO4 on PSMN (ENS de Lyon HPC) with partition E5. was originally published by Christopher Bull at Christopher Bull on November 23, 2023.

Building ROSS025 on NEMO4.2 on ARCHER2.

2023-09-14T00:30:00+00:00

We take the steps of building a regional Ross Sea configuration using NEMO 4.2 on ARCHER2. This post is written by TiPACCs PDRA Alethea Mountford (alethea.mountford@northumbria.ac.uk), thanks Alethea!

Pre-requisites:

Suggested reading:

Create ROSS025 configuration

We can make a new ROSS025 configuration using WED025 as a reference (-r), as below

./makenemo -n ROSS025 -r WED025 -m X86_ARCHER2-Cray_4.2 -j 8

OR create and compile a new configuration from scratch

export CONFIG='ROSS025'
echo “$CONFIG OCE ICE” >> cfgs/ref_cfgs.txt
mkdir cfgs/$CONFIG
echo “bld::tool::fppkeys key_xios key_si3 key_qco key_isf key_nosignedzdero” > cfgs/${CONFIG}/cpp_${CONFIG}.fcm
./makenemo -r ${CONFIG} -m X86_ARCHER2-Cray_4.2 -j 8

And check compile was successful

ls -al ${CONFIG}/BLD/bin/nemo.exe

Pre-processing tools (Nico’s regional tools)

git clone https://github.com/nicojourdain/BUILD_CONFIG_NEMO.git
cd GDW-Fortran/test
vi makefile # check the FC and NETCDF_INC variables

make
./gsw_check # no need to worry about *all* working

cd ../..

Back in the main BUILD_CONFIG_NEMO directory, compile all of the pre-processing tools

vi compile_ALL.sh # check the FC, NETCDF_INC and NETCDF_LIB variables

./compile_ALL.sh

Create a new namelist for the regional configuration (e.g. namelist_ROSS025) and link it to the pre-processing namelist that is read by the processing tools

vi namelist_${CONFIG}
ln -s -v namelist_${CONFIG} namelist_pre

Make sure to adapt the regional namelist (i.e. specify the domain bounds in order to cover the desired region).

The exact domain bounds for the ROSS025 configuration are specified as

! Choose the domain bounds on global grid that is extracted (e.g. eORCA12; WARNING: will be rounded to closest point of the coarse grid if ln_coarse_bdy = true) :
nn_imin_extract       =  199
nn_imax_extract       =  821
nn_jmin_extract       =    2
nn_jmax_extract       =  452

in the namelist_pre.

It is worth noting that all parent files used for the creation of the regional configuration must be the same grid size. Parent files were provided by Pierre Mathiot (pierre.mathiot@univ-grenoble-alpes.fr – thanks Pierre!) either through personal communication (namely the BDY files) or sourced from his global eORCA025 input files. Global files were cropped as follows

ncks -L 1 --cnk_dmn time_counter,1 --cnk_dmn y,200 --cnk_dmn x,200 --cnk_dmn nav_lev,200 -d y,0,500 source_file.nc output_file.nc

Create the bathymetry and coordinates files

This requires filling in the &general and &griddata sections of your namelist_ROSS025 file with the parent bathymetry/coordinate files and specifying the domain bounds

./extract_bathy_coord

Which should create your bathy_meter_${CONFIG}.nc and coordinate_${CONFIG}.nc files

Create the domain_cfg and mesh_mask files

Compile the NEMO DOMAINcfg tool in NEMO/tools/DOMAINcfg and REBUILD_NEMO tool in NEMO/tools/REBUILD_NEMO, i.e.

./maketools -m X86_ARCHER2-Cray_4.2 -n DOMAINcfg
./maketools -m X86_ARCHER2-Cray_4.2 -n REBUILD_NEMO

Note, that this needs to be done in the main NEMO directory (the same as when compiling NEMO itself using ./makenemo)

Create the initial state and atmospheric forcing files

Refer to the detailed instructions written by Nico Jourdain and Chris Bull using the BUILD_CONFIG_NEMO pre-processing tools

Run NEMO

With all of the files in place and the relevant .xml and namelist files, you should be ready to run NEMO. See Building and Running NEMO4 on ARCHER2 full system for a refresher on run scripts (but remember to use X86_ARCHER-Cray_4.2 when compiling and xios_trunk)

Building ROSS025 on NEMO4.2 on ARCHER2. was originally published by Christopher Bull at Christopher Bull on September 14, 2023.

Building WED025 on NEMO4.2 on ARCHER2.

2022-04-07T00:30:00+00:00

We take the steps of downloading and compiling NEMO4.2 code on ARCHER2. This post is written by TiPACCs PDRA Alethea Mountford (alethea.mountford@northumbria.ac.uk), thanks Alethea!

Pre-requisites:

ARCHER account

Suggested reading:

Installing NEMO

Get NEMO4.2.0, note that as of the NEMO4.2.0 release, svn is no longer supported. Further reading on the NEMO site

git clone --branch 4.2.0 https://forge.nemo-ocean.eu/nemo/nemo.git NEMORC

Build NEMO

Log-into ARCHER2. Based on NEMO4 ARCHER2 help page. Set-up your environment with:

module -s restore /work/n01/shared/acc/n01_modules/ucx_env

From: /work/n01/shared/nemo/ARCH/arch-X86_ARCHER2-Cray_4.2.fcm put into your ARCH folder, i.e., /work/n02/n02/asmou/NEMORC/arch/arch-X86_ARCHER2-Cray_4.2.fcm

%NCDF_HOME           $NETCDF_DIR
%HDF5_HOME           $HDF5_DIR
%XIOS_HOME           /work/n01/shared/nemo/xios-trunk
#OASIS_HOME          

%NCDF_INC            -I%NCDF_HOME/include -I%HDF5_HOME/include
%NCDF_LIB            -L%HDF5_HOME/lib -L%NCDF_HOME/lib -lnetcdff -lnetcdf -lhdf5_hl -lhdf5 -lz
%XIOS_INC            -I%XIOS_HOME/inc
%XIOS_LIB            -L%XIOS_HOME/lib -lxios
#OASIS_INC           -I%OASIS_HOME/build/lib/mct -I%OASIS_HOME/build/lib/psmile.MPI1
#OASIS_LIB           -L%OASIS_HOME/lib -lpsmile.MPI1 -lmct -lmpeu -lscrip

%CPP                 cpp -Dkey_nosignedzero
%FC                  ftn
%FCFLAGS             -em -s integer32 -s real64 -O0 -hflex_mp=intolerant -N1023
%FFLAGS              -em -s integer32 -s real64 -O0 -hflex_mp=intolerant -N1023
%LD                  CC
%FPPFLAGS            -P -traditional
%LDFLAGS             -lmpifort_cray
%AR                  ar
%ARFLAGS             -r
%MK                  gmake
%USER_INC            %XIOS_INC %NCDF_INC
%USER_LIB            %XIOS_LIB %NCDF_LIB
#USER_INC            %XIOS_INC %OASIS_INC %NCDF_INC
#USER_LIB            %XIOS_LIB %OASIS_LIB %NCDF_LIB

%CC                  cc -Wl,"--allow-multiple-definition"
%CFLAGS              -O1 -Wl,"--allow-multiple-definition"

Add -J flag to FC_MODSEARCH in /work/n02/n02/asmou/NEMORC/ext/FCM/lib/Fcm/Config.pm, should look like

FC_MODSEARCH => '-J',                # FC flag, specify "module" path

Test compile works, i.e.,

#path to where you cloned NEMO
cd /work/n02/n02/asmou/NEMORC
./makenemo -r WED025 -m X86_ARCHER2-Cray_4.2 -j 16
ls -lah cfgs/WED025/BLD/bin/nemo.exe

If -Dkey_nosignedzero has been added to arch-X86_ARCHER2-Cray_4.2.fcm then key_nosignedzero should have been added to cpp_WED025.fcm, i.e should have:

bld::tool::fppkeys key_xios key_si3 key_qco key_isf key_nosignedzero

If key_nosignedzero is not present in cpp_WED025.fcm, this can be added manually during compile:

./makenemo -r WED025 -m X86_ARCHER2-Cray_4.2 add_key “key_nosignedzero” -j 16

Make sure to link the correct xios server (trunk, not 2.5) in your working directory:

ln -s /work/n01/shared/nemo/xios-trunk/bin/xios_server.exe xios_server.exe

Download the WED025 auxiliary input files

wget https://zenodo.org/record/6817000/files/WED025_demonstrator_forcings.tar.zip?download=1
mv WED025_demonstrator_forcings.tar.zip\?download\=1  WED025_demonstrator_forcings.tar.zip
unzip WED025_demonstrator_forcings.tar.zip
tar -tvf WED025_demonstrator_forcings.tar

For further details, see: https://zenodo.org/record/6817000

Building WED025 on NEMO4.2 on ARCHER2. was originally published by Christopher Bull at Christopher Bull on April 07, 2022.

Building and Running NEMO4 on ARCHER2 full system.

2021-12-01T00:30:00+00:00

We take the first step of downloading, compiling and testing NEMO4 code on ARCHER2 on the full system (December 2021).

Pre-requisites:

ARCHER account

Suggested reading:

NEMO4 ARCHER2 help page

Installing NEMO

Get NEMO4, in my case I’m using a 4.0.4 branch..

svn -r 15194 co https://forge.ipsl.jussieu.fr/nemo/svn/NEMO/branches/UKMO/NEMO_4.0.4_momentum_trends NEMO404_MO

Build NEMO

The following is based on NEMO4 ARCHER2 help page. Log-into ARCHER2 (full system). Set-up your environment with:

module restore
module load cray-hdf5-parallel
module load cray-netcdf-hdf5parallel
module load xpmem
module load perftools-base

Originally from ARCHER2 4c: /work/n01/shared/acc/arch-X86_ARCHER2-Cray.fcm put into your ARCH folder, i.e., /work/n02/n02/chbull/temp/NEMO404_MO/arch/arch-X86_ARCHER2-Cray.fcm

%NCDF_HOME           $NETCDF_DIR
%HDF5_HOME           $HDF5_DIR
%XIOS_HOME           /work/n02/shared/chbull/xios-2.5

%NCDF_INC            -I%NCDF_HOME/include -I%HDF5_HOME/include
%NCDF_LIB            -L%HDF5_HOME/lib -L%NCDF_HOME/lib -lnetcdff -lnetcdf -lhdf5_hl -lhdf5 -lz
%XIOS_INC            -I%XIOS_HOME/inc
%XIOS_LIB            -L%XIOS_HOME/lib -lxios

%CPP	             cpp
%FC                  ftn
%FCFLAGS             -em -s integer32 -s real64 -O1 -hflex_mp=intolerant
%FFLAGS              -em -s integer32 -s real64 -O1 -hflex_mp=intolerant
%LD                  CC -Wl,"--allow-multiple-definition"
%FPPFLAGS            -P -traditional
%LDFLAGS             -lmpifort_cray
%AR                  ar
%ARFLAGS             -r
%MK                  gmake
%USER_INC            %XIOS_INC %NCDF_INC
%USER_LIB            %XIOS_LIB %NCDF_LIB

%CC                  cc
%CFLAGS              -O0

Add -J flag to FC_MODSEARCH in /work/n02/n02/chbull/temp/NEMO404_MO/ext/FCM/lib/Fcm/Config.pm, should look like

FC_MODSEARCH => '-J',                # FC flag, specify "module" path

Add key_nosignedzero to /work/n02/n02/chbull/temp/NEMO404_MO/tests/ISOMIPY/cpp_ISOMIPY.fcm, should have:

chbull@uan01:/work/n02/n02/chbull/temp/NEMO404_MO/tests/ISOMIPY> cat cpp_ISOMIPY.fcm 
 bld::tool::fppkeys   key_iomput key_mpp_mpi key_nosignedzero

Test compile works, i.e.,

cd /work/n02/n02/chbull/temp/NEMO404_MO/
./makenemo -n 'ISOMIPY' -a 'ISOMIP' -m X86_ARCHER2-Cray -j 16
ls -lah tests/ISOMIPY/BLD/bin/nemo.exe
# beer time ?

Run NEMO

We use Andrew’s script to create a template for a run file (saving to something like /work/n02/n02/chbull/temp/NEMO404_MO/tests/ISOMIPY/EXP00/runnemo.sh; which will make sense in a moment)..

#!/bin/bash
#SBATCH --job-name=nemo_test
#SBATCH --time=00:10:00
#SBATCH --nodes=1
#SBATCH --ntasks=24
#SBATCH --account=n02-PROPHET
#SBATCH --partition=standard
#SBATCH --qos=short
# Created by: mkslurm -S 0 -s 0 -m 2 -C  24 -c  2 -t 00:10:00 -a n01 -j nemo_test

module restore
module load cray-hdf5-parallel
module load cray-netcdf-hdf5parallel
module load xpmem
module load perftools-base

export OMP_NUM_THREADS=1
#
cat > myscript_wrapper2.sh << EOFB
#!/bin/ksh
#
set -A map ./xios_server.exe ./nemo
exec_map=( 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 )
#
exec \${map[\${exec_map[\$SLURM_PROCID]}]} 
##
EOFB
chmod u+x ./myscript_wrapper2.sh
#
srun --mem-bind=local --cpu-bind=v,map_cpu:00,0x2,0x4,0x6,0x8,0xa,0xc,0xe,0x10,0x12,0x14,0x16,0x18,0x1a,0x1c,0x1e,0x20,0x22,0x24,0x26,0x28,0x2a,0x2c,0x2e, ./myscript_wrapper2.sh

Submit with

cd /work/n02/n02/chbull/temp/NEMO404_MO/tests/ISOMIPY/EXP00
# nb: https://docs.archer2.ac.uk/archer2-migration/archer2-differences/
# no short flag required anymore..
sbatch /work/n02/n02/chbull/temp/NEMO404_MO/tests/ISOMIPY/EXP00/runnemo.sh
# keep an eye on it with
squeue -u USERNAME
tail -f ocean.output
# second beer ?

Building and Running NEMO4 on ARCHER2 full system. was originally published by Christopher Bull at Christopher Bull on December 01, 2021.

Building and Running NEMO4 on ARCHER2.

2021-02-08T00:30:00+00:00

We take the first step of downloading, compiling and testing NEMO4 code on ARCHER2.

Pre-requisites:

ARCHER account

Suggested reading:

Installing NEMO

Get NEMO4, in my case I’m using a fork of Pierre’s custom build.

This uses the latest stable version as it’s base, namely: NEMO4.0.4.

svn co -r 13653  https://forge.ipsl.jussieu.fr/nemo/svn/NEMO/releases/r4.0/r4.0.4 NEMO4
#optional
#copy code from Pierre into NEMO folder, something like:
#git clone git@github.com:chrisb13/NEMOCFG.git
#cd NEMOCFG
#git reset --hard 7695557b1a2f67bae2ae0ab1989d878a281c1bab
#rm -r /path/to/NEMO4/src
#cp -r /path/from/NEMOCFG/cfgs/eORCA025.L121-OPM006/src /path/to/NEMO4

Build NEMO

Log-into ARCHER2. Based on NEMO4 ARCHER2 help page. Set-up your environment with:

module -s restore /work/n01/shared/acc/n01_modules/ucx_env

From: /work/n01/shared/acc/arch-X86_ARCHER2-Cray.fcm put into your ARCH folder, i.e., /lus/cls01095/work/n02/n02/chbull/nemo/models/NEMO4/arch/arch-X86_ARCHER2-Cray.fcm

%NCDF_HOME           $NETCDF_DIR
%HDF5_HOME           $HDF5_DIR
%XIOS_HOME           /work/n01/shared/acc/xios-2.5

%NCDF_INC            -I%NCDF_HOME/include -I%HDF5_HOME/include
%NCDF_LIB            -L%HDF5_HOME/lib -L%NCDF_HOME/lib -lnetcdff -lnetcdf -lhdf5_hl -lhdf5 -lz
%XIOS_INC            -I%XIOS_HOME/inc
%XIOS_LIB            -L%XIOS_HOME/lib -lxios

%CPP	             cpp
%FC                  ftn
%FCFLAGS             -em -s integer32 -s real64 -O1 -hflex_mp=intolerant
%FFLAGS              -em -s integer32 -s real64 -O1 -hflex_mp=intolerant
%LD                  CC -Wl,"--allow-multiple-definition"
%FPPFLAGS            -P -traditional
%LDFLAGS             -lmpifort_cray
%AR                  ar
%ARFLAGS             -r
%MK                  gmake
%USER_INC            %XIOS_INC %NCDF_INC
%USER_LIB            %XIOS_LIB %NCDF_LIB

%CC                  cc
%CFLAGS              -O0

Add -J flag to FC_MODSEARCH in /lus/cls01095/work/n02/n02/chbull/nemo/models/NEMO4/ext/FCM/lib/Fcm/Config.pm, should look like

FC_MODSEARCH => '-J',                # FC flag, specify "module" path

Add key_nosignedzero to /lus/cls01095/work/n02/n02/chbull/nemo/models/NEMO4/tests/ISOMIPY/cpp_ISOMIPY.fcm, should have:

chbull@uan01:/work/n02/n02/chbull/nemo/models/NEMO4/tests/ISOMIPY> cat cpp_ISOMIPY.fcm 
 bld::tool::fppkeys   key_iomput key_mpp_mpi key_nosignedzero

Test compile works, i.e.,

cd /lus/cls01095/work/n02/n02/chbull/nemo/models/NEMO4
./makenemo -n 'ISOMIPY' -a 'ISOMIP' -m X86_ARCHER2-Cray -j 16
ls -lah tests/ISOMIPY/BLD/bin/nemo.exe
# beer time ?

Run NEMO

We use Andrew’s script to create a template for a run file.

chbull@uan01:/work/n01/shared/acc> ./mkslurm -S 0 -s 0 -C 24
Running: mkslurm -S 0 -s 0 -m  2 -C  24 -c  2 -t 00:10:00 -a n01 -j nemo_test
.
.

With a few trivial changes (saving to something like /lus/cls01095/work/n02/n02/chbull/nemo/models/NEMO4/tests/ISOMIPY/EXP00/runnemo.sh; which will make sense in a moment)..

#!/bin/bash
#SBATCH --job-name=nemo_test
#SBATCH --time=00:10:00
#SBATCH --nodes=1
#SBATCH --ntasks=24
#SBATCH --account=n02
#SBATCH --partition=standard
#SBATCH --qos=short
# Created by: mkslurm -S 0 -s 0 -m 2 -C  24 -c  2 -t 00:10:00 -a n01 -j nemo_test
module -s restore /work/n01/shared/acc/n01_modules/ucx_env
export OMP_NUM_THREADS=1
#
cat > myscript_wrapper2.sh << EOFB
#!/bin/ksh
#
set -A map ./xios_server.exe ./nemo
exec_map=( 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 )
#
exec \${map[\${exec_map[\$SLURM_PROCID]}]} 
##
EOFB
chmod u+x ./myscript_wrapper2.sh
#
srun --mem-bind=local --cpu-bind=v,map_cpu:00,0x2,0x4,0x6,0x8,0xa,0xc,0xe,0x10,0x12,0x14,0x16,0x18,0x1a,0x1c,0x1e,0x20,0x22,0x24,0x26,0x28,0x2a,0x2c,0x2e, ./myscript_wrapper2.sh

Submit with

cd /lus/cls01095/work/n02/n02/chbull/nemo/models/NEMO4/tests/ISOMIPY/EXP00
ln -s /work/n01/shared/acc/xios-2.5 xios_server.exe
sbatch --reservation=shortqos /lus/cls01095/work/n02/n02/chbull/nemo/models/NEMO4/tests/ISOMIPY/EXP00/runnemo.sh
# second beer ?

Build NEMO tools

Compile REBUILD_NEMO (used to recombine outputs at the end of the jobs – see NEMO4/tools/REBUILD_NEMO/README.rst) and DOMAINcfg.

cd NEMO4/tools/
./maketools -h 
./maketools -n REBUILD_NEMO -m X86_ARCHER2-Cray
./maketools -n DOMAINcfg -m X86_ARCHER2-Cray
# to check that everything went fine
ls -lah REBUILD_NEMO/BLD/bin/rebuild_nemo.exe
ls -lah DOMAINcfg/BLD/bin/make_domain_cfg.exe

Building and Running NEMO4 on ARCHER2. was originally published by Christopher Bull at Christopher Bull on February 08, 2021.

Build the inital state and forcing fields for WED12 (Step 5)

2018-05-21T10:30:00+00:00

Building the initial state and forcing fields for WED12.

Suggested reading:

ARCHER fork of Nicolas Jourdain GitHub repository for WED12 configuration, this guide is based on these notes

Build the bathymetry (and ice shelf draft if needed) and coordinates files for the regional domain. Go to your repo, and update the following file for your machine: vi BUILD_CONFIG_NEMO/submit.sh (in the ARCHER case this was unnecessary).

./extract_bathy_coord   ## -> should create the bathy and coordinate files,
                        ##    e.g. bathy_meter_WED12.nc and coordinates_WED12.nc
                        ##    (stored in directory defined as config_dir in the namelist)

Build the mesh mask

Prerequisite: successful compile of NEMO (see Step 3).

Create mesh_mask file.

To do this, you need to run nemo.exe to calculate the exact mesh and mask. Run nemo with nn_msh = 1 in the namelist. No need to have all the input files here, just: xios_files, namelists, coordinates.nc, isf_draft_meter.nc and bathy_meter.nc. It will crash shortly after creating mesh_mask.nc. Details:

mkdir -p /fs2/n02/n02/chbull/nemo/run/create_mesh_mask_wed12
cd /fs2/n02/n02/chbull/nemo/run/create_mesh_mask_wed12
ln -s /fs2/n02/n02/chbull/nemo/bld_configs/input_WED12/coordinates_WED12.nc coordinates.nc
ln -s /fs2/n02/n02/chbull/nemo/bld_configs/input_WED12/bathy_meter_WED12.nc bathy_meter.nc
ln -s /fs2/n02/n02/chbull/nemo/bld_configs/input_WED12/bathy_meter_WED12.nc isf_draft_meter.nc 
ln -s /fs2/n02/n02/chbull/nemo/models/XIOS/bin/xios_server.exe xios_server.exe
ln -s /fs2/n02/n02/chbull/nemo/models/dev_r5518_GO6_package/NEMOGCM/CONFIG/WED12/BLD/bin/nemo.exe nemo.exe
#cp namelist_ref AND namelist_cfg into create_mesh_mask_wed12 folder
#cp namelist_ice_ref AND namelist_ice_cfg into create_mesh_mask_wed12 folder

#grab any xios2 files just so the model will start, e.g.
cp /fs2/n02/n02/chbull/nemo/models/dev_r5518_GO6_package/NEMOGCM/CONFIG/GYRE_XIOS/EXP00/iodef.xml /fs2/n02/n02/chbull/nemo/run/create_mesh_mask_wed12_two/
cp /fs2/n02/n02/chbull/nemo/models/dev_r5518_GO6_package/NEMOGCM/CONFIG/SHARED/field_def.xml /fs2/n02/n02/chbull/nemo/run/create_mesh_mask_wed12_two/
cp /fs2/n02/n02/chbull/nemo/models/dev_r5518_GO6_package/NEMOGCM/CONFIG/GYRE_XIOS/EXP00/file_def.xml /fs2/n02/n02/chbull/nemo/run/create_mesh_mask_wed12_two/
cp /fs2/n02/n02/chbull/nemo/models/dev_r5518_GO6_package/NEMOGCM/CONFIG/GYRE_XIOS/EXP00/domain_def.xml /fs2/n02/n02/chbull/nemo/run/create_mesh_mask_wed12_two/

## NB: set jpni=jpnj=jpnij=0 in the &nammpp section of NEMO's namelist.
##     and use the same &namdom parameters as in the simulation used as BDYs.
qsub run_nemo.sh    ## this will create mesh_mask_${CONFIG}.nc before crashing

In my ARCHER case the run_nemo.sh file looks like:

#!/bin/bash 
#!
#PBS -N PC-eos80
#PBS -l walltime=1:00:00
#PBS -l select=1
#PBS -j oe
#PBS -A n02-bas

export PBS_O_WORKDIR=$(readlink -f $PBS_O_WORKDIR)               
export OMP_NUM_THREADS=1
cd $PBS_O_WORKDIR

aprun -n 2 ./nemo.exe
aprun -n 2 ./xios_server.exe

This should result in mesh_mask_0000.nc and mesh_mask_0001.nc out of this (check that mbathy, misf, isfdraft look sensible). We now need to stick them back together…

cd /fs2/n02/n02/chbull/nemo/run/create_mesh_mask_wed12
#From BUILD_CONFIG_NEMO, make sure NEMOdir is correct
./rebuild_mesh_mask.sh 
mv mesh_mask.nc $WORKDIR/input/nemo_${CONFIG}/mesh_mask_${CONFIG}.nc ## directory defined as config_dir

Create the inital state

Create the initial state (temperature and salinity)

./extract_istate            ## -> should create the initial state for temperature and salinity
                            ##    e.g. dta_temp_WED12.nc and dta_sal_WED12.nc
                            ##    (stored in directory defined as config_dir in the namelist)

Ocean forcing files

Create the lateral boundary conditions (u, v, T, S, SSH, sea-ice thic, sea-ice frac)

./build_coordinates_bdy       ## -> creates the coordinate file for lateral boundaries
                              ##    e.g. coordinates_bdy_WED12.nc

./extract_bdy_gridT           ## -> creates T,S bdy files and store them in a BDY folder
                              ##    itself located in directory defined as config_dir

./extract_bdy_gridU           ## -> creates U   bdy files and store them in a BDY folder
./extract_bdy_gridV           ## -> creates V   bdy files and store them in a BDY folder
./extract_bdy_icemod          ## -> creates ice bdy files and store them in a BDY folder
./extract_bdy_ssh             ## -> creates SSH bdy files and store them in a BDY folder

./concatenate_yearly_BDY.sh   ## Edit this file first.
                              ## -> concatenate the bdy files into yearly files

./extract_bdy_tides           ## if you want to put tidal signals along the BDYs

Interpolate CICE onto NEMO

If necessary, interpolate CICE onto NEMO grid before running ./extract_bdy_icemod (above). We use CDO to do a linear interpoation. Here’s an example:

#hack up a griddes file so the source file looks more like the target..
cdo griddes mi-an854o_1m_19960101_20050130_01_grid_T_toy.nc > new_grid.txt
cdo griddes mi-an854i.1m.1995-01_small_toy.nc > old_grid.txt

#xname/yname as nav_lon/nav_lat (the latter seems to be important to match the target x/y names)
sed -i -e 's/TLON/nav_lon/g' old_grid.txt
sed -i -e 's/TLAT/nav_lat/g' old_grid.txt

#remove area from new_grid.txt, the following will help find it.
#grep –n “[a-d][f-z] new_grid.txt 

#re-write with hacked grid defn
cdo setgrid,old_grid.txt -select,name=aice,hi,hs ../mi-an854i.1m.1995-01.nc ./mi-an854i.1m.1995-01_small_newgriddefn.nc

#Re-mapping, creating the weights first.. 
cdo gennn,new_grid.txt mi-an854i.1m.1995-01_small_newgriddefn.nc weights.nc
cdo remap,new_grid.txt,weights.nc mi-an854i.1m.1995-01_small_newgriddefn.nc mi-an854i.1m.1995-01_small_newgriddefn_remapnn_two.nc

#copy interpolated data to a G07 file and match LIM variable names according to BUILD_CONFIG_NEMO/src/extract_bdy_icemod.f90

Other files (SSS for restoring, runoff, chlorophyll)

./extract_SSS_restoring       ## -> creates SSS files and store them in a SSS folder
./concatenate_yearly_SSS.sh   ## Edit this file first.
                              ## -> concatenate the bdy files into yearly files

./extract_runoff_icebergs     ## -> creates iceberg runoff file
                              ##    (stored in config_dir defined in the namelist)

./extract_chloro              ## -> creates chlorophyll file
                              ##    (stored in config_dir defined in the namelist)

Atmospheric forcing files

Weights for the interpolation of atmospheric forcing

# here it is assumed that you have NEMO downloaded and installed as explained in step #1
cd $WORKDIR/models/MY_NEMO/NEMOGCM/TOOLS
maketools -m XC_ARCHER_INTEL -n WEIGHTS           ## Adapt for any machine different from ARCHER (try makenemo -h)
cd WEIGHTS                                ## See nice README file in this directoy.
cp -p nocsutil/namelist_example_bilin namelist_WEIGHTS_${CONFIG}_bilin ## Edit this namelist

NB: you may have to increase char_len in src/kinds_mod.f90 if you include long path for file names.

Below is an example for namelist_WEIGHTS_${CONFIG}_bilin:

&grid_inputs
    input_file = 'drowned_precip_DFS5.2_y1993.nc'
    nemo_file = 'coordinates_WED12.nc'
    datagrid_file = 'remap_data_grid.nc'
    nemogrid_file = 'remap_nemo_grid.nc'
    method = 'regular'
    input_lon = 'LON'
    input_lat = 'LAT'
    nemo_lon = 'glamt'
    nemo_lat = 'gphit'
    nemo_mask = 'none'
    nemo_mask_value = 10
    input_mask = 'none'
    input_mask_value = 10
/

&remap_inputs
    num_maps = 1
    grid1_file = 'remap_data_grid.nc'
    grid2_file = 'remap_nemo_grid.nc'
    interp_file1 = 'data_nemo_bilin.nc'
    interp_file2 = 'nemo_data_bilin.nc'
    map1_name = 'data to nemo bilin Mapping'
    map2_name = 'nemo to data bilin Mapping'
    map_method = 'bilinear'
    normalize_opt = 'frac'
    output_opt = 'scrip'
    restrict_type = 'latitude'
    num_srch_bins = 90
    luse_grid1_area = .false.
    luse_grid2_area = .false.
/

&interp_inputs
    input_file = "drowned_precip_DFS5.2_y1993.nc"
    interp_file = "data_nemo_bilin.nc"
    input_name = "snow"
    input_start = 1,1,1,1
    input_stride = 1,1,1,1
    input_stop = 0,0,0,1
    input_vars = 'initial_time0_hours'
/

&interp_outputs
    output_file = "snow_orca.nc"
    output_mode = "create"
    output_dims = 'x', 'y', 'time_counter'
    output_scaling = "snow|1.0", "time_counter|86400.0"
    output_name = 'snow'
    output_lon = 'x'
    output_lat = 'y'
    output_vars = 'time_counter'
    output_attributes = 'time_counter|units|seconds since 1995-00-00 00:00:00',
                        'time_counter|calendar|noleap',
                        'snow|units|mm/s'
/

Then, execute these things:

#create links to relevant input files
#coordinates_WED12.nc -> /fs2/n02/n02/chbull/nemo/bld_configs/input_WED12/coordinates_WED12.nc
#drowned_precip_DFS5.2_y1993.nc -> /fs2/n02/n02/chbull/nemo/bld_configs/input_WED12_mathiot_eORCA025-GO7/precip_core_y1978.nc
#snow_orca.nc -> /fs2/n02/n02/chbull/nemo/bld_configs/input_WED12_mathiot_eORCA025-GO7/snow_core_y1970.nc

./scripgrid.exe   ## namelist_${CONFIG}_bilin
./scrip.exe       ## namelist_${CONFIG}_bilin
./scripshape.exe  ## namelist_${CONFIG}_bilin

Then create another namelist: namelist_${CONFIG}_bicub and just change these parameters:

interp_file1 = 'data_nemo_bicubic.nc'
interp_file2 = 'nemo_data_bicubic.nc'
map1_name = 'data to nemo bicubic Mapping'
map2_name = 'nemo to data bicubic Mapping'
map_method = 'bicubic'
interp_file = 'data_nemo_bicubic.nc'
output_file = 'weights_bicubic.nc'

Then execute :

./scrip.exe       ## namelist_${CONFIG}_bicub
./scripshape.exe  ## namelist_${CONFIG}_bicub

At the end of this, you should have: weights_bicubic.nc and weights_bilin.nc.

Build the inital state and forcing fields for WED12 (Step 5) was originally published by Christopher Bull at Christopher Bull on May 21, 2018.

Pre-processing tools for WED12 (Step 4)

2018-05-21T08:30:00+00:00

Building the pre-processing tools

Suggested reading:

ARCHER fork of Nicolas Jourdain GitHub repository for WED12 configuration, this guide is almost verbatim based on these notes

Prepare the pre-processing tools (GSW TOOLBOX and Nico’s regional tools):

First you may need the TEOS10 toolbox to convert from EOS80 to TEOS10. To avoid issues with updates on the GSW-Fortran tools, Nico has cloned the 2016 GSW-Fortran in this repository:

git clone https://github.com/chrisb13/BUILD_CONFIG_NEMO.git
cd GSW-Fortran/test
vi makefile   #in particular fill the FC and NETCDF_INCDIR variables
make
./gsw_check   # to check (some have to work, maybe not all of them)
cd ../..

NB: In case you want to check for updates (not recommended), you can still do: git clone https://github.com/TEOS-10/GSW-Fortran.git

The gsw_data_v3_0.nc file provided in this repository should be fine, but in case you start again from the original file (i.e. from github.com/TEOS-10/GSW-Fortran.git), you will need to modify it to avoid NaN in some places (otherwise, skip the next 4 lines):

./compile.sh remove_NaN_from_gsw_data_v3_0.f90
./remove_NaN_from_gsw_data_v3_0
ncks -x -v ocean_ref,ndepth_ref,deltaSA_ref,SA_ref,SAAR_ref GSW-Fortran/test/gsw_data_v3_0.nc gsw_data_v3_0.nc
ncks -A gsw_data_v3_0_to_be_ncks-A.nc gsw_data_v3_0.nc

Back in root of BUILD_CONFIG_NEMO.git. Edit your own namelist, e.g. namelist_WED12, and link namelist_pre to it:

vi namelist_${CONFIG}
ln -s -v namelist_${CONFIG} namelist_pre  # in BUILD_CONFIG_NEMO dir

Edit compile_ALL.sh, adapt fortran compiler (and maybe netcdf path), then execute it: ./compile_ALL.sh. Create a directory where you will store all the netcdf files created along the following steps, e.g.:

mkdir $WORKDIR/input/nemo_${CONFIG}  # WARNING: this must correspond to the namelist entry "config_dir"

Pre-processing tools for WED12 (Step 4) was originally published by Christopher Bull at Christopher Bull on May 21, 2018.

Building WED12 (Step 3)

2018-05-18T08:30:00+00:00

Starting the WED12 configuration (compiling NEMO and NEMO’s maketools)

Suggested reading:

ARCHER fork of Nicolas Jourdain GitHub repository for WED12 configuration, this guide is based on these notes

Create WED12 configuration

First of all, choose a configuration name, e.g. WED12, AMU12, PERIANT025, etc, and save it, i.e.: export CONFIG='WED12'

Compile for a new configuration, e.g. the WED12 configuration (you can have several ones)

cd MY_NEMO/NEMOGCM/CONFIG
echo "$CONFIG OPA_SRC LIM_SRC_3" >> cfg.txt  ## here for OPA+LIM3 (see cfg.txt for other options)
mkdir $CONFIG

Choose fppkeys prior to compilation (see NEMO documentations or examples in existing configurations, e.g. MY_NEMO/NEMOGCM/CONFIG/AMM12/cpp_AMM12.fcm), and compile, e.g. :

cd /NEMOGCM/CONFIG
echo " bld::tool::fppkeys key_dynldf_c2d key_dynspg_flt key_iomput  key_ldfslp  key_tide key_lim3 key_mpp_mpi key_mpp_rep key_nosignedzero key_trabbl key_traldf_c2d key_vvl key_xios2  key_zdfddm key_zdftke key_zdftmx " > ${CONFIG}/cpp_${CONFIG}.fcm
time ./makenemo -n $CONFIG -m XC_ARCHER_INTEL -j 8 &> compile.log &
tail -f compile.log            ## to follow the compilation
ls -al ${CONFIG}/BLD/bin/nemo.exe  ## to check that everything went fine

NB: if you recompile, it is recommended to remove these directories: rm -rf ${CONFIG}/WORK ${CONFIG}/BLD;cd ..

NEMO maketools

Compile REBUILD_NEMO (used to recombine outputs at the end of the jobs):

cd TOOLS
./maketools -h  ## use same architecture as for NEMO's compilation (e.g. XC_ARCHER_INTEL)
./maketools -m XC_ARCHER_INTEL -n REBUILD_NEMO
ls -al REBUILD_NEMO/BLD/bin/  ## to check that everything went fine
cd ..

Building WED12 (Step 3) was originally published by Christopher Bull at Christopher Bull on May 18, 2018.

Testing NEMO (Step 2)

2018-05-18T00:30:00+00:00

Running an idealised test case configuration with NEMO

Testing NEMO

We will now test NEMO by running a double gyre test case. cd into NEMOGCM/CONFIG and run ./makenemo -n GYRE_XIOS -m XC_ARCHER_INTEL -j 8 (took 121 seconds).

If you have loaded netCDF, HDF5 and installed XIOS correctly this should work. Go into NEMOGCM/CONFIG/GYRE_XIOS/EXP00. Make a link to XIOS by: ln -sf /path/to/xios/bin/xios_server.exe xios_server.exe

This particular branch needs the following namelist tweak (/fs2/n02/n02/chbull/nemo/models/dev_r5518_GO6_package/NEMOGCM/CONFIG/GYRE_XIOS/EXP00/namelist_cfg):

&nameos        !   ocean physical parameters
!-----------------------------------------------------------------------
   nn_eos      =   0       !  type of equation of state and Brunt-Vaisala frequency
   ln_useCT    = .false.   ! use of Conservative Temp. ==> surface CT converted in Pot. Temp. in sbcssm

Make a runscript named “run_gyre.pbs” like this:

#!/bin/bash 
#!
#PBS -N PC-eos80
#PBS -l walltime=1:00:00
#PBS -l select=7
#PBS -j oe
#PBS -A n02-bas

  export PBS_O_WORKDIR=$(readlink -f $PBS_O_WORKDIR)               
  export OMP_NUM_THREADS=1
  cd $PBS_O_WORKDIR

  echo " ";
  OCEANCORES=144
  XIOCORES=3
###############################################################

ulimit -c unlimited
ulimit -s unlimited
        aprun -b -n $XIOCORES -N 3 ./xios_server.exe : -n $OCEANCORES -N 24 ./opa

Run qsub run_gyre.pbs

NEMO will now run on 144 cores (6 nodes) and pass all output to XIOS on 3 cores (1 node) which will glue data from all processors into one file. You’re output should be in: GYRE_5d_0001010100011230_grid*.nc

(Hopefully!) congratulations, you’ve just run NEMO!

Testing NEMO (Step 2) was originally published by Christopher Bull at Christopher Bull on May 18, 2018.

Downloading NEMO and installing XIOS on ARCHER (Step 1)

2018-05-17T00:30:00+00:00

We take the first step of downloading, compiling and testing NEMO code on ARCHER

Pre-requisites:

Suggested reading:

Installing NEMO and XIOS

We will first install XIOS, and then NEMO. If you haven’t done so already, you’ll need to load netCDF and HDF5 libraries as well as Intel compilers. On ARCHER, run:

module load cray-netcdf-hdf5parallel/4.4.1.1 
module load cray-hdf5-parallel/1.10.0.1
module swap PrgEnv-cray PrgEnv-intel

This worked with:

cray-netcdf-hdf5parallel/4.4.1.1
cray-hdf5-parallel/1.10.0.1
PrgEnv-intel/5.2.82

And doesn’t work if you do a module purge first, ARCHER on default currently loads:

chbull@eslogin008:~> module list
Currently Loaded Modulefiles:
  1) modules/3.2.10.6                      12) gni-headers/4.0-1.0502.10859.7.8.ari  23) bolt/0.6
  2) eswrap/1.3.3-1.020200.1280.0          13) xpmem/0.1-2.0502.64982.5.3.ari        24) nano/2.2.6
  3) switch/1.0-1.0502.60522.1.61.ari      14) dvs/2.5_0.9.0-1.0502.2188.1.116.ari   25) leave_time/1.3.0
  4) cce/8.5.8                             15) alps/5.2.4-2.0502.9774.31.11.ari      26) quickstart/1.0
  5) craype-network-aries                  16) rca/1.0.0-2.0502.60530.1.62.ari       27) ack/2.14
  6) craype/2.5.10                         17) atp/2.1.0                             28) xalt/0.6.0
  7) cray-libsci/16.11.1                   18) PrgEnv-cray/5.2.82                    29) openssl/1.1.0g_build1
  8) udreg/2.3.2-1.0502.10518.2.17.ari     19) pbs/12.2.401.141761                   30) curl/7.58.0_build1
  9) ugni/6.0-1.0502.10863.8.29.ari        20) craype-ivybridge                      31) git/2.16.2_build1
 10) pmi/5.0.13                            21) cray-mpich/7.5.5                      32) epcc-tools/8.0
 11) dmapp/7.0.1-1.0502.11080.8.76.ari     22) packages-archer

XIOS

We’ll do XIOS first. The recommended compiler suite for ARCHER is Cray. But as Admiral Ackbar said: “It’s a trap!”. You will need to use Intel!

Get the latest version of XIOS (currently XIOS 2), which is an I/O server to handle reading and writing data in NEMO. Source code and instructions can be found here.

You will need the .fcm, .path, and .env files for XIOS. Get the “arch” files for XIOS at:

/work/n02/n02/shared/baspog/arch-for-xios/

Put them in the “arch” directory in the XIOS folder. If they are not available, they are as follows. The env file, /work/n02/n02/shared/baspog/arch-for-xios/arch-XC30_ARCHER.env:

export HDF5_INC_DIR=${HDF5_DIR}/include
export HDF5_LIB_DIR=${HDF5_DIR}/lib
export NETCDF_INC_DIR=${NETCDF_DIR}/include
export NETCDF_LIB_DIR=${NETCDF_DIR}/lib

The fcm file, /work/n02/n02/shared/baspog/arch-for-xios/arch-XC30_ARCHER.fcm:

################################################################################
###################        Projet xios - xmlioserver       #####################
################################################################################

%CCOMPILER      cc
%FCOMPILER      ftn
%LINKER         CC

%BASE_CFLAGS    -DMPICH_SKIP_MPICXX -h msglevel_4 -h zero -h noparse_templates
%PROD_CFLAGS    -O3 -DBOOST_DISABLE_ASSERTS
%DEV_CFLAGS     -g -O2
%DEBUG_CFLAGS   -g

%BASE_FFLAGS    -warn all -zero
%PROD_FFLAGS    -O3 -fp-model precise -warn all -zero
%DEV_FFLAGS     -g -O2
%DEBUG_FFLAGS   -g

%BASE_INC       -D__NONE__
%BASE_LD        -D__NONE__

%CPP            cpp
%FPP            cpp -P -CC
%MAKE           gmake

The ‘path’ file: /work/n02/n02/shared/baspog/arch-for-xios/arch-XC30_ARCHER.path:

NETCDF_INCDIR="-I $NETCDF_INC_DIR"
NETCDF_LIBDIR='-Wl,"--allow-multiple-definition" -Wl,"-Bstatic" -L $NETCDF_LIB_DIR'
NETCDF_LIB="-lnetcdf -lnetcdff"

MPI_INCDIR=""
MPI_LIBDIR=""
MPI_LIB=""

#HDF5_INCDIR="-I $HDF5_INC_DIR"
HDF5_LIBDIR="-L $HDF5_LIB_DIR"
HDF5_LIB="-lhdf5_hl -lhdf5 -lz"

OASIS_INCDIR=""
OASIS_LIBDIR=""
OASIS_LIB=""
#OASIS_INCDIR="-I$PWD/../../prism/X64/build/lib/psmile.MPI1"
#OASIS_LIBDIR="-L$PWD/../../prism/X64/lib"
#OASIS_LIB="-lpsmile.MPI1 -lmpp_io"

And then install XIOS by running the following command in the XIOS folder.

In my case, all of the above was:

svn co -r 1011 http://forge.ipsl.jussieu.fr/ioserver/svn/XIOS/trunk XIOS     #need to use a tested version of the xios 2
cd /path/to/where/you/cloned/xios/arch
cp /work/n02/n02/shared/baspog/arch-for-xios/* . #copy the arch files; the .fcm, .path and .env files are the same as before
cd ..
chmod +x make_xios 
./make_xios --prod --full --arch XC30_ARCHER  --jobs 8

The build took 1435 seconds (using the above module versions).

NEMO

We are using the G06 Met Office revision, clone it (put your NEMO repository username in):

mkdir -p /fs2/n02/n02/chbull/nemo/models
cd /fs2/n02/n02/chbull/nemo/models
svn --username 'chrisbull' co http://forge.ipsl.jussieu.fr/nemo/svn/branches/UKMO/dev_r5518_GO6_package@9583

Open NEMOGCM/ARCH/arch-XC_ARCHER_INTEL.fcm and make sure the variables “NDF_HOME”, “HDF5_HOME” and “XIOS_HOME” point to the netCDF, HDF, and XIOS directories. netCDF and HDF5 directories can be found by running nc-config --all. XIOS_HOME should be where you installed XIOS.

In my case, this looked like the following “arch” file (compiler settings):

%NCDF_HOME           /opt/cray/netcdf-hdf5parallel/4.4.1.1/INTEL/15.0/
%NCDF_HOME_FORTRAN   /opt/cray/netcdf-hdf5parallel/4.4.1.1/INTEL/15.0/
%HDF5_HOME           /opt/cray/hdf5/1.10.0.1/INTEL/15.0/
%XIOS_HOME           /fs2/n02/n02/chbull/nemo/models/XIOS/

%NCDF_INC            -I%NCDF_HOME/include -I%NCDF_HOME_FORTRAN/include
%NCDF_LIB            -L%NCDF_HOME/lib -lnetcdf -L%NCDF_HOME_FORTRAN/lib -lnetcdff
%XIOS_INC            -I%XIOS_HOME/inc 
%XIOS_LIB            -L%XIOS_HOME/lib -lxios

%CPP                 cpp
%FC                  ftn
%FCFLAGS             -integer-size 32 -real-size 64 -O3 -fp-model precise -zero -fpp -warn all
%FFLAGS              -integer-size 32 -real-size 64 -O3 -fp-model precise -zero -fpp -warn all
%LD                  CC -Wl,"--allow-multiple-definition"
%FPPFLAGS            -P -C -traditional
%LDFLAGS
%AR                  ar 
%ARFLAGS             -r
%MK                  gmake
%USER_INC            %XIOS_INC %NCDF_INC
%USER_LIB            %XIOS_LIB %NCDF_LIB

Put here: /fs2/n02/n02/chbull/nemo/models/dev_r5518_GO6_package/NEMOGCM/ARCH/arch-XC_ARCHER_INTEL.fcm.

Downloading NEMO and installing XIOS on ARCHER (Step 1) was originally published by Christopher Bull at Christopher Bull on May 17, 2018.

Python package for gridding subplots, fast!

2016-03-14T00:30:00+00:00

Plthacks is a Python package that wraps Matplotlib to make common (tedius) tasks faster, this post is about creating a grid of an arbitrary number of plots from an ordered dictionary.

I reguarly find myself wanting to display multiple fields. I find it frustrating how many lines you need in Matplotlib to grid plots such that they have things like:

when sharing an axis removing ticks and labels where appropriate;
displaying one colourbar for each subplot or a global colourbar for all plots;
adjusting subplot size according to the above.

Plthacks.Grid Class is an attempt to make this much faster. Let’s see an example.

This package is under development, this post refers to this SHA of the code.

‘Install’ the package somewhere you’ll import it..

git clone https://github.com/chrisb13/plthacks
cd plthacks
git reset --hard 5b193d89ae018aa8a0822c5d722f4dcad2f345c1

To start, we’ll just create some data and store it in an ordered dictionary called plotdict. We also optionally specify each subplot’s dimension labels and colorbars, called dimlab and colorbars respectively.

import plthacks as plth 

#this package requires numpy and matplotlib
import numpy as np

#these two are part of core python
import collections
import itertools

#place to put the plot, if an output is not specified (when calling Grid Class), then it will return the matplotlib figure object
plotoutputs='/tmp/testplots/'

#we will show how you can specify different colormaps in a minute
colormaps=itertools.cycle(['Accent', 'Dark2', 'Paired', 'Pastel1',\
        'Pastel2', 'Set1', 'Set2', 'Set3'])

#The following three ordered dictionaries are ordered by each subplot where they contain:
#plotdict is the fields to contourf
plotdict=collections.OrderedDict()
#(optional) dimlab is the labels for the x/y axis
dimlab=collections.OrderedDict()
#(optional) colorbars are the colourmaps to use 
colorbars=collections.OrderedDict()

#this will be the title in the box of each subplot, we are making 12 but you can try a different number if you like...
cnts=[str(cnt) for cnt in np.arange(12)+1]

for cnt in cnts:
    #plotdict[cnt]=np.random.rand(200,150)
    plotdict[cnt]=np.random.rand(20,15)
    dimlab[cnt]=('xlabel','ylabel')
    colorbars[cnt]=colormaps.next()

We’ll now show 10 different usages of the Grid class so you get the idea of what it can do.

plth.Grid(plotdict,(4,3),dimlabels=dimlab,sepcbar=True,outputpath=plotoutputs+'001_GridEgDimLabSepcbar.png')

Results in:

Neat right? So you’ll notice we’ve opted for 4 rows and 3 columns with 12 plots but the grid can be whatever we like (I hope!).

We can do a few other things too, such as:

share x-axis/share y-axis
include/exclude dimension labels
include/exclude local or global colourbar(s)
have different colourmaps for each subplot

These options are described in the Grid docstring:

"""
Class to create a gridded set of subplots from a dictionary.

Parameters
----------
pdict: dictionary containing the arrays to plot, titles are taken from the key
pdims: tuple with (rowdim,coldim)
sharex (optional): share all the x axis in the grid
sharey (optional): share all the y axis in the grid
dimlabels (optional): dictionary containing tuples of xlabel and ylabels 
sepcbar (optional): separate colorbars for each subplot (True or False)
globalcbar (optional): a single colorbar for the whole plot (string where the options are: 'True' which will use default colormap or name of matplotlib colormap)
cbars (optional): dictionary containing matplotlib colourbars to use
clevels (optional): integer for the number of contour levels
outputpath (optional): full path of file to put plot in (if left out it won't be created)

Returns
-------

Notes
-------
* If you use sepcbar, then best not to use sharex or sharey.

You’ll notice in the above example we used the optional dimlabels and sepcbar. Here’s some more examples.

plth.Grid(plotdict,(4,3),dimlabels=dimlab,cbars=colorbars,outputpath=plotoutputs+'002_GridEgDimLabcbars.png')

Results in:

plth.Grid(plotdict,(4,3),dimlabels=dimlab,cbars=colorbars,clevels=8,outputpath=plotoutputs+'003_GridEgDimLabclevelscbars.png')

Results in:

plth.Grid(plotdict,(4,3),dimlabels=dimlab,globalcbar='True',clevels=20,sharex=True,sharey=True,outputpath=plotoutputs+'004_GridEgDimLabShareXShareYglobalcbarclevels.png')

Results in:

plth.Grid(plotdict,(4,3),dimlabels=dimlab,globalcbar='Accent',clevels=20,sharex=True,sharey=True,outputpath=plotoutputs+'005_GridEgDimLabShareXShareYglobalcbarAccentclevels.png')

Results in:

plth.Grid(plotdict,(4,3),sharex=True,outputpath=plotoutputs+'006_GridEgShareX.png')

Results in:

plth.Grid(plotdict,(4,3),sharey=True,outputpath=plotoutputs+'007_GridEgSHareY.png')

Results in:

plth.Grid(plotdict,(4,3),sharex=True,sharey=True,outputpath=plotoutputs+'008_GridEgShareXShareY.png')

Results in:

plth.Grid(plotdict,(4,3),dimlabels=dimlab,outputpath=plotoutputs+'009_GridEgDimLab.png')

Results in:

plth.Grid(plotdict,(4,3),sharex=True,sharey=True,dimlabels=dimlab,outputpath=plotoutputs+'010_GridEgShareXShareYDimLab.png')

Results in:

All of these examples are in tests.py here.

Of course, it’s super hard to create a Class that will work in all instances. I’ve written it with the idea that one could just copy the whole Class in main.py and hack it, if the set up described here is inadequate. Similarly, the Class will return all the matplotlib axis (see self.paxis dictionary). Hopefully it’s a useful starting point though. Pull requests most welcome!

Python package for gridding subplots, fast! was originally published by Christopher Bull at Christopher Bull on March 14, 2016.

Python package for one-line creation of movies from netCDF files.

2016-01-22T00:30:00+00:00

Update 10.10.2016. MkMov 0.4 now supports a bunch of other outputs, see usage for details.

MkMov is a Python package that creates a movie from a netCDF file with a single line, interface is command line (no python knowledge needed)!

I reguarly find myself making movies for sanity checks on model runs, other times I’ll want a movie for a talk or perhaps to look at how some diagnostic plot varies over time. Suffice to say a few people have asked me how I create movies from netCDF files, such questions spurred the development of…

MkMov, a python package for making movies. In can be used in two ways:

from a netCDF file
from a list of png files (use –stitch option)

Examples tell a thousand words, here’s a movie (AVISO global allsat madt, output is daily and variable is adt)

Created with a single line command, namely

python mkmov.py 2d adt --lmask -214748 --cmap Set3 -o /srv/ccrc/data42/z3457920/mkmovmovies4/AVISOdt_global_allsat_madt_Set3.mov /srv/ccrc/data42/z3457920/RawData/AVISO/RawData/dt_global_allsat_madt/ftp.aviso.altimetry.fr/global/delayed-time/grids/madt/all-sat-merged/h/*/*.nc &>  /srv/ccrc/data42/z3457920/mkmovmovies4/15.log &

More examples with their respective run commands here. Official docs are here. Contributions are most welcome.

You might be wondering what this tool offers over something like ncview and a screen grabber program.

Pros:

bash scriptable
customisable output (e.g. –clev, –cmap etc)
can take multiple files as input
can wrap ffmpeg to stitch together any png files

Cons:

slow in comparison to ncview
need to install dependencies

Python package for one-line creation of movies from netCDF files. was originally published by Christopher Bull at Christopher Bull on January 22, 2016.

Bash function for super fast loading of netCDF files in Python.

2015-12-02T00:30:00+00:00

I often finding myself wanting basic information about netCDF files and needing to inspect it in Python quickly.

I’ve written a bash function that takes a passed path of a netCDF file, it then writes a short python script and loads it into a ipython console. The console then prints out basic information about the file and all the variable names, ready for you to hack away!

For this bash function you’ll need:

python 2.x
python netCDF4 library
ipython

Here’s the function:

The above is a GitHub gists, feel free to comment/fork/pull! If you’ve found this handy, you might also like my Vim and UltiSnip for similar stuff.

Bash function for super fast loading of netCDF files in Python. was originally published by Christopher Bull at Christopher Bull on December 02, 2015.

Programmatic paper writing with MkPaper

2015-11-25T00:30:00+00:00

MkPaper creates a figure document in 3 + n lines of code, where n is the number of figures you have!

Have you ever wished a machine would write your paper for you?

Wish no more, the future is here!* I’ve created a Python package that will compile a LaTeX or word document of figures. Basically, you feed the package a path to a figure and a caption and then it will create either:

a PDFLaTeX document or a
Word Document

with those figures and captions inserted! I’ve tried to make it as simple and painless to install as possible (just clone the repo’); dependencies are kept to a minimum, specifically:

python 2.x
pdflatex (optional)
python-docx (optional)

Without pdflatex, the package still works (you can create the tex file and not compile). To create a Word document you need python-docx (installable with Anaconda).

To install the package:

git clone https://github.com/chrisb13/mkpaper.git

And then, suppose you want to create a word doc, usage looks like…

import mkpaper as mp

#get a list of your figures in PNG format...
import glob
ifiles=sorted(glob.glob(path_to_pngplots + '*.png' ))
assert(ifiles!=[]),"glob didn't find anything!"

#alternatively, we could be more explicit
#ifile[0]=path_to_doc+'penguin1.png'
#ifile[1]=path_to_doc+'penguin2.png'

#instantiate WordFigureDoc class:
#arguments are: path/to/put/figuredocs and name of figure doc..
figobj=mp.WordFigureDoc(path_to_doc,'penguinfigs')
figobj.add_figure(ifiles[0],'first penguin')
figobj.add_figure(ifiles[1],'second penguin')
#etc

#insert the tail of the doc
figobj.end_doc()

Want to get started? Head over to the GitHub project here.. Full working examples in here, so you should be up and running in no time!

Feedback and pull requests welcome!

*A touch of hyperbole here ;)

Programmatic paper writing with MkPaper was originally published by Christopher Bull at Christopher Bull on November 25, 2015.

Vim and netCDF files.

2015-11-11T00:30:00+00:00

I often finding myself wanting basic information about netCDF files when writing scripts that read/write netCDF files…

The following ncdump command provides a useful overview. The mapping grabs the clipboard register and opens the corresponding netcdf header in a new vim vertical split..

nnoremap <leader>nc :vnew<Bar>0r!ncdump -c <C-R>+<CR>

What about if you want to have a quick look at a plot of one of the variables? Works with ncview too!

nnoremap <leader>ncv :exec '!ncview <C-R>+'<CR>

Speaking of working with netCDF files, here’s a handy snippet for the python library…

snippet loadnetcdf
from netCDF4 import Dataset
infile='/path/to/netcdf4/file.nc'
ifile=Dataset(infile, 'r')
varone=ifile.variables['']
endsnippet

Vim and netCDF files. was originally published by Christopher Bull at Christopher Bull on November 11, 2015.

Beautiful argument parsing with docopt and ultisnips.

2015-10-01T00:30:00+00:00

I recently discovered docopt, this super neat command line parser. I’ve used sys.argv and the argparse module but they’ve always felt too limited or overly complicated (respectively). What’s great about docopt is that it understands what to do based on the POSIX-style doc string you write! This has two advantages…

You’ve likely used enough *nix command-line tools to have a sense of what your doc string should look like (if you do get lost the docs are great).
It forces you to write documentation!

There’s a video here if you want a demo. Whilst it’s not part of core python, one of the neat things is that it is fully self-contained within a single file. Thus, you can ‘install’ it with:

wget https://raw.githubusercontent.com/docopt/docopt/master/docopt.py

Anyway, I’ve written a Vim Ultisnip to do all the work for you… This is one of the more useful Ultisnip interpolation uses I’ve found, you read correctly, it’ll ‘install’ docopt for you!

The above is a gist, feel free to comment/fork/pull.

Beautiful argument parsing with docopt and ultisnips. was originally published by Christopher Bull at Christopher Bull on October 01, 2015.

Grabbing paths straight to the clipboard.

2015-09-05T00:30:00+00:00

Do you remember the moment when you realised in Linux that clicking the mouse wheel pasted selected text? I had a similar reveleation when I discovered xclip recently, a handy utility for pushing text into the clipboard. I’m forever copying and pasting paths from SSH sessions so here’s a bash function that does the work for you…

function clipit() #clipit path 
{
    if [[ ( $# -eq 0 ) || ( $1 == "--help" ) || ( $1 == "-h" ) ]] ; then
        echo "Usage: clipit FILENAME_PATHNAME" 
        echo "Purpose: gets full path of file or path and places in clipboard." 
        echo "       " 
        echo "Requires: xclip" 
        echo "       " 
        echo "Mandatory arguments: " 
        echo "FILENAME_PATHNAME: path to folder or file" 
        echo "       " 
        echo "Example." 
        echo "clipit filename.txt" 
        echo "       " 
        echo "becomes:       " 
        echo "readlink -e filename.txt|tr -d '\n'|xclip -selection c"

        return 1
    fi
  readlink -e ${1}|tr -d '\n'|xclip -selection c
}

If you’re got apt-get and sudo rights, then you can install xclip with:

sudo apt-get install xclip

If you don’t have either of the above, then you can compile from source with (defaults to the slow but likely available GNU compiler):

cd 
wget wget http://downloads.sourceforge.net/project/xclip/xclip/0.12/xclip-0.12.tar.gz
tar -xvf xclip-0.12.tar.gz
mv xclip-0.12 xclip
cd xclip
./configure --prefix=$HOME/xclip
make

Seems like a bit of work but it’s pretty quick. And then, once compiled, add the following to your ~/.bashrc…

#add xclip to the end of bash path
PATH=$PATH:~/xclip/

This assumes that you compiled in ~/xclip/ as above. Happy speedy copy/pasting!

Grabbing paths straight to the clipboard. was originally published by Christopher Bull at Christopher Bull on September 05, 2015.

Bash function for painless rsync.

2015-08-18T00:30:00+00:00

My life consists of two problems…

I work on many different machines and the files I want are invariably on the wrong machine.
I can never remember the right rsync flags for the different machines I use and my fingers are lazy.

The second problem means I never want to deal with the first two.. Sound familiar?

Here’s some bash functions to help you push and pull files from a remote location.

We have one bash function for pulling files….

And another (slightly more complicated) bash function for pushing files….

So, if you put these in your .bashrc with your relevant server details you should be good to go! Feel free to change the switches if you prefer something else, do get in touch if you improve these, it’s impractical to currently pass include/exclude statements at present for instance… The above are GitHub gists, feel free to comment/fork/pull!

Also, life is a lot more smooth if you’re using ssh keys.

Bash function for painless rsync. was originally published by Christopher Bull at Christopher Bull on August 18, 2015.

Editing files remotely

2015-01-01T00:30:00+00:00

I often find myself wanting to edit a file on a remote machine but ssh is too slow to be running a text editor on the remote machine.. Solution? Vim can use rsync to download a copy of the file and then when you save it, it saves it back to the remote location!

Here’s a bash function I wrote to make it all easy….

function vim_servername() 
{
    if [[ ( $# -eq 0 ) || ( $1 == "--help" ) || ( $1 == "-h" ) ]] ; then
        echo "Usage:   vim_servername [path_to]." 
        echo "Purpose: Allows remote editing of files on servers..." 
        echo "       " 
        echo "Mandatory arguments: " 
        echo "path or file: vim will either edit or open path passed" 
        echo "       " 
        echo "Example." 
        echo "This:" 
        echo "vim_servername /server/path/or/file/"
        echo "       " 
        echo "Becomes:" 
        echo "vim scp://username@server.address.edu.au//server/path/or/file/"

        return 1
    fi

    vim scp://username@server.address.edu.au/${1} 

}

So, just put these in your .bashrc with your relevant server details and you should be good to go! Best to be using ssh keys, that way you’re not typing in your password everytime you save..

Editing files remotely was originally published by Christopher Bull at Christopher Bull on January 01, 2015.

Gotcha in Pandas using HDFStore frame_tables

2014-01-08T00:30:00+00:00

Recently, I changed from using regular pytable HDFStores to fancy frame_tables. The move was mostly for the better as my HDFStores were big. Of course, my old code modifying the dataframes broke because of this little bit of syntax…

In [1]: import pandas as pd
In [2]: p=pd.HDFStore('/path/to/some/hdfstore.h5')
In [3]: p.DF['latitude'][0]=20
In [4]: p.DF['latitude'][0]
Out[4]: -25.0
In [5]: DF=p.select('DF')
In [6]: DF['latitude'][0]=20
In [7]: DF['latitude'][0]
Out[7]: 20.0
In [8]: p                                                                                                                            
Out[8]:                                                                                                                              
                                                                                               
File path: /path/to/some/hdfstore.h5                              
/DF                frame_table  (typ->appendable,nrows->24165608,ncols->7,indexers->[index],dc->[longitude,latitude,depth_m,abstime])
/PARLOC            frame_table  (typ->appendable,nrows->17913,ncols->4,indexers->[index])
In [10]: pd.__version__                                                                                                              
Out[10]: '0.12.0'

Of course, the annoying part is that line four doesn’t throw an error message!

Gotcha in Pandas using HDFStore frame_tables was originally published by Christopher Bull at Christopher Bull on January 08, 2014.