======= TEST DATA SET GEMDM RUN =========================== Before deploying the model on real-time initial conditions it is highly recommended to run the model on a test data set to ensure things are working as expected. Two tests can be performed: ================================================================ (1) Global Uniform Resolution Test ================================================================ Steps: ===== 1) link Executables (FOR GEM MODEL SOURCE BUILD) ============================================= The following should work if it is fresh install ln -sf ../gem/gemntr.Abs . ln -sf ../gem/gemdm.Abs . OR link Executables (FOR GEM MODEL LINUX BINARIES: openmpi or mpich2) ================================================================== ln -sf ../gem/gemntr.Abs.openmpi gemntr.Abs OR ln -sf ../gem/gemntr.Abs.mpich2 gemntr.Abs ln -sf ../gem/gemdm.Abs.openmpi gemdm.Abs OR ln -sf ../gem/gemntr.Abs.mpich2 gemntr.Abs IF USING GEM MODEL BINARIES (WITH OPENMPI ONLY) =========================================================== ln -s ../gem/orterun ../gem/mpirun ../gem/mpiexec . export PATH=`pwd`:$PATH 2) Make sure you have CASES subdirectory with ./CASES ----> clim_gef_400_mars96 (Climatology) ----> geophy_400 or gengeoout_global_6km.fst_0000001-0000001 (geophysical flds) ----> 20010920.120000 (analysis data for TEST RUN) OR ----> YYYYMMDD.HH0000 (analysis data for REAL TIME RUN) 3) A sample listing of run_test directory contents looks like (SOURCE BUILD) : drwxr-sr-x 2 badri badri 4096 2011-01-04 13:59 CASES -rwxr-xr-x 1 badri badri 1833 2011-06-11 11:52 rungemntr.sh -rwxr-xr-x 1 badri badri 4205 2011-06-11 11:52 rungemdm.sh -rw-r--r-- 1 badri badri 2155 2011-06-11 11:52 outcfg.out_GV -rw-r--r-- 1 badri badri 2222 2011-06-11 11:52 outcfg.out -rw-r--r-- 1 badri badri 12743 2011-06-11 11:52 gem_settings.nml_GV -rw-r--r-- 1 badri badri 4592 2011-06-11 11:52 gem_settings.nml -rwxr-xr-x 1 badri badri 453 2011-06-18 22:33 envvars.sh drwxr-sr-x 2 badri badri 4096 2011-06-25 10:18 listings lrwxrwxrwx 1 badri badri 17 2011-06-25 11:18 gemntr.Abs -> ../gem/gemntr.Abs lrwxrwxrwx 1 badri badri 16 2011-06-25 11:18 gemdm.Abs -> ../gem/gemdm.Abs -rw-r--r-- 1 badri badri 7843 2011-06-25 11:28 README.TESTRUNGEMDM 4) GEMNTR (pre-processor) a) The settings/namelist file for the test run will execute on 8 processors. You may want to increase or decrease it depending on your hardware setup. In gem_settings.nml adjust Ptopo_npex and Ptopo_npey for processors along x and y directions. Sample: &ptopo Ptopo_npex = 4 , Ptopo_npey = 2 , b) vi rungemntr.sh Modify Analysis data date (YYYMMDD.HH0000) For example for 24 Jan 2011 / 12 UTC analysis data time: ln -s $EXECDIR/CASES/20110124.120000 analysis c) ./rungemntr.sh > gemntr.list 2> gemntr.2 5)GEMDM (Model run) a) Prepare the machines.txt file containing the list of nodes on which the model will execute: Sample machines.txt: yoyou yoyou yoyou yoyou yoyou yoyou yoyou yoyou The hostname yoyou is repeated 8 times because it has 8 cores/processors. b) ./rungemdm.sh > gemdm.list 2> gemdm.2 6) For comparison of the listing you may want to check these under subdirectory listings 8 processor desktop was used for the test run: -rw-r--r-- 1 badri badri 6956 2011-01-21 11:52 pgf90_pgcc_openmpi_gemntr.list.bz2 -rw-r--r-- 1 badri badri 22609 2011-01-21 11:52 pgf90_pgcc_openmpi_gemdm.list.bz2 -rw-r--r-- 1 badri badri 7071 2011-01-21 11:52 ifort_icc_openmpi_gemntr.list.bz2 -rw-r--r-- 1 badri badri 110925 2011-01-21 11:52 ifort_icc_openmpi_gemdm.list.bz2 -rw-r--r-- 1 badri badri 7077 2011-01-21 11:52 ifort_icc_mpich2_gemntr.list.bz2 -rw-r--r-- 1 badri badri 112428 2011-01-21 11:52 ifort_icc_mpich2_gemdm.list.bz2 -rw-r--r-- 1 badri badri 6971 2011-01-21 11:52 gfortran_gcc_openmpi_gemntr.list.bz2 -rw-r--r-- 1 badri badri 22785 2011-01-21 11:52 gfortran_gcc_openmpi_gemdm.list.bz2 -rw-r--r-- 1 badri badri 6957 2011-01-21 11:52 gfortran_gcc_mpich2_gemntr.list.bz2 -rw-r--r-- 1 badri badri 22047 2011-01-21 11:52 gfortran_gcc_mpich2_gemdm.list.bz2 The Details =========== The test runs TABLE of compiler and parallel library combinations ================================================================= # MPICH2/OPENMPI/interconnect on which tested # ----------------------------------------------------------------- # ----------------------------------------------------------------- # Compiler Combo | Interconnect | MPICH/OPENMPI | Remarks # ----------------------------------------------------------------- # ifort/icc | GigE | WORKS*/WORKS | OK to use # ----------------------------------------------------------------- # gfortran/gcc | GigE | WORKS*/WORKS | OK to use # ------------------------------------------------------------------ # pgf90 / pgcc | infiniband | FAILS**/WORKS | use openmpi only # ------------------------------------------------------------------- # ------------------------------------------------------------------- # * WORKS with MPICH if you disable output of Variable PN in outcfg.out # ** TESTED with mpich2 versions 2-1.2.1p1, 2-1.3.1, and # mvapich version 2-1.6rc2. # -to enable mpd process manager compilation please include # --with-pm=mpd for mpich2/mvapich versions 2-1.3.1/2-1.6rc2. # #======== HIGHLY RECOMMENDED =========================== # # IT IS SAFE TO USE OPENMPI ON ANY INTERCONNECT. # #======== HIGHLY RECOMMENDED =========================== #--Listing A sample copy of gemntr.list and gemdm.list using gfortran/gcc, ifort/icc, pgf90/pgcc executables in included in run_test_listings.tar.bz2. Note, the listing files for ifort/icc combination is too big for unknown reasons at this time! ================================================================ (2) Global Variable Resolution Test ================================================================ Steps: ===== Hint: You may want to do this in a fresh directory of your choice: 1) link Executables (FOR GEM MODEL SOURCE BUILD) ============================================= The following should work if it is fresh install ln -sf ../gem/gemntr.Abs . ln -sf ../gem/gemdm.Abs . OR link Executables (FOR GEM MODEL LINUX BINARIES: openmpi or mpich2) ================================================================== ln -sf ../gem/gemntr.Abs.openmpi gemntr.Abs OR ln -sf ../gem/gemntr.Abs.mpich2 gemntr.Abs ln -sf ../gem/gemdm.Abs.openmpi gemdm.Abs OR ln -sf ../gem/gemntr.Abs.mpich2 gemntr.Abs IF USING GEM MODEL BINARIES (ONLY FOR OPENMPI) ============================================== ln -s ../gem/orterun ../gem/mpirun ../gem/mpiexec . export PATH=`pwd`:$PATH 2) Make sure you have CASES subdirectory with ./CASES ----> clim_gef_400_mars96 (Climatology) ----> geophy_400 or gengeoout_global_6km.fst_0000001-0000001 (geophysical flds) ----> 20010920.120000 (analysis data for TEST RUN) OR ----> YYYYMMDD.HH0000 (analysis data for REAL TIME RUN) 3) A listing of run_test directory contents looks like (source build): drwxr-sr-x 2 badri badri 4096 2011-01-04 13:59 CASES -rwxr-xr-x 1 badri badri 1833 2011-06-11 11:52 rungemntr.sh -rwxr-xr-x 1 badri badri 4205 2011-06-11 11:52 rungemdm.sh -rw-r--r-- 1 badri badri 2155 2011-06-11 11:52 outcfg.out_GV -rw-r--r-- 1 badri badri 2222 2011-06-11 11:52 outcfg.out -rw-r--r-- 1 badri badri 12743 2011-06-11 11:52 gem_settings.nml_GV -rw-r--r-- 1 badri badri 4592 2011-06-11 11:52 gem_settings.nml -rwxr-xr-x 1 badri badri 453 2011-06-18 22:33 envvars.sh drwxr-sr-x 2 badri badri 4096 2011-06-25 10:18 listings lrwxrwxrwx 1 badri badri 17 2011-06-25 11:18 gemntr.Abs -> ../gem/gemntr.Abs lrwxrwxrwx 1 badri badri 16 2011-06-25 11:18 gemdm.Abs -> ../gem/gemdm.Abs -rw-r--r-- 1 badri badri 7843 2011-06-25 11:28 README.TESTRUNGEMDM 4) GEMNTR (pre-processor) a) Design your Global Variable Resolution Grid: -cd tools/gem_designgrid_gv -Follow the instructions in README.design.gv.grid -CAUTION: cp run_test/gem_settings.nml_GV gem_settings.nml Set all the grid parameters in gem_settings.nml b) The settings/namelist file for the test run will execute on 2 processors. You may want to increase or decrease it depending on your hardware setup and number of grid points in your domain.. In gem_settings.nml adjust Ptopo_npex and Ptopo_npey for processors along x and y directions. Sample: &ptopo Ptopo_npex = 2 , Ptopo_npey = 1 , c) vi rungemntr.sh Modify Analysis data date (YYYMMDD.HH0000) For example for 24 Jan 2011 / 12 UTC analysis data time: ln -s $EXECDIR/CASES/20110124.120000 analysis d) ./rungemntr.sh > gemntr.list 2> gemntr.2 5)GEMDM (Model run) a) Prepare the machines.txt file containing the list of nodes on which the model will execute: Sample machines.txt: yoyou yoyou The hostname yoyou is repeated 8 times because it has 8 cores/processors. b) cp outcfg.out_GV outcfg.out c) ./rungemdm.sh > gemdm.list 2> gemdm.2 6) For comparison of the listing you may want to check these under subdirectory listings 2 processor desktop was used for the test run: Look for file: gfortran_gcc_openmpi_gemntr_gv.list and gfortran_gcc_openmpi_gemdm_gv.list NOTE: we have not performed the tests with ifort/icc or pgf90/pgcc compiler combo. Known Issue: =========== When user outputs PN field through outcfg.out file while using gemdm.Abs.mpich2 runs the model crashes. For now, please proceed without PN field in outcfg.out. DETAILED DESCRIPTION OF SCRIPTS, EXECUTABLES, ENVI VARIABLES: #================================================================ 1)envvars.sh ------------ DESCRIPTION =========== Initializes/exports environment variables: GEMDM specific: a) rep_from_which_model_is_launched: Basically the directory holding the *.Abs files b) AFSISIO : Path to the radiation, ozone and physical constants Example: -------- AFSISIO=/d1/badri/model/GEM/gem/data NOTE: The model internally uses ${AFSISIO}/datafiles/constants (dont ask me why!) Listing of ${AFSISIO}/datafiles/constants: -rwxr-xr-x 1 badri badri 2387 2006-07-14 15:43 thermodynamic -rwxr-xr-x 1 badri badri 2387 2006-07-14 15:43 thermoconsts -rwxr-xr-x 1 badri badri 34128 2006-07-14 15:43 ozoclim_32 -rwxr-xr-x 1 badri badri 40960 2006-07-14 15:43 ozoclim -rwxr-xr-x 1 badri badri 2675880 2006-07-14 15:43 irtab5_std -rwxr-xr-x 1 badri badri 2869496 2006-07-14 15:43 irtab5_32 -rwxr-xr-x 1 badri badri 23267 2006-07-14 15:43 dict.data -r-xr-xr-x 1 badri badri 2387 2006-07-14 15:43 constantes c) bash script : EXECDIR : Basically same as rep_from_which_model_is_launched d) INTEL compiler: F_UFMTENDIAN : Used for Intel compiled executables by setting F_UFMTENDIAN=big 2) gem_settings.nml ------------------- DESCRIPTION =========== Model namelist file Details: ======== http://collaboration.cmc.ec.gc.ca/science/rpn.comm/wiki1/doku.php?id=gem:gem_settings_doc#the_gem_settings.nml_file 3) outcfg.out ------------- DESCRIPTION =========== File containing all the model variables a user wishes to output. Details: ======== http://collaboration.cmc.ec.gc.ca/science/rpn.comm/wiki1/doku.php?id=gem:outcfg_doc#the_outcfg.out_file 4) GEMNTR and GEMDM executables ------------------------------- gemntr.Abs and gemdm.Abs gemntr.Abs (What it does in brief) ---------- a) Reads in 3D meteorological data and horizontally interpolates on to the specified grid in gem_settings.nml b) Reads in climatological fields and horizontally interpolates on to the specified grid in gem_settings.nml c) Reads in geophysical fields (static ones) and horizontally interpolates on to the specified grid in gem_settings.nml gemdm.Abs (What it does in brief) --------- a) Vertical interpolation of 3D meteorological data to model levels b) Domain decomposed according to number of processors along x and y directions (from gem_settings.nml) c) Digital filter initialization d) Forward integration in time of the model to produce forecasts e) Output user specified variables on model / pressure levels. DESCRIPTION =========== The executables are created after a successful compilation of the armnlib/lapack/gem package (described in README.GEM) 5) bash model run scripts ------------------------- rungemntr.sh and rungemdm.sh DESCRIPTION =========== These provide the basic template which the user may want to customize according to his/her hardware/cluster setup.