Meteo Archive at NOAA ESRL

From tm5
Jump to: navigation, search

The three supercomputers at NOAA ESRL Global Systems Division (GSD) are called iJet, eJet, wJet and share three file systems:

Our meteo archives for each disk are under:


The archive consists of three major components:

Within (1), several flavors of meteo data are stored, these are

The latter two can be created at NOAA ESRL, and their required input data are stored in (3). Summarizing, the meteo archive looks like:

METEO/ <-- Main Branch
shared <-- location for processed TM5 input data in archive form (usually tar)
/e4 <-- The ECMWF ERA40 archive
/od <-- The ECMWF Operational Data (OD) archive
/ncep_cdc <-- The NCEP reanalysis archive
tmm-buf <-- buffer dir with all unpacked data, directly readable for TM5
e4-buf <-- buffer dir for ERA40 specifically
raw_input <-- original data needed to create TM5 meteo files
/ncep.reanalysis <-- NCEP reanalysis files
/ncep.gfs <-- NCEP GFS forecast model files
tools <-- set of tools to manipulate, archive, back-up meteo data

Further branching of these directories (levels, years, grids, etc) follows regular TM5 conventions as can be found in the TM5 Meteo preprocessing manual. While we have two versions of the meteo archive it is important to keep them synchronized, using tools described below. Also, regular back ups to the mass storage system are important since both the regular and lustre file systems are NOT BACKED UP. To help this synchronization, several tools for maintenance are available:

shared_to_mss : sends specified files in the shared directory to the MSS mirror archive
buffer_to_mss : sends specified files in the buffer directory to the MSS mirror archive
synch_buffer : performs an rsync between /p73/co2/METEO/tmm-buf and /tg4/projects/co2/METEO/tmm-buf
synch_shared : performs an rsync between /p73/co2/METEO/shared and /tg4/projects/co2/METEO/shared
synch_tg4 : performs an rsync between the METEO archive on the machine ASTER in Amsterdam and /tg4/projects/co2/METEO/shared
ncep_from_cdc : get NCEP reanalysis data from the CDC anonymous ftp server
ecmwf_from_teras : get ECMWF operational data from the teras (Amsterdam) archive
get_variable : simple script to extract a particular dataset from the TM5 HDF files and send to binary files for other use

Most scripts have a header that defines what to do on which directories, which files, and which time range. The two ftp scripts (_from_) create an output file that has to be executed, e.g.: }}} > ./ncep_from_cdc > ftpncep > chmod a+x ftpncep > ./ftpncep ??? Finally, example rc-file settings for creating/running NCEP data on eJet can be found in the rcfiles directory of the CVS archive:


Further Notes: the scripts to send files to MSS are not yet implemented for non-ECMWF data. No NCEP data has been archived on MSS so far. The ecmwf_to_teras and synch_tg4 script only runs for persons with an account on ASTER in Amsterdam Last back-up to MSS is quite old and some newer (2006) data is likely not on MSS yet. NCEP GFS data has not yet been processed or run with the TM5 2006 source code

Personal tools