nomads/doc/getting_dods_data... 20070618
from using the SREF 1/4 degree data set
A note on the status of NOMADS SREF datasets 20070621:
The NOMADS archive component at NCDC is planning an SREF archive at NCDC (I am not sure what is archived at this time but ensembles are under discussion). NCDC archive servers are operational 24/7. Tne NCEP backup server at http://nomads6.ncdc.noaa.gov/ncep_data is planning to backup http://nomad5.ncep.noaa.gov with a real time SREF data set when it is available on ftpprd ftp server. (nomad5 SREF contains 4 cycles/day but nomad3 can hold 2 cycles/day -- we are working on getting SREF to nomads6.ncdc...)
One
can obtain repackaged GRIB files sliced, diced and area subset,
according to your data needs, from the NOMADS NCEP real time SREF
with the "ftp2u" application. Or obtain values from SREF
GRIB files called from inside your program that makes your product
(like our disk storage is on your workstation).
Some knowledge of
unix is needed for this.
If you have not tried the NOMAD server
before there are some howto links at
http://nomad3.ncep.noaa.gov
http://nomad5.ncep.noaa.gov
http://nomads6.ncdc.noaa.gov/ncep_data
Also add "/ncep_data" to any of the above to find
the NCEP data table where links to all NOMADS applications are found.
The ftp2u application for SREF is on the ncep_data table, for
example, see
http://nomad5.ncep.noaa.gov/ncep_data
and
find the SREF (toward the bottom of the table) entry in the table and
click on "ftp2u". Select a day(date) from the list.
Part I, ftp2u:
The goal
is an automatic download and custom data flow for repackaged GRIB
files. Lets start with how to manually get a single data set and
then we will automate the process by creating a URL and script that
does the job. ftp2u is designed to do both of these things. Start by
selecting (for now) a sample file from the many that are presented.
The first one is "sref_em.t03z.pgrb212.ctl.f00" which is
one of hundreds that are listed. Select this by checking the check
box next to it. em is the WRF at 45 km with NCAR physics.
Use
the "GRIB filter", following the long list of files, to
select variables. For example, if we wanted only the 2m temperature
and total precip, 2D fields, one need check the "sfc" and
"2 m above gnd" in the "level" section as well as
"APCP" and "TMP", respectively in the "variables"
section (this will include a surface temperature as well).
Select
the area subset by checking the "make subregions" switch
and enter your lat/lon corners, eg., (38-42N,85-80W).
By
default, ftp2u will send this data set to its own ftp/http server
where you can manually download it with a "file-save" but
if you have your own anonymous ftp server it will deliver the file to
your ftp server. This feature is depreciated by NOAA/NCEP security
and will not be around in the future.
Click "start download"
and the next screen will display the location of the repackaged GRIB
file. You can manually file-save this data set or use ftp to get it
and check that it is what you want. The URL for the above looks
like:
http://nomad5.ncep.noaa.gov/cgi-bin/ftp2u_sref.sh?file=sref_em.t03z.pgrb212.ctl.f${ft}&wildcard=&lev_sfc=on&lev_2_m_above_gnd=on&var_APCP=on&var_TMP=on&subregion=&leftlon=280&rightlon=285&toplat=42&bottomlat=38&results=SAVE&rtime=1hr&machine=140.90.192.197&user=anonymous&passwd=&ftpdir=%2Fincoming_1hr&prefix=&dir=%2Fsref20070618
(No carriage returns or line feeds or spaces)
Copy-paste
the entire URL into a unix script file. One could manually make
changes to this URL (try the date) and re-send it in a browser for
different server requests.
The above URL is placed into a
unix script, sref.sh, as the quoted object of a non-interactive web
download command, wget, to create the repackaged GRIB file, and flag
that GRIB files location on the server, and "wget" it to
your workstation. The script to do this is below.
script: sref.sh
# When using a loop add a "sleep 21" wait 21 seconds, to stop run
# away loops (There is a throttle that will slow you down if you don't).
# Unix script using the non-interactive web download commnad, wget,
# to have the NOMADS server create a repackaged GRIB data set and send it to
# the NOMADS ftp site, then find that location of the data on the anonymous
# ftp site and (w)get it to your workstation. The URL was created with a
# successful ftp2u session and copy-pasted and modified (changed the
# forecast time). The new repackaged GRIB1 file is downloaded with a second
# wget command. note: sed or awk would be improvements in place of cut.
#
# What is going on is a successful ftp2u writes "web page" html to display
# the location of where the ftp2u session puts the data on our anonymous
# ftp site. The web page location is obtained by the first wget call which
# starts and completes a successful ftp2u session according to the keyword
# values in the URL. The output of this sent URL is stuffed into stuff.
# For each model forecast time (ft) this URL example gets the 12-h and 24-h
# forecast for the SREF ...sref_em... forecast and then by using the unix cut
# command (found by the 8th ">" in the URL web page text) from the last ftp2u
# web page showing the location of the created GRIB file, and from which, the "cut"
# command isolates the ftp address listed therein, which a second wget gets using
# the location loc2. The location needs to have some extraneous characters
# removed, like the "</>" for a proper web IP address which is taken care of
# by the unix construct "%???". (No carriage returns or line feeds in the URL even though web pages often show long lines with them when reprinted.)
#
# (*j*)
#
for ft in 12 24
do
echo "ft=$ft"
stuff=`wget -O - "http://nomad5.ncep.noaa.gov/cgi-bin/ftp2u_sref.sh?file=sref_em.t03z.pgrb212.ctl.f${ft}&wildcard=&lev_sfc=on&lev_2_m_above_gnd=on&var_APCP=on&var_TMP=on&subregion=&leftlon=280&rightlon=285&toplat=42&bottomlat=38&results=SAVE&rtime=1hr&machine=140.90.192.197&user=anonymous&passwd=&ftpdir=%2Fincoming_1hr&prefix=&dir=%2Fsref20070618"`
echo stuff=$stuff
location=`echo $stuff|cut -d\> -f8`
echo location=$location
export loc2=${location%???}
wget "$loc2"
sleep 21
done-----------------------------------------------------------
The (raw) output from the script, which has echo's (diagnostic prints), showing the contents of the variables, is also below.
ft=12 stuff=<head><title>Data Transfer: NCEP SREF Forecasts (1 degree grid)</title></head> <body bgcolor="#ffffff"> ******************************************************<br> sref_em.t03z.pgrb212.ctl.f12 was copied to <a href="ftp://nomad5.ncep.noaa.gov/pub/NOMAD_1hr/tdir18-13.20042/sref_em.t03z.pgrb212.ctl.f12">ftp://nomad5.ncep.noaa.gov/pub/NOMAD_1hr/tdir18-13.20042/sref_em.t03z.pgrb212.ctl.f12</a>, 946 bytes<br> grib filter=egrep ':(APCP|TMP):' | egrep ':(2 m above gnd|sfc):'<br> subregion: W=280 E=285 N=42 S=38<br> ******************************************************<br> <br> <br>Results were saved in <a href="ftp://nomad5.ncep.noaa.gov/pub/NOMAD_1hr/tdir18-13.20042">ftp://nomad5.ncep.noaa.gov/pub/NOMAD_1hr/tdir18-13.20042</a><br> <br>Or <a href="http://nomad5.ncep.noaa.gov/datalink/ftp_data/pub/NOMAD_1hr/tdir18-13.20042">http://nomad5.ncep.noaa.gov/datalink/ftp_data/pub/NOMAD_1hr/tdir18-13.20042</a><br> transferred 1 out of 1 files<br> </body> location=ftp://nomad5.ncep.noaa.gov/pub/NOMAD_1hr/tdir18-13.20042/sref_em.t03z.pgrb212.ctl.f12</a ft=24 stuff=<head><title>Data Transfer: NCEP SREF Forecasts (1 degree grid)</title></head> <body bgcolor="#ffffff"> ******************************************************<br> sref_em.t03z.pgrb212.ctl.f24 was copied to <a href="ftp://nomad5.ncep.noaa.gov/pub/NOMAD_1hr/tdir18-13.20131/sref_em.t03z.pgrb212.ctl.f24">ftp://nomad5.ncep.noaa.gov/pub/NOMAD_1hr/tdir18-13.20131/sref_em.t03z.pgrb212.ctl.f24</a>, 1036 bytes<br> grib filter=egrep ':(APCP|TMP):' | egrep ':(2 m above gnd|sfc):'<br> subregion: W=280 E=285 N=42 S=38<br> ******************************************************<br> <br> <br>Results were saved in <a href="ftp://nomad5.ncep.noaa.gov/pub/NOMAD_1hr/tdir18-13.20131">ftp://nomad5.ncep.noaa.gov/pub/NOMAD_1hr/tdir18-13.20131</a><br> <br>Or <a href="http://nomad5.ncep.noaa.gov/datalink/ftp_data/pub/NOMAD_1hr/tdir18-13.20131">http://nomad5.ncep.noaa.gov/datalink/ftp_data/pub/NOMAD_1hr/tdir18-13.20131</a><br> transferred 1 out of 1 files<br> </body>
location=ftp://nomad5.ncep.noaa.gov/pub/NOMAD_1hr/tdir18-13.20131/sref_em.t03z.pgrb212.ctl.f24</a
---------------------------------------------
A
documentation link for wgrib (See below the NCEP data_table, click on
GRIB), will lead to a web page that you can download a copy of the
wgrib program to unpack, inventory and get values from GRIB files if
you do not have it yet.
---------------------------------------------
Part II: OPENDAP/GDS queries
OPENDAP(DODS)/GDS:
is a software framework used for data networking that make local data
accessible to remote locations.
Each file in NOMADS has a man or
machine readable metadata description describing the file. These can
be accessed on the NOMADS ncep_data table by clicking the link,
"DODS". Follow the links to the .info page where the top
line labeled "DODS URL" contains the location of the data,
which is the same as that of the .info page without the .info. Note
also that the .info metadata descriptions contain information about
what variables are present, time, and space, units, projection info,
etc... information.
Using the nomad5 first SREF file listed
as the one for today we have (only two past days are kept so to try
this you have to fix the date from 20070618 to near the present):
http://nomad5.ncep.noa.gov:9090/dods/sref/archive/sref20070618/sref_em_03z.ctl.info
OPENDAP(DODS)/GDS returns unpacked values in response to
queries. I may query the data obtained in part I above, using
information from the metadata description:
The order of the query constraint is first the DODS command address and directory (including the cycle and model type) which is the URL address, followed by the command ".ascii" (instead of .info), followed by a separator "?", followed by the variable name "apcpsfc" (for total precip), followed by the forecast time (00 is the initial condition), followed by the latitudes, and then the longitudes:
....ascii?<variable>[time1:time2][lat1:lat2][lon1:lon2].
If
there are vertical levels (3-D variable) then [lev1:lev2] is between
[times] and [lats].
If we were on a lon/lat equally spaced 1x1
grid then we would count from the south pole as lat1 would be 0 and
the equator would be 90 and 360 degrees around from the 0 meridian to
find position. But the SREF is on a 1/4 degree Lambert conformal
projection over the CONUS starting from (10.N,-150W). Or for a
rectangle (38-42N) and (-85 to -80), the area is covered by
[112:128][260:280] and the first point is [112:112][260:260] (for
example: 1st latitude=38.N= (38.*4)-(10.*4)=28*4=112).
So,
our DODS query of total precip of the sref_em... (NCAR model) can be
constrained to the above area subset (first [0:0] means the zeroth or
first time or initial condition) with the phrase:
apcpsfc[0:0][112:128][260:280]
added to the URL
address of the dataset. Or putting it together with the server IP
address and dods directory location and ascii command:
http://nomad5.ncep.noa.gov:9090/dods/sref/archive/sref20070618/sref_em_03z.ctl.ascii?apcpsfc[0:0][112:128][260:280]
Of course, there is no precip for the initial condition as it
is a diagnosed model variable so we need the first [1:1] (is the 3-h
interval for SREF) or later forecast times, like the first 10 3-h
forecasts on 3-h intervals is queried by changing the [0:0] to
[1:10]. We will show attached the results for the above for the
first 3-h (3-h accumulated) precip [1:1] forecast.
http://nomad5.ncep.noa.gov:9090/dods/sref/archive/sref20070618/sref_em_03z.ctl.ascii?apcpsfc[1:1][112:128][260:280]
Entering this query (no carriage returns and no line feeds)
would deliver the 3-h forecast 17x21 values near Ohio. We can send
this manually in any Internet browser or place it in a unix script as
the quoted object of an interactive web download program like wget
just as we did above for ftp2u. A script that does this is below:
# #for ft in 12 24 #do #echo "ft=$ft" stuff=`wget -O - "http://nomad5.ncep.noaa.gov:9090/dods/sref/archive/sref20070618/sref_em_03z.ctl.ascii?apcpsfc[1:1][112:128][260:280]"` echo stuff=$stuff #done exit # ---- output from the above script. The stuff in variable stuff. apcpsfc, [1][17][21] [0][0], 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0 [0][1], 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0071407785, 0.039973475, 0.06471348, 0.07151604, 0.051903863, 0.009578861, 0.0, 0.0, 0.0, 0.0 [0][2], 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0014081001, 0.0036619422, 0.00907135, 0.07767105, 0.13427894, 0.17397517, 0.16264972, 0.07997042, 0.03512177, 0.005511228, 0.0, 0.0 [0][3], 0.0, 0.0, 0.0, 0.0, 0.010245933, 0.04702538, 0.071105, 0.07699108, 0.083005905, 0.08914518, 0.05163065, 7.991791E-4, 0.069236755, 0.14827555, 0.2829012, 0.3231454, 0.18657005, 0.08878136, 0.020427704, 0.0, 0.0 [0][4], 0.0, 0.0, 0.0, 0.0, 0.0143013, 0.08264732, 0.125, 0.125, 0.11119209, 0.080942295, 0.04046594, 0.003615652, 0.060791016, 0.13057745, 0.22072244, 0.26390532, 0.17502739, 0.098033905, 0.029844284, 0.0, 0.0 [0][5], 0.0, 0.0, 0.0, 0.0, 0.007144928, 0.07532883, 0.125, 0.12217665, 0.08846188, 0.023802904, 0.0, 0.0, 0.06657565, 0.15956731, 0.21181284, 0.24626544, 0.21190554, 0.16594902, 0.063007936, 0.0, 0.0 [0][6], 0.0, 0.0, 0.0, 0.0, 0.0, 0.024845826, 0.039951324, 0.03410673, 0.023179375, 0.006171884, 0.0, 0.0, 0.08669673, 0.22039026, 0.24419744, 0.24967644, 0.25, 0.23525275, 0.12088901, 0.04581356, 0.053287983 [0][7], 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.046699338, 0.13066432, 0.18776333, 0.24047406, 0.25, 0.25, 0.18247034, 0.12432818, 0.11495771 [0][8], 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.016510734, 0.053017825, 0.082259916, 0.10402845, 0.1278297, 0.16090956, 0.14415663, 0.12501758, 0.068709806 [0][9], 0.15841049, 0.05780053, 0.019757992, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.02636117, 0.0670052, 0.072193146, 0.06489658, 0.030971779 [0][10], 0.82308596, 0.50698173, 0.20515099, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0 [0][11], 2.3276112, 1.5762827, 0.61305356, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0 [0][12], 1.7698854, 0.65231925, 0.2600122, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0 [0][13], 1.2086434, 0.21342008, 0.09096429, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0 [0][14], 0.6531795, 0.16258192, 0.05446726, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0 [0][15], 0.1479355, 0.036573347, 0.008052319, 4.4334546E-4, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0 [0][16], 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0 time, [1] 732846.25 lat, [17] 38.0, 38.25, 38.5, 38.75, 39.0, 39.25, 39.5, 39.75, 40.0, 40.25, 40.5, 40.75, 41.0, 41.25, 41.5, 41.75, 42.0 lon, [21] -85.0, -84.75, -84.5, -84.25, -84.0, -83.75, -83.5, -83.25, -83.0, -82.75, -82.5, -82.25, -82.0, -81.75, -81.5, -81.25, -81.0, -80.75, -80.5, -80.25, -80.0
Once
the data is in "stuff" (the variable $stuff) it is in
memory, and then can be parsed with sed/awk or cut. One could
place it in a file too, but any language can get the values it needs
into a variable (memory) and utilize it.
Notice:
a) That the response of the server to this query returns the variable name and information about the array of values returned.
b) The data is returned with units as indicated in the metadata description (.info page) which is also obtainable in the same way with a wget and can also be parsed.
c) The GDS (Grads Data Server) also returns the time (since the year 1), latitudes and longitudes of the values which can be used and check by the user.
Part
IIb: The GDS/OPENDAP(DODS) server and user clients.
Many open source and commercial clients understand the OPENDAP protocol. For example, the open source program Grads, freeware at http://www.iges.org/grads/grads.html
which should not be confused with GDS used in NOMADS, can display and manipulate meteorological data. If grads is running on a users workstation then that user can use the Grads open dataset command "sdfopen" to open a dataset on the NOMADS server from the users workstation. The address is the same as that above, for example, the URL from the metadata description web page (.info). Once opened the user client program, Grads, can display or operate with any of its commands on the datasets on NOAMDS disk storage as if they were locally on the users workstation. This is also true for many commercial packages such as MATLAB and IDL. These programs understand the standard protocol language of OPENDAP. A server side client that does this on NOMADS is "pdisp" which is an example fronted by a web page for displaying data from NOMADS datasets.