20. Secondary Variables - Eeeeeek!! Yes the time has come to attack what even
Tim seems to have been unhappy about (reading between the lines). To assist
me I have 12 lines in the gridding ReadMe file.. so par for the course.
Almost immediately I hit that familiar feeling of ambiguity: the text
suggests using the following three IDL programs:
frs_gts_tdm.pro
rd0_gts_tdm.pro
vap_gts_anom.pro
So.. when I look in the code/idl/pro/ folder, what do I find? Well:
3447 Jan 22 2004 fromdpe1a
/code/idl/pro/frs_gts_anom.pro
2774 Jun 12 2002 fromdpe1a
/code/idl/pro/frs_gts_tdm.pro
2917 Jan 8 2004 fromdpe1a
/code/idl/pro/rd0_gts_anom.pro
2355 Jun 12 2002 fromdpe1a
/code/idl/pro/rd0_gts_tdm.pro
5880 Jan 8 2004 fromdpe1a
/code/idl/pro/vap_gts_anom.pro
In other words, the *anom.pro scripts are much more recent than the *tdm
scripts. There is no way of knowing which Tim used to produce the current
public files. The scripts differ internally but - you guessed it! - the
descriptions at the start are identical. WHAT IS GOING ON? Given that the
'README_GRIDDING.txt' file is dated 'Mar 30 2004' we will have to assume
that the originally-stated scripts must be used.
To begin with, we need binary output from quick_interp_tdm2, so it's run
again for tmp and pre, and (for the first time) for dtr. This time, the
command line looks like this for tmp:
IDL> quick_interp_tdm2,1901,2002,'idlbinout/idlbin',1200,gs=2.5,dumpbin='dumpbin',pts_prefix='tmp_txt_4idl/tmp.'
This gives screen output for each year, typically:
1991
grid 1991 non-zero 0.9605 2.0878 2.1849 cells= 27048
And produces output files (in, in this case, 'idlbinout/'), like this:
-rw------- 1 f098 cru 248832 Sep 21 12:20 idlbin_tmp/idlbin_tmp1991
At this point, did some logical renaming. So..
.txt files (pre-IDL) are typically 'tmp.1901.01.txt' in 'tmp_txt_4idl/'
binary files (post-IDL) are typically 'idlbin_tmp1991' in 'idlbin_tmp/'.
These changes rolled back to the quoted command lines, to avoid confusion.
Next, precip command line:
IDL> quick_interp_tdm2,1901,2002,'idlbin_pre/idlbin_pre',450,gs=2.5,dumpbin='dumpbin',pts_prefix='pre_txt_4idl/pre.'
(note new filenaming schema)
This gives example screen output:
1991
grid 1991 non-zero -4.8533 36.2155 51.0738 cells= 51060
And produces output files like:
-rw------- 1 f098 cru 248832 Sep 21 12:50 idlbin_pre/idlbin_pre1991
Finally for the primaries, the first stab at dtr. Ran anomdtb with the
database file dtr.0312221128.dtb, and the standard/recommended responses.
Screen output:
> NORMALS MEAN percent STDEV percent
> .dtb 0 0.0
> .cts 3375441 84.1 3375441 84.1
> PROCESS DECISION percent %of-chk
> no lat/lon 3088 0.1 0.1
> no normal 638538 15.9 15.9
> out-of-range 70225 1.7 2.1
> duplicated 135457 3.4 4.1
> accepted 3167636 78.9
> Dumping years 1901-2002 to .txt files...
Then for the gridding:
IDL> quick_interp_tdm2,1901,2002,'idlbin_dtr/idlbin_dtr',750,gs=2.5,dumpbin='dumpbin',pts_prefix='dtr_txt_4idl/dtr.'
Giving screen output:
1991
grid 1991 non-zero -0.3378 1.6587 1.7496 cells= 3546
And files such as:
-rw------- 1 f098 cru 248832 Sep 21 13:39 idlbin_dtr/idlbin_dtr1991
And.. at this point, I read the ReadMe file properly. I should be gridding at
2.5 degrees not 0.5 degrees! For some reason, secondary variables are not
derived from the 0.5 degree grids. Re-did all three generations (the sample
command lines and outputs above have been altered to reflect this, to avoid
confusion).
So, to the generation of the synthetic grids.
Tried running frs_gts_tdm but it complained it couldn't find the normals file:
IDL> frs_gts_tdm,dtr_prefix='idlbin_dtr/idlbin_dtr',tmp_prefix='idlbin_tmp/idlbin_tmp',1901,2002,outprefix='syngrid_frs/syngrid_frs'
% Compiled module: FRS_GTS_TDM.
% Attempt to call undefined procedure/function: 'FRS_GTS_TDM'.
% Execution halted at: $MAIN$
IDL> frs_gts,dtr_prefix='idlbin_dtr/idlbin_dtr',tmp_prefix='idlbin_tmp/idlbin_tmp',1901,2002,outprefix='syngrid_frs/syngrid_frs'
% Compiled module: RDBIN.
% Compiled module: STRIP.
ls: /home/cru/f098/m1/gts/frs/glo/glo.frs.norm not found
ls: /home/cru/f098/m1/gts/frs/glo/glo.frs.norm.Z not found
ls: /home/cru/f098/m1/gts/frs/glo/glo.frs.norm.gz not found
% READF: End of file encountered. Unit: 99, File: foo
% Execution halted at: RDBIN 25 /cru/u2/f080/Idl/rdbin.pro
% FRS_GTS 18 /cru/cruts/fromdpe1a
/code/idl/pro/frs_gts_tdm.pro
% $MAIN$
IDL>
However when I eventually found what I hope is the normals file:
/cru/cruts/fromdpe1a/data/grid/twohalf/glo25.frs.6190
..and altered the IDL prog to read it.. same error! Turns out it's preferring
to pick up Mark N's version so tried explicitly compiling,
('.compile xxxxxx.pro') that worked, in that the error changed:
IDL> frs_gts,dtr_prefix='idlbin_dtr/idlbin_dtr',tmp_prefix='idlbin_tmp/idlbin_tmp',1901,2002,outprefix='syngrid_frs/syngrid_frs'
% Compiled module: RDBIN.
% Compiled module: STRIP.
yes
% Variable is undefined: NF.
% Execution halted at: RDBIN 68 /cru/u2/f080/Idl/rdbin.pro
% FRS_GTS 21 /cru/cruts/fromdpe1a
/code/idl/pro/frs_gts_tdm.pro
% $MAIN$
IDL>
So what is this mysterious variable 'nf' that isn't being set? Well strangely,
it's in Mark N's 'rdbin.pro'. I say strangely because this is a generic prog
that's used all over the place! Nonetheless it does have what certainly looks
like a bug:
38 if keyword_set(gridsize) eq 0 then begin
39 info=fstat(lun)
40 if keyword_set(seas) then info.size=info.size*2.0
41 if keyword_set(ann) then info.size=info.size*12.0
42 nlat=sqrt(info.size/48.0)
43 gridsize=180.0/nlat
44 if keyword_set(quiet) eq 0 then print,'filesize=',info.size
45 if keyword_set(quiet) eq 0 then print,'gridsize=',gridsize
46 endif
47 if keyword_set(had) then had=1 else had=0
48 if keyword_set(echam) then echam=1 else echam=0
49 if keyword_set(gfdl) then gfdl=1 else gfdl=0
50 if keyword_set(ccm) then ccm=1 else ccm=0
51 if keyword_set(csiro) then csiro=1 else csiro=0
52 ;create array to read data into
53 if keyword_set(seas) then nf=6 else nf=12
54 if keyword_set(ann) then nf=1
55 defxyz,lon,lat,gridsize,grid=grid,nf=nf,had=had,echam=echam,gfdl=gfdl,ccm=ccm,csiro=csiro
56 if keyword_set(quiet) eq 0 then help,grid
57 grid=fix(grid)
58 ;read data
59 readu,lun,grid
60 close,lun
61 spawn,string('rm -f ',fff)
62 endif else begin
63 openr,lun,fname
64 ; check file size and work out grid spacing if gridsize isn't set
65 if keyword_set(gridsize) eq 0 then begin
66 info=fstat(lun)
67 if keyword_set(quiet) eq 0 then print,'yes'
68 nlat=sqrt((info.size/nf)/4.0)
69 gridsize=180.0/nlat
70 if keyword_set(quiet) eq 0 then print,'filesize=',info.size
71 if keyword_set(quiet) eq 0 then print,'gridsize=',gridsize
72 endif
73 if keyword_set(seas) then nf=6.0 else nf=12.0
74 if keyword_set(ann) then nf=1
In other words, 'nf' is set in the first conditional set of statements, but in
the alternative (starting on #62) it is only set AFTER it's used
(set #73,#74; used #68). So I shifted #73 and #74 to between #64 and #65, and..
with precompiling to pick up the local version of rdbin, too.. it worked!
Er, perhaps.
Lots of screen output, and lots of files. A set of synthetic grids in 'syngrid_frs/' as requested, typically:
-rw------- 1 f098 cru 20816 Sep 17 22:10 syngrid_frs/syngrid_frs1991.Z
..but also a set of some binariy files in the working directory! They look like this:
-rw------- 1 f098 cru 51542 Sep 17 22:10 glo.frs.1991.Z
Having read the program it looks as though the latter files are absolutes,
whereas the former are anomalies. With this in mind, they are renamed:
glo.frs.1991 -> glo.frs.abs.1991
..and put into folder syngrid_frs_abs/
Then - a real setback. Looked for a database file for frost.. nothing. Is
this a real secondary parameter? Answer: yes. Further digging revealed that
quick_interp_tdm2.pro has a 'nostn' command line option. It's undocumented,
as usual, but it does seem to avoid the use of the 'pts_prefix' option.. so
I set it, and it at least *ran* for the full term (though very slow compared
to primary variables)!
IDL> quick_interp_tdm2,1901,2002,'glo_frs_grids/frs.grid.',750,gs=0.5,dumpglo='dumpglo',nostn=1,synth_prefix='syngrid_frs/syngrid_frs'
It does produce output grids. Without converting to absolutes with the normals file,
it's hard to know if they're realistic.
Then, I moved on to rd0 (wet-day frequency). This time, when I searched for the
normals files required ('glo.pre.norm' and 'glo.rd0.norm'), I could not (as before)
find exact matches. The difference this time is that the program checks that the
normals file supplied is a 0.5-degree grid, so glo25.pre.6190 failed. This implies
to me that my approach to frs (above) was wrong as well. Where is the documenatation
to explain all this?!
Finally - a breakthrough. A search of Mark New's old directory hierarchy revealed
what look like the required files:
crua6[/cru/mark1/f080] find . -name 'glo.*.norm*'
./gts/cld/glo/glo.cld.norm.Z
./gts/dtr/glo_old/glo.dtr.norm.Z
./gts/frs/glo.frs.norm.Z
./gts/frs/glo/glo.frs.norm.Z
find: cannot open < ./gts/frs/glo_txt >
./gts/pre/glo_quick_abs/glo.pre.norm.Z
./gts/pre/glo_quick_log/glo.pre.norm.Z
./gts/pre/glo_spl/glo.pre.norm.Z
find: cannot open < ./gts/pre_perc/station_list >
./gts/rad/glo/glo.rad.norm.Z
./gts/rd0/glo/glo.rd0.norm.Z
./gts/rd0/glo_old/glo.rd0.norm.Z
./gts/sunp/glo/glo.sunp.norm
./gts/sunp/means/glo.sunp.norm.Z
./gts/tmp/glo/glo.tmp.norm.Z
./gts/tmp/glo_old/glo.tmp.norm.Z
find: cannot open < ./gts/tmp/station_list >
./gts/vap/glo/glo.vap.norm.Z
./gts/wnd/glo/glo.wnd.norm.Z
A listing of /cru/mark1/f080/gts gives:
drwxr-x--- 2 f080 cru 1024 Sep 12 2005 cdrom
drwxr-x--- 10 f080 cru 57344 Nov 1 2001 cld
drwxr-xr-x 19 f080 cru 24576 Feb 27 2001 dtr
drwxr-x--- 2 f080 cru 8192 Feb 25 1998 elev
drwxr-x--- 2 f080 cru 8192 Jun 8 1998 euroclivar
-rw-r----- 1 f080 cru 0 Aug 3 1999 foo
drwxr-x--- 6 f080 cru 8192 Aug 6 2002 frs
-rw-r-x--- 1 f080 cru 438 May 12 1998 gts.errors
-rw-r----- 1 f080 cru 10 Jul 21 1999 in
drwxr-x--- 5 f080 cru 8192 Jan 6 1999 jiang
drwxr-x--- 2 f080 cru 8192 Apr 7 1998 landsea
-rw-r----- 1 f080 cru 240 May 12 1998 normal.errors
drwxr-x--- 5 f080 cru 8192 Aug 6 2002 plots
drwxr-xr-x 12 f080 cru 106496 May 22 2000 pre
drwxr-x--- 9 f080 cru 114688 Aug 6 2002 pre_perc
drwxr-x--- 4 f080 cru 1024 Jan 6 1999 rad
drwxr-x--x 6 f080 cru 8192 Nov 1 2001 rd0
-rwxr-xr-- 1 f080 cru 1779 Dec 5 1997 readme.txt
drwxr-x--- 8 f080 cru 1024 Apr 5 2000 reg_series
drwxr-x--- 3 f080 cru 1024 Oct 18 1999 reh
drwxr-x--- 2 f080 cru 8192 Jan 19 2000 scengen
drwxr-x--- 5 f080 cru 24576 Nov 5 1998 sunp
drwxr-x--- 2 f080 cru 1024 Aug 6 2002 test
drwxr-x--- 4 f080 cru 1024 Aug 3 1999 tmn
drwxr-xr-x 20 f080 cru 122880 Mar 19 2002 tmp
drwxr-x--- 4 f080 cru 1024 Aug 3 1999 tmx
drwxr-x--- 6 f080 cru 1024 Jul 8 1998 ukcip
drwxr-x--- 5 f080 cru 8192 Nov 5 2001 vap
drwxr-x--- 4 f080 cru 1024 Jul 2 1998 wnd
And a listing of, for example, the 'frs' directory:
drwxr-x--- 2 f080 cru 16384 Jul 18 2002 glo
-rw-r-x--- 1 f080 cru 433393 Aug 12 1998 glo.frs.1961.Z
-rw-r-x--- 1 f080 cru 321185 Aug 12 1998 glo.frs.ano.1961.Z
-rw-r-x--- 1 f080 cru 740431 Aug 12 1998 glo.frs.norm.Z
drwxr-xr-x 2 f080 cru 16384 Jul 27 1999 glo25
drwx------ 2 f080 cru 8192 Jul 18 2002 glo_txt
drwxr-xr-x 2 f080 cru 8192 Aug 28 1998 means
So, the following were copied to the working area:
cp /cru/mark1/f080/gts/frs/glo.frs.norm.Z /cru/cruts/rerun1/data/cruts/rerun_synth/
cp /cru/mark1/f080/gts/cld/glo/glo.cld.norm.Z /cru/cruts/rerun1/data/cruts/rerun_synth/
cp /cru/mark1/f080/gts/dtr/glo_old/glo.dtr.norm.Z /cru/cruts/rerun1/data/cruts/rerun_synth/
precip looked like it might be a problem (3 matching files, see above),
but on investigation they were found to be identical! Wonderful.
cp /cru/mark1/f080/gts/pre/glo_quick_log/glo.pre.norm.Z /cru/cruts/rerun1/data/cruts/rerun_synth/
cp /cru/mark1/f080/gts/rad/glo/glo.rad.norm.Z /cru/cruts/rerun1/data/cruts/rerun_synth/
cp /cru/mark1/f080/gts/rd0/glo/glo.rd0.norm.Z /cru/cruts/rerun1/data/cruts/rerun_synth/
There were two 'sunp' norm files, but one was 0 bytes in length.
cp /cru/mark1/f080/gts/sunp/means/glo.sunp.norm.Z /cru/cruts/rerun1/data/cruts/rerun_synth/
cp /cru/mark1/f080/gts/tmp/glo/glo.tmp.norm.Z /cru/cruts/rerun1/data/cruts/rerun_synth/
cp /cru/mark1/f080/gts/vap/glo/glo.vap.norm.Z /cru/cruts/rerun1/data/cruts/rerun_synth/
cp /cru/mark1/f080/gts/wnd/glo/glo.wnd.norm.Z /cru/cruts/rerun1/data/cruts/rerun_synth/
The synthetics generation was then re-run for frs (records above have
been modified to reflect this).
Next, rd0. Synthetics generated OK..
IDL> rd0_gts,1901,2002,1961,1990,outprefix="syngrid_rd0/syngrid_rd0",pre_prefix="idlbin_pre/idlbin_pre"
..until the end:
2001
yes
filesize= 248832
gridsize= 2.50000
2002
yes
filesize= 248832
gridsize= 2.50000
% Program caused arithmetic error: Floating divide by 0
% Program caused arithmetic error: Floating illegal operand
IDL>
However, all synthetic grids appear to have been written OK, including 2002.
Grid generation proceeded without error:
IDL> quick_interp_tdm2,1901,2002,'glo_rd0_grids/rd0.grid.',450,gs=0.5,dumpglo='dumpglo',nostn=1,synth_prefix='syngrid_rd0/syngrid_rd0'
Onto vapour pressure, and the crunch. For here, the recommended program for
synthetic grid production is 'vap_gts_anom.pro'. In fact, there is no sign
of a 'vap_gts_tdm.pro'. And, in the program notes, it reads:
; required inputs are:
; ** vapour pressure and temperature normals on 2.5deg grid
; (these come ready-supplied for a 1961-90 normal period)
; ** temp and dtr monthly anomalies on 2.5deg grid, including normal period
So, we face a situation where some synthetics are built with 0.5-degree
normals, and others are built with 2.5-degree normals. I can find no
documentation of this. There are '*_anom.pro' versions of the frs and rd0
programs, both of which use 2.5-degree normals, however they are dated
Jan 2004, and Tim's Read_Me (which refers to the '*_tdm.pro' 0.5-degree
versions) is dated end March 2004, so we have to assume these are his
best suggestions.
The 2.5 normals are found here:
> ls -l /cru/cruts/fromdpe1a/data/grid/twohalf/
total 1248
-rwxr-xr-x 1 f098 cru 248832 Jan 9 2004 glo25.frs.6190
-rwxr-xr-x 1 f098 cru 248832 Jan 8 2004 glo25.pre.6190
-rwxr-xr-x 1 f098 cru 248832 Jan 8 2004 glo25.rd0.6190
-rwxr-xr-x 1 f098 cru 248832 Jan 7 2004 glo25.tmp.6190
-rwxr-xr-x 1 f098 cru 248832 Jan 6 2004 glo25.vap.6190
-rwxr-xr-x 1 f098 cru 86 Feb 25 2004 readme.txt
readme.txt:
2.5deg climatology files
Tim Mitchell, 25.2.04
These are in Mark New's binary format
(end)
Set up the required inputs, and ran it:
IDL> vap_gts_anom,dtr_prefix='idlbin_dtr/idlbin_dtr',tmp_prefix='idlbin_tmp/idlbin_tmp',1901,2002,outprefix='syngrid_vap/syngrid_vap',dumpbin=1
Producing screen output like this:
1991 vap (x,s2,<<,>>): 0.000493031 0.000742087 -0.0595093 1.86497
And output files like this:
-rw------- 1 f098 cru 248832 Sep 22 10:56 syngrid_vap/syngrid_vap1991
On, without further ado, to the gridding. For this secondary, there *are* database
files, so the 'nostn' option is not used, and anomdtb.f is wheeled out again
to construct .txt files for the run:
crua6[/cru/cruts/rerun1/data/cruts/rerun_vap] ./anomdtb
> ***** AnomDTB: converts .dtb to anom .txt for gridding *****
> Enter the suffix of the variable required:
.vap
> Select the .cts or .dtb file to load:
vap.0311181410.dtb
> Specify the start,end of the normals period:
1961,1990
> Specify the missing percentage permitted:
25
> Data required for a normal: 23
> Specify the no. of stdevs at which to reject data:
3
> Select outputs (1=.cts,2=.ann,3=.txt,4=.stn):
3
> Check for duplicate stns after anomalising? (0=no,>0=km range)
8
> Select the generic .txt file to save (yy.mm=auto):
vap.txt
> Select the first,last years AD to save:
1901,2002
> Operating...
Values loaded: 1239868112; No. Stations: 7691
> NORMALS MEAN percent STDEV percent
> .dtb 887754 46.9
> .cts 34175 1.8 921929 48.7
> PROCESS DECISION percent %of-chk
> no lat/lon 105 0.0 0.0
> no normal 969384 51.3 51.3
> out-of-range 2661 0.1 0.3
> duplicated 25557 1.4 2.8
> accepted 893711 47.3
> Dumping years 1901-2002 to .txt files...
crua6[/cru/cruts/rerun1/data/cruts/rerun_vap]
Moved straight onto the gridding, which, of course, failed:
IDL> quick_interp_tdm2,1901,2002,'glo_vap_grids/vap.grid.',1000,gs=0.5,dumpglo='dumpglo',synth_prefix='syngrid_vap/syngrid_vap',pts_prefix='../rerun_vap/vap_txt_4idl/vap.'
Defaults set
1901
1902
% Array dimensions must be greater than 0.
% Execution halted at: QUICK_INTERP_TDM2 88 /cru/cruts/fromdpe1a
/code/idl/pro/quick_interp_tdm2.pro
% QUICK_INTERP_TDM2 88 /cru/cruts/fromdpe1a
/code/idl/pro/quick_interp_tdm2.pro
% $MAIN$
IDL>
This turns out to be because of the sparcity of VAP station measurements in the
early years. The program cannot handle anom files of 0 length, even though it
checks the length! Bizarre. The culprit is 'vap.1902.03.txt', the only month to
have no station reading at all (45 months have only 1 however). I decided to mod
the program to use the 'nostn' option if the length is 0. Hope that's right - the
synthetics are read in first and the station data is added to that grid so this
should be OK.. and it looks OK:
IDL> quick_interp_tdm2,1901,2002,'vap.grid.',1000,gs=0.5,dumpglo='dumpglo',synth_prefix='syngrid_vap/syngrid_vap',pts_prefix='../rerun_vap/vap_txt_4idl/vap.'
% Compiled module: GLIMIT.
Defaults set
1901
1902
no stations found in: ../rerun_vap/vap_txt_4idl/vap.1902.03.txt
1903
(..etc..)
Pause for reflection: the list of CRU_TS_2.1 parameters is as follows:
pre primary, done
tmp primary, done
tmx derived, not done
tmn derived, not done
dtr primary, done
vap secondary, done
cld/spc secondary, not done
wet secondary, done
frs secondary, done
Now the interesting thing is that the 'Read Me' file for gridding only
mentions frs, rd0 (which I'm assuming == wet) and vap. How, then, do I
produce cld/spc and the two derived vars??
Well, there's a /cru/cruts/fromdpe1a/code/idl/pro/cal_cld_gts_tdm.pro,
also:
/cru/cruts/fromdpe1a
/code/idl/pro/cloudcorrspc.pro
/cru/cruts/fromdpe1a
/code/idl/pro/cloudcorrspcann.pro
/cru/cruts/fromdpe1a
/code/idl/pro/cloudcorrspcann9196.pro
Loading just the first program opens up another huge can o' worms. The
program description reads:
pro cal_cld_gts_tdm,dtr_prefix,outprefix,year1,year2,info=info
; calculates cld anomalies using relationship with dtr anomalies
; reads coefficients from predefined files (*1000)
; reads DTR data from binary output files from quick_interp_tdm2.pro (binfac=1000)
; creates cld anomaly grids at dtr grid resolution
; output can then be used as dummy input to splining program that also
; includes real cloud anomaly data
So, to me this identifies it as the program we cannot use any more because
the coefficients were lost. As it says in the gridding read_me:
Bear in mind that there is no working synthetic method for cloud, because Mark New
lost the coefficients file and never found it again (despite searching on tape
archives at UEA) and never recreated it. This hasn't mattered too much, because
the synthetic cloud grids had not been discarded for 1901-95, and after 1995
sunshine data is used instead of cloud data anyway.
But, (Lord how many times have I used 'however' or 'but' in this file?!!), when
you look in the program you find that the coefficient files are called:
rdbin,a,'/cru/tyn1/f709762/cru_ts_2.0/_constants/_7190/a.25.7190',gridsize=2.5
rdbin,b,'/cru/tyn1/f709762/cru_ts_2.0/_constants/_7190/b.25.7190',gridsize=2.5
And, if you do a search over the filesystems, you get:
crua6[/cru/cruts] ls fromdpe1a/data/grid/cru_ts_2.0/_makecld/_constants/_7190/spc2cld/_ann/
a.25.01.7190.glo.Z a.25.05.7190.glo.Z a.25.09.7190.glo.Z a.25.7190.eps.Z b.25.04.7190.glo.Z b.25.08.7190.glo.Z b.25.12.7190.glo.Z
a.25.02.7190.glo.Z a.25.06.7190.glo.Z a.25.10.7190.glo.Z b.25.01.7190.glo.Z b.25.05.7190.glo.Z b.25.09.7190.glo.Z b.25.7190.eps.Z
a.25.03.7190.glo.Z a.25.07.7190.glo.Z a.25.11.7190.glo.Z b.25.02.7190.glo.Z b.25.06.7190.glo.Z b.25.10.7190.glo.Z
a.25.04.7190.glo.Z a.25.08.7190.glo.Z a.25.12.7190.glo.Z b.25.03.7190.glo.Z b.25.07.7190.glo.Z b.25.11.7190.glo.Z
crua6[/cru/cruts] ls fromdpe1a/data/grid/cru_ts_2.0/_makecld/_constants/_7190/spc2cld/_mon/
a.25.01.7190.glo.Z a.25.05.7190.glo.Z a.25.09.7190.glo.Z a.25.7190.eps.Z b.25.04.7190.glo.Z b.25.08.7190.glo.Z b.25.12.7190.glo.Z
a.25.02.7190.glo.Z a.25.06.7190.glo.Z a.25.10.7190.glo.Z b.25.01.7190.glo.Z b.25.05.7190.glo.Z b.25.09.7190.glo.Z b.25.7190.eps.Z
a.25.03.7190.glo.Z a.25.07.7190.glo.Z a.25.11.7190.glo.Z b.25.02.7190.glo.Z b.25.06.7190.glo.Z b.25.10.7190.glo.Z
a.25.04.7190.glo.Z a.25.08.7190.glo.Z a.25.12.7190.glo.Z b.25.03.7190.glo.Z b.25.07.7190.glo.Z b.25.11.7190.glo.Z
So.. we don't have the coefficients files (just .eps plots of something). But
what are all those monthly files? DON'T KNOW, UNDOCUMENTED. Wherever I look,
there are data files, no info about what they are other than their names. And
that's useless.. take the above example, the filenames in the _mon and _ann
directories are identical, but the contents are not. And the only difference
is that one directory is apparently 'monthly' and the other 'annual' - yet
both contain monthly files.
Lots of further investigation.. probably the most useful program found is
cal_cld_gts_tdm.pro, the description of which reads as follows:
pro cal_cld_gts_tdm,dtr_prefix,outprefix,year1,year2,info=info
; calculates cld anomalies using relationship with dtr anomalies
; reads coefficients from predefined files (*1000)
; reads DTR data from binary output files from quick_interp_tdm2.pro (binfac=1000)
; creates cld anomaly grids at dtr grid resolution
; output can then be used as dummy input to splining program that also
; includes real cloud anomaly data
It also tellingly contains:
; unnecessary because 61-90 normals have already been created
; print, "@@@@@ looking for 2.5 deg DTR 1961-90 @@@@@"
; mean_gts,'~/m1/gts/dtr/glo25/glo25.dtr.',nor1,nor2
; mean_gts_tdm,'/cru/mark1/f080/gts/dtr/glo25/glo25.dtr.',nor1,nor2
;print, "@@@@@ looking for 2.5 deg DTR normal @@@@@"
;; rdbin,dtrnor,'~/m1/gts/dtr/glo25/glo25.dtr.'+string(nor1-1900,nor2-1900,form='(2i2.2)')
;dtrnorstr='/cru/mark1/f080/gts/dtr/glo25/glo25.dtr.'+string(nor1-1900,nor2-1900,form='(2i2.2)')
;rdbin,dtrnor,dtrnorstr
The above has seemingly been replaced with:
rdbin,a,'/cru/tyn1/f709762/cru_ts_2.0/_constants/_7190/a.25.7190',gridsize=2.5
rdbin,b,'/cru/tyn1/f709762/cru_ts_2.0/_constants/_7190/b.25.7190',gridsize=2.5
These are the files that have been lost according to the gridding read_me
(see above).
The conclusion of a lot of investigation is that the synthetic cloud grids
for 1901-1995 have now been discarded. This means that the cloud data prior
to 1996 are static.
Edit: have just located a 'cld' directory in Mark New's disk, containing
over 2000 files. Most however are binary and undocumented..
Eventually find fortran (f77) programs to convert sun to cloud:
sh2cld_tdm.for converts sun hours monthly time series to cloud percent
sp2cld_m.for converts sun percent monthly time series to cloud oktas
There are also programs to convert sun parameters:
sh2sp_m.for sun hours to sun percent
sh2sp_normal.for sun hours monthly .nrm to sunshine percent
sh2sp_tdm.for sun hours monthly time series to sunshine percent
AGREED APPROACH for cloud (5 Oct 06).
For 1901 to 1995 - stay with published data. No clear way to replicate
process as undocumented.
For 1996 to 2002:
1. convert sun database to pseudo-cloud using the f77 programs;
2. anomalise wrt 96-00 with anomdtb.f;
3. grid using quick_interp_tdm.pro (which will use 6190 norms);
4. calculate (mean9600 - mean6190) for monthly grids, using the
published cru_ts_2.0 cloud data;
5. add to gridded data from step 3.
This should approximate the correction needed.
On we go.. firstly, examined the spc database.. seems to be in % x10.
Looked at published data.. cloud is in % x10, too.
First problem: there is no program to convert sun percentage to
cloud percentage. I can do sun percentage to cloud oktas or sun hours
to cloud percentage! So what the hell did Tim do?!! As I keep asking.
Examined the program that converts sun % to cloud oktas. It is
complicated! Have inserted a line to multiple the result by 12.5 (the
result is in oktas*10 and ranges from 0 to 80, so the new result will
range from 0 to 1000).
Next problem - which database to use? The one with the normals included
is not appropriate (the conversion progs do not look for that line so
obviously are not intended to be used on +norm databases). The non
normals databases are either Jan 03 (in the '_ateam' directory) or
Dec 03 (in the regular database directory). The newer database is
smaller! So more weeding than planting in 2003. Unfortunately both
databases contain the 6190 normals line, just unpopulated. So I will
go with the 'spc.0312221624.dtb' database, and modify the already-
modified conversion program to process the 6190 line.
Then - comparing the two candidate spc databases:
spc.0312221624.dtb
spc.94-00.0312221624.dtb
I find that they are broadly similar, except the normals lines (which
both start with '6190') are very different. I was expecting that maybe
the latter contained 94-00 normals, what I wasn't expecting was that
thet are in % x10 not %! Unbelievable - even here the conventions have
not been followed. It's botch after botch after botch. Modified the
conversion program to process either kind of normals line.
Decided to go with the 'spc.94-00.0312221624.dtb' database, as it
hopefully has some of the 94-00 normals in. I just wish I knew more.
Conversion was hampered by the discovery that some stations have a mix
of % and % x10 values! So more mods to Hsp2cldp_m.for. Then conversion,
producing cldfromspc.94000312221624.dtb. Copied the .dts file across
as is, not sure what it does unfortunately (or can't remember!).
After conversion, ran anomdtb:
crua6[/cru/cruts/rerun1/data/cruts/rerun_cld] ./anomdtb
> ***** AnomDTB: converts .dtb to anom .txt for gridding *****
> Enter the suffix of the variable required:
.cld
> Select the .cts or .dtb file to load:
cldfromspc.94000312221624.dtb
> Specify the start,end of the normals period:
1994,2000
> Specify the missing percentage permitted:
25
> Data required for a normal: 6
> Specify the no. of stdevs at which to reject data:
3
> Select outputs (1=.cts,2=.ann,3=.txt,4=.stn):
3
> Check for duplicate stns after anomalising? (0=no,>0=km range)
8
> Select the generic .txt file to save (yy.mm=auto):
cldfromspc.txt
> Select the first,last years AD to save:
1994,2002
> Operating...
> .cts 96309 19.6 280712 57.2
> PROCESS DECISION percent %of-chk
> no lat/lon 0 0.0 0.0
> no normal 209619 42.8 42.8
> out-of-range 177298 36.2 63.2
> duplicated 154 0.0 0.1
> accepted 103260 21.1
> Dumping years 1994-2002 to .txt files...
crua6[/cru/cruts/rerun1/data/cruts/rerun_cld]
Then ran quick_interp_tdm2:
IDL> .compile /cru/cruts/fromdpe1a
/code/idl/pro/quick_interp_tdm2.pro
% Compiled module: QUICK_INTERP_TDM2.
IDL> .compile /cru/cruts/fromdpe1a/code/idl/pro/rdbin.pro
% Compiled module: RDBIN.
IDL> quick_interp_tdm2,1994,2002,'glo_from_idl/cld.',600,gs=0.5,pts_prefix='txt_4_idl/cldfromspc.',dumpglo='dumpglo'
Defaults set
1994
% Compiled module: MAP_SET.
% Compiled module: CROSSP.
% Compiled module: STRIP.
% Compiled module: SAVEGLO.
% Compiled module: SELECTMODEL.
1995
1996
1997
1998
1999
2000
2001
2002
IDL>
Tadaa: .glo files produced for 1994 to 2002.
Then retracked to produce regular 0.5-degree grids for dtr (having only
produced 2.5-degree binaries for synthetics earlier):
IDL> quick_interp_tdm2,1901,2002,'glo_dtr_grids/dtr.',750,gs=0.5,pts_prefix='dtr_txt_4idl/dtr.',dumpglo='dumpglo'
That went off without any apparent hitches, so I wrote a fortran prog,
'maxminmaker.for', to produce tmn and tmx grids from tmp and dtr. It ran.
However - yup, more problems - when I checked the inputs and outputs I found
that in numerous instances there was a value for mean temperature in the grid,
with no corresponding dtr value. This led to tmn = tmx = tmp for thos cells.
NOT GOOD.
Actually, what was NOT GOOD was my grasp of context. Oh curse this poor
memory! For the IDL gridding program produces ANOMALIES not ACTUALS.
Wrote a program, 'glo2abs.for' does a file-for-file conversion of .glo
files (as produced by quick_interp_tdm2.pro) to absolute-value files (also
gridded and with headers). After some experiments realised that the .glo
anomalies are in degrees, but the normals are in 10ths of a degree :-)
Produced absolutes for TMP. Then wrote a program, 'cmpcruts.for', to
compare the absolute grids with the published cru_ts_2.10 data. The
comparison simply measures absolute differences between old and new, and
categorises as either (1) identical, (2) within 0.5 degs, (3) within 1 deg,
(4) over 1 deg apart. Results for temperature (TMP):
Identical <0.5deg 0.5-1deg >1deg
30096176 48594200 2755281 1076423
And for temperature range (DTR):
45361058 31267870 3893754 1999398
These are very promising. The vast majority in both cases are within 0.5
degrees of the published data. However, there are still plenty of values
more than a degree out.
The total number of comparisons is 67420*102*12 = 82,522,080
It seems prudent to add percentage calculations..
TMP:
Final Diff Totals: 30096176 48594200 2755281 1076423
Percentages: 36.47 58.89 3.34 1.30
TMP has a comforting 95%+ within half a degree, though one still wonders
why it isn't 100% spot on..
DTR:
Final Diff Totals: 45361058 31267870 3893754 1999398
Percentages: 54.97 37.89 4.72 2.42
DTR fares perhaps even better, over half are spot-on, though about
7.5% are outside a half.
However, it's not such good news for precip (PRE):
Final Diff Totals: 11492331 21163924 9264554 40601271
Percentages: 13.93 25.65 11.23 49.20