The HARRY_READ_ME.txt file

So where were we.. Rain Days. Family tree:

wet.0311061611.dtb
+
rdy.0709111032.dtb (MCDW composite)
+
rdy.0710151817.dtb (CLIMAT composite with metadata added)
V
V
wet.0710161148.dtb


Now it gets tough. The current model for a secondary is that it is derived from one or more primaries,
plus their normals, plus the normals for the secondary.

The IDL secondary generators do not allow 'genuine' secondary data to be incorporated. This would have
been ideal, as the gradual increase in observations would have gradually taken precedence over the
primary-derived synthetics.

The current stats for the wet database were derived from the new proglet, dtbstats.for:

<BEGIN QUOTE>
crua6[/cru/cruts/version_3_0/secondaries/rd0] ./dtbstat

DTBSTAT: Database Stats Report

Please enter the (18ch.) database name: wet.0710161148.dtb

Report for: wet.0710161148.dtb

Stations in Northern Hemisphere: 5365
Stations in Southern Hemisphere: 778
Total: 6143

Maximum Timespan in Northern Hemisphere: 1840 to 2007
Maximum Timespan in Southern Hemisphere: 1943 to 2007
Global Timespan: 1840 to 2007

crua6[/cru/cruts/version_3_0/secondaries/rd0]
<END QUOTE>

So, without further ado, I treated RD0 as a Primary and derived gridded output from the database:

<BEGIN QUOTE>
crua6[/cru/cruts/version_3_0/secondaries/rd0] ./anomdtb

> ***** AnomDTB: converts .dtb to anom .txt for gridding *****

> Enter the suffix of the variable required:
.rd0
> Select the .cts or .dtb file to load:
wet.0710161148.dtb
> Specify the start,end of the normals period:
1961,1990
> Specify the missing percentage permitted:
25
> Data required for a normal: 23
> Specify the no. of stdevs at which to reject data:
3
> Select outputs (1=.cts,2=.ann,3=.txt,4=.stn):
3
> Check for duplicate stns after anomalising? (0=no,>0=km range)
0
> Select the generic .txt file to save (yy.mm=auto):
rd0.txt
> Select the first,last years AD to save:
1901,2007
> Operating...

> NORMALS MEAN percent STDEV percent
> .dtb 0 0.0
> .cts 731118 45.4 730956 45.4
> PROCESS DECISION percent %of-chk
> no lat/lon 0 0.0 0.0
> no normal 878015 54.6 54.6
> out-of-range 56 0.0 0.0
> accepted 731062 45.4
> Dumping years 1901-2007 to .txt files...

crua6[/cru/cruts/version_3_0/secondaries/rd0]
<END QUOTE>

Not particularly good - the bulk of the data being recent, less than half had valid normals (anomdtb
calculates normals on the fly, on a per-month basis). However, this isn't so much of a problem as the
plan is to screen it for valid station contributions anyway.

<BEGIN QUOTE>
IDL> quick_interp_tdm2,1901,2007,'rd0glo/rd0.',450,gs=0.5,dumpglo='dumpglo',pts_prefix='rd0txt/rd0.'
% Compiled module: QUICK_INTERP_TDM2.
% Compiled module: GLIMIT.
Defaults set
1901
% Compiled module: MAP_SET.
% Compiled module: CROSSP.
% Compiled module: STRIP.
% Compiled module: SAVEGLO.
% Compiled module: SELECTMODEL.
1902
(etc)
2007
no stations found in: rd0txt/rd0.2007.08.txt
no stations found in: rd0txt/rd0.2007.09.txt
no stations found in: rd0txt/rd0.2007.10.txt
no stations found in: rd0txt/rd0.2007.11.txt
no stations found in: rd0txt/rd0.2007.12.txt
IDL>
<END QUOTE>

<BEGIN QUOTE>
crua6[/cru/cruts/version_3_0/secondaries/rd0] ./glo2abs
Welcome! This is the GLO2ABS program.
I will create a set of absolute grids from
a set of anomaly grids (in .glo format), also
a gridded version of the climatology.
Enter the path and name of the normals file: clim.6190.lan.wet
Enter a name for the gridded climatology file: clim.6190.lan.wet.grid2
Enter the path and stem of the .glo files: rd0glo/rd0.
Enter the starting year: 1901
Enter the ending year: 2007
Enter the path (if any) for the output files: rd0abs/
Now, CONCENTRATE. Addition or Percentage (A/P)? A ! this was a guess! We'll see how the results look
Right, erm.. off I jolly well go!
rd0.01.1901.glo
(etc)
<END QUOTE>

Then.. wait a minute! I checked back, and sure enough, quick_interp_tdm.pro DOES allow both synthetic and 'real' data
to be included in the gridding. From the program description:

<BEGIN QUOTE>
; TDM: the dummy grid points default to zero, but if the synth_prefix files are present in call,
; the synthetic data from these grids are read in and used instead
<END QUOTE>

And so.. (after some confusion, and renaming so that anomdtb selects percentage anomalies)..

IDL> quick_interp_tdm2,1901,2006,'rd0pcglo/rd0pc',450,gs=0.5,dumpglo='dumpglo',synth_prefix='rd0syn/rd0syn',pts_prefix='rd0pctxt/rd0pc.'

The trouble is, we won't be able to produce reliable station count files this way. Or can we use the same strategy,
producing station counts from the wet database route, and filling in 'gaps' with the precip station counts? Err.

<BEGIN QUOTE>
crua6[/cru/cruts/version_3_0/secondaries/rd0] ./glo2abs
Welcome! This is the GLO2ABS program.
I will create a set of absolute grids from
a set of anomaly grids (in .glo format), also
a gridded version of the climatology.
Enter the path and name of the normals file: clim.6190.lan.wet
Enter a name for the gridded climatology file: clim.grid
Enter the path and stem of the .glo files: rd0pcglo/rd0pc.
Enter the starting year: 1901
Enter the ending year: 2006
Enter the path (if any) for the output files: rd0pcgloabs/
Now, CONCENTRATE. Addition or Percentage (A/P)? P
Right, erm.. off I jolly well go!
rd0pc.01.1901.glo
(etc)
<END QUOTE>

<BEGIN QUOTE>
crua6[/cru/cruts/version_3_0/secondaries/rd0] ./mergegrids
Welcome! This is the MERGEGRIDS program.
I will create decadal and full gridded files
from the output files of (eg) glo2abs.for.

Enter a gridfile with YYYY for year and MM for month: rd0pcgloabs/rd0pc.MM.YYYY.glo.abs
Enter Start Year: 1901
Enter Start Month: 01
Enter End Year: 2006
Enter End Month: 12

Please enter a sample OUTPUT filename, replacing
start year with SSSS and end year with EEEE: cru_ts_3_00.SSSS.EEEE.rd0.dat
Writing cru_ts_3_00.1901.1910.rd0.dat
Writing cru_ts_3_00.1911.1920.rd0.dat
Writing cru_ts_3_00.1921.1930.rd0.dat
Writing cru_ts_3_00.1931.1940.rd0.dat
Writing cru_ts_3_00.1941.1950.rd0.dat
Writing cru_ts_3_00.1951.1960.rd0.dat
Writing cru_ts_3_00.1961.1970.rd0.dat
Writing cru_ts_3_00.1971.1980.rd0.dat
Writing cru_ts_3_00.1981.1990.rd0.dat
Writing cru_ts_3_00.1991.2000.rd0.dat
Writing cru_ts_3_00.2001.2006.rd0.dat
crua6[/cru/cruts/version_3_0/secondaries/rd0]
<END QUOTE>

All according to plan.. except the values themselves!

For January, 2001:

Minimum = 0
Maximum = 32630
Vals >31000 = 1

For the whole of 2001:

Minimum = 0
Maximum = 56763
Vals >31000 = 5

Not good. We're out by a factor of at least 10, though the extremes are few enough to just cap at DiM. So where has
this factor come from?

Well here's the January 2001 climatology:

Minimum = 0
Maximum = 3050
Vals >3100 = 0

That all seems fine for a percentage normals set. Not entirly sure about 0 though.

so let's look at the January 2001 gridded anomalies file:

Minimum = -48.046
Maximum = 0.0129

This leads to a show-stopper, I'm afraid. It looks as though the calculation I'm using for percentage anomalies is,
not to put too fine a point on it, cobblers.

This is what I use to build actuals from anomalies in glo2abs.for:

absgrid(ilon(i),ilat(i)) = nint(normals(i,imo) +
* anoms(ilon(i),ilat(i)) * normals(i,imo) / 100)

or, to put it another way, V = N(A+N)/100

This is what anomdtb.f90 uses to build anomalies from actuals:

DataA(XAYear,XMonth,XAStn) = nint(1000.0*((real(DataA(XAYear,XMonth,XAStn)) / &
real(NormMean(XMonth,XAStn)))-1.0))

or, in the same terms, A = 1000((V/N)-1)

which reverses to: V = N(A+1000)/1000

This could well explain things. It could also mean that I have to reproduce v3.00 precip AFTER it's been used (against
my wishes) by Dave L and Dimitrious.

Well to start with, I'll try the new calculation in glo2abs to reproduce the rd0 data.

<BEGIN QUOTE>
crua6[/cru/cruts/version_3_0/secondaries/rd0] ./glo2abs
Welcome! This is the GLO2ABS program.
I will create a set of absolute grids from
a set of anomaly grids (in .glo format), also
a gridded version of the climatology.
Enter the path and name of the normals file: clim.6190.lan.wet
Enter a name for the gridded climatology file: c.grid
Enter the path and stem of the .glo files: rd0pcglo/rd0pc.
Enter the starting year: 1901
Enter the ending year: 2006
Enter the path (if any) for the output files: rd0pcgloabs
Now, CONCENTRATE. Addition or Percentage (A/P)? P
Right, erm.. off I jolly well go!
rd0pc.01.1901.glo
(etc)
<END QUOTE>

This *does* improve matters considerably. Now, for January 2001:

Minimum = 0
Maximum = 5090 (a little high but not fatal)
Vals >3100 = 556
Vals >3500 = 110
Vals >4000 = 2 (so the bulk of the excessions are only a few days over)

In fact the 2nd highest Max is 4369, well below 5090.

So, good news - but only in the sense that I've found the error. Bad news in that it's a further confirmation that my
abilities are short of what's required here.

Rushed back to precip. Found the .glo files in /cru/cruts/version_3_0/primaries/precip/pre0km0612181221glo/, and
re-ran glo2abs with the revised percentage anomaly equation:

<BEGIN QUOTE>
crua6[/cru/cruts/version_3_0/primaries/precip] ./glo2abs
Welcome! This is the GLO2ABS program.
I will create a set of absolute grids from
a set of anomaly grids (in .glo format), also
a gridded version of the climatology.
Enter the path and name of the normals file: clim.6190.lan.pre
Enter a name for the gridded climatology file: clim.6190.lan.pre.gridded2
Enter the path and stem of the .glo files: pre0km0612181221glo/pregrid.
Enter the starting year: 1901
Enter the ending year: 2006
Enter the path (if any) for the output files: pre0km0612181221abs/
Now, CONCENTRATE. Addition or Percentage (A/P)? P
Right, erm.. off I jolly well go!
pregrid.01.1901.glo
(etc)
<END QUOTE>

<BEGIN QUOTE>
crua6[/cru/cruts/version_3_0/primaries/precip] ./mergegrids
Welcome! This is the MERGEGRIDS program.
I will create decadal and full gridded files
from the output files of (eg) glo2abs.for.

Enter a gridfile with YYYY for year and MM for month: pre0km0612181221abs/pregrid.MM.YYYY.glo.abs
Enter Start Year: 1901
Enter Start Month: 01
Enter End Year: 2006
Enter End Month: 12

Please enter a sample OUTPUT filename, replacing
start year with SSSS and end year with EEEE: cru_ts_3_00.SSSS.EEEE.pre.dat
Writing cru_ts_3_00.1901.1910.pre.dat
Writing cru_ts_3_00.1911.1920.pre.dat
Writing cru_ts_3_00.1921.1930.pre.dat
Writing cru_ts_3_00.1931.1940.pre.dat
Writing cru_ts_3_00.1941.1950.pre.dat
Writing cru_ts_3_00.1951.1960.pre.dat
Writing cru_ts_3_00.1961.1970.pre.dat
Writing cru_ts_3_00.1971.1980.pre.dat
Writing cru_ts_3_00.1981.1990.pre.dat
Writing cru_ts_3_00.1991.2000.pre.dat
Writing cru_ts_3_00.2001.2006.pre.dat
crua6[/cru/cruts/version_3_0/primaries/precip]
<END QUOTE>

Then back to finish off rd0. Modified glo2abs to allow the operator to set minima and maxima, with a
specific option to set wet day limits (DiM*100):

<BEGIN QUOTE>
crua6[/cru/cruts/version_3_0/secondaries/rd0] ./glo2abs
Welcome! This is the GLO2ABS program.
I will create a set of absolute grids from
a set of anomaly grids (in .glo format), also
a gridded version of the climatology.
Enter the path and name of the normals file: clim.6190.lan.wet
Enter a name for the gridded climatology file: clim...grid
Enter the path and stem of the .glo files: rd0pcglo/rd0pc.
Enter the starting year: 1901
Enter the ending year: 2006
Enter the path (if any) for the output files: rd0pcgloabs/
Now, CONCENTRATE. Addition or Percentage (A/P)? P
Do you wish to limit the output values? (Y/N): Y
1. Set minimum to zero
2. Set a single minimum and maximum
3. Set monthly minima and maxima (for wet/rd0)

Choose: 3
Right, erm.. off I jolly well go!
rd0pc.01.1901.glo
(etc)
<END QUOTE>

Output was checked.. and as expected, January 2001 had 556 values of 3100 :-)

<BEGIN QUOTE>
crua6[/cru/cruts/version_3_0/secondaries/rd0] ./mergegrids
Welcome! This is the MERGEGRIDS program.
I will create decadal and full gridded files
from the output files of (eg) glo2abs.for.

Enter a gridfile with YYYY for year and MM for month: rd0pcgloabs/rd0pc.MM.YYYY.glo.abs
Enter Start Year: 1901
Enter Start Month: 01
Enter End Year: 2006
Enter End Month: 12

Please enter a sample OUTPUT filename, replacing
start year with SSSS and end year with EEEE: cru_ts_3_00.SSSS.EEEE.rd0.dat
Writing cru_ts_3_00.1901.1910.rd0.dat
Writing cru_ts_3_00.1911.1920.rd0.dat
Writing cru_ts_3_00.1921.1930.rd0.dat
Writing cru_ts_3_00.1931.1940.rd0.dat
Writing cru_ts_3_00.1941.1950.rd0.dat
Writing cru_ts_3_00.1951.1960.rd0.dat
Writing cru_ts_3_00.1961.1970.rd0.dat
Writing cru_ts_3_00.1971.1980.rd0.dat
Writing cru_ts_3_00.1981.1990.rd0.dat
Writing cru_ts_3_00.1991.2000.rd0.dat
Writing cru_ts_3_00.2001.2006.rd0.dat
crua6[/cru/cruts/version_3_0/secondaries/rd0]
<END QUOTE>

Back to where this all started - Vapour Pressure.

We have:

1. 'Master' (ie original) database vap.0311181410.dtb
2. MCDW updates database vap.0709111032.dtb
3. CLIMAT updates database *with added metadata* vap.0710151817.dtb

so first we incorporate the MCDW updates..

<BEGIN QUOTE>
uealogin1[/cru/cruts/version_3_0/db/vap] ./newmergedb

WELCOME TO THE DATABASE UPDATER

Before we get started, an important question:
If you are merging an update - CLIMAT, MCDW, Australian - do
you want the quick and dirty approach? This will blindly match
on WMO codes alone, ignoring data/metadata checks, and making any
unmatched updates into new stations (metadata permitting)?

Enter 'B' for blind merging, or <ret>: B
Please enter the Master Database name: vap.0311181410.dtb
Please enter the Update Database name: vap.0709111032.dtb

Reading in both databases..
Master database stations: 7691
Update database stations: 2398

Looking for WMO code matches..
2 reject(s) from update process 0710241541

Writing vap.0710241541.dtb

+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+
+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+

OUTPUT(S) WRITTEN

New master database: vap.0710241541.dtb

Update database stations: 2398
> Matched with Master stations: 1847
(automatically: 1847)
(by operator: 0)
> Added as new Master stations: 549
> Rejected: 2
Rejects file: vap.0709111032.dtb.rejected
uealogin1[/cru/cruts/version_3_0/db/vap]
<END QUOTE>

Then, the CLIMAT ones:

<BEGIN QUOTE>
uealogin1[/cru/cruts/version_3_0/db/vap] ./newmergedb

WELCOME TO THE DATABASE UPDATER

Before we get started, an important question:
If you are merging an update - CLIMAT, MCDW, Australian - do
you want the quick and dirty approach? This will blindly match
on WMO codes alone, ignoring data/metadata checks, and making any
unmatched updates into new stations (metadata permitting)?

Enter 'B' for blind merging, or <ret>: B
Please enter the Master Database name: vap.0710241541.dtb
Please enter the Update Database name: vap.0710151817.dtb

Reading in both databases..
Master database stations: 8240
Update database stations: 2870

Looking for WMO code matches..
68 reject(s) from update process 0710241549

Writing vap.0710241549.dtb

+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+
+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+

OUTPUT(S) WRITTEN

New master database: vap.0710241549.dtb

Update database stations: 2870
> Matched with Master stations: 2599
(automatically: 2599)
(by operator: 0)
> Added as new Master stations: 203
> Rejected: 68
Rejects file: vap.0710151817.dtb.rejected
uealogin1[/cru/cruts/version_3_0/db/vap]
<END QUOTE>

So, not as good as the MCDW update.. lost 68.. but then of course we are talking about station data that
arrived with NO metadata AT ALL.

So we will try the unaltered rd0 process on vap. It should be the same; a mix of synthetic and observed.