The HARRY_READ_ME.txt file

Part 30

30. Being an idiot much experience I decided to go back to the 'perfectly-good' precip
generation for v3.0 and re-do the anomalies with the new anomdtb. At 8km, we got the
duplicates down from 5.9% to 2.1%:


> NORMALS MEAN percent STDEV percent
> .dtb 7315040 73.8
made it to here
> .cts 299359 3.0 7613600 76.8
> PROCESS DECISION percent %of-chk
> no lat/lon 17527 0.2 0.2
> no normal 2355659 23.8 23.8
> out-of-range 13253 0.1 0.2
> duplicated 586206 5.9 7.8
> accepted 6934807 70.0
> Dumping years 1901-2006 to .txt files...


> NORMALS MEAN percent STDEV percent
> .dtb 7315040 73.8
made it to here
> .cts 299359 3.0 7613600 76.8
> PROCESS DECISION percent %of-chk
> no lat/lon 17527 0.2 0.2
> no normal 2355659 23.8 23.8
> out-of-range 13253 0.1 0.2
> duplicated 207391 2.1 2.8
> accepted 7313622 73.8
> Dumping years 1901-2006 to .txt files...

And, of course, all in with 0km range:

> NORMALS MEAN percent STDEV percent
> .dtb 7315040 73.8
made it to here
> .cts 299359 3.0 7613600 76.8
> PROCESS DECISION percent %of-chk
> no lat/lon 17527 0.2 0.2
> no normal 2355659 23.8 23.8
> out-of-range 13253 0.1 0.2
> accepted 7521013 75.9
> Dumping years 1901-2006 to .txt files...

Happy? well.. no. Because something is happening for precip that does not happen for
temp! But of course. Here are the first few lines from various 1962.12 text files..

tmptxt8km/tmp.1962.12.txt
70.90 8.70 10.0 2.10000 10010
78.30 -15.50 28.0 -3.30000 10080
69.70 -18.90 10.0 -1.40000 -999
69.70 -18.90 100.0 -1.50000 10260
74.50 -19.00 16.0 -1.20000 10280
69.50 -25.50 129.0 -3.10000 10650
70.40 -31.10 14.0 -0.20000 10980
66.00 -2.00 0.0 0.50000 11000
67.30 -14.40 13.0 -1.00000 11520
66.80 -14.00 39.0 -0.70000 11530

tmptxt0km/tmp.1962.12.txt
70.90 8.70 10.0 2.10000 10010
78.30 -15.50 28.0 -3.30000 10080
69.70 -18.90 10.0 -1.40000 10250
69.70 -18.90 100.0 -1.50000 10260
74.50 -19.00 16.0 -1.20000 10280
69.50 -25.50 129.0 -3.10000 10650
70.40 -31.10 14.0 -0.20000 10980
66.00 -2.00 0.0 0.50000 11000
67.30 -14.40 13.0 -1.00000 11520
66.80 -14.00 39.0 -0.70000 11530

preanoms/pre.1962.12.txt (old anomdtb output)
61.00 10.60 190.0 48.20000-511900
54.45 -6.07 116.0 -3.70000 -999
50.83 -4.55 15.0 -22.40000-389870
50.22 -5.30 76.0 39.70000 -999
50.63 -3.45 9.0 -28.10000-388730
51.43 -2.67 51.0 -36.90000 -999
51.05 -3.60 314.0 -27.80000-386030
51.72 -2.77 245.0 -37.70000-385850
51.62 -3.97 10.0 -46.10000-384130
52.35 -3.82 301.0 -4.40000-380860

pretxt8km/pre.1962.12.txt
610.00 106.00 190.0 48.20000-511900
544.50 -60.70 116.0 -3.70000-392380
508.30 -45.50 15.0 -22.40000-389870
502.20 -53.00 76.0 39.70000-389280
506.30 -34.50 9.0 -28.10000-388730
514.30 -26.70 51.0 -36.90000-386780
510.50 -36.00 314.0 -27.80000-386030
517.20 -27.70 245.0 -37.70000-385850
516.20 -39.70 10.0 -46.10000-384130
523.50 -38.20 301.0 -4.40000-380860

pretxt0km/pre.1962.12.txt
610.00 106.00 190.0 48.20000-511900
544.50 -60.70 116.0 -3.70000-392380
508.30 -45.50 15.0 -22.40000-389870
502.20 -53.00 76.0 39.70000-389280
506.30 -34.50 9.0 -28.10000-388730
514.30 -26.70 51.0 -36.90000-386780
510.50 -36.00 314.0 -27.80000-386030
517.20 -27.70 245.0 -37.70000-385850
516.20 -39.70 10.0 -46.10000-384130
523.50 -38.20 301.0 -4.40000-380860

..As a result of fixing the lats and lons for temperature, and indeed
precip it seems, we have buggered up the outputs!!! Obviously the
correction factor is expecting 100 not 10, but why isn't this a problem
for temperature?! Went back and ran exactly the same version of anomdtb
on temperature - exactly the same as last time (2nd from top above). So
it is precip specific (or, erm, .not.temp specific?).

On the other hand, we've fixed the -999 WMO codes..

..and actually, those anomalies had better be percentage anomalies!

(checks a few) - yes, they are :-)

So oookay, LoadCTS reports the divisor is still 10 for lon/lat, so the
stored values for the first station (-511900, BIRI) should be 61 and 10.6,
sounds about right for Norway. The bit in anomdtb (actually the subroutine
'Dumping', LOL) that writes the .txt files just writes directly from the
arrays.. so they must have been modified somewhere in 'Anomalise' (there's
nothing else in 'Dumping'). Modified anomdtb to dump the first station's
lat & lon at key stages - they were too high throughout, so LoadCTS assumed
to be the troublemaker. Modified LoadCTS in the same way, and it was
holding them at x100 from their true values, ie 61.0 -> 6100. It was about
now that I spotted something I'd not thought to examine before: precip
headers use two decimal places for their coordinates!

Temperature header:
10010 709 87 10 Jan Mayen NORWAY 1921 2006 341921 -999.00

Precipitation header:
100100 7093 -867 10 JAN MAYEN NORWAY 1921 2006 -999 -999.00

So.. this begs the question, how does the software suite know which it's got?
By rights it should look at the most extreme values for each.. something tells
me that's not the case. Decided to look at the ranges of values for different
versions of the databases, starting with temperature:

crua6[/cru/cruts] head -1 fromdpe1a/data/cruts/database/+norm/tmp.0311051552.dtb
-990017 -9999 -99999 -999 UNKNOWN MARINE 1948 1990 -999 -999.00
crua6[/cru/cruts] head -1 fromdpe1a/data/cruts/database/+norm/_old/tmp.0310311715.dtb
-176000 3520 3330 220 NICOSIA CYPRUS 1932 1974 -999 nocode
crua6[/cru/cruts] head -1 rerun1/data/cruts/rerun_tmp/tmp.0311051552.dtb
-990017 -9999 -99999 -999 UNKNOWN MARINE 1948 1990 -999 -999.00
crua6[/cru/cruts] head -1 rerun1/data/cruts/rerun_tmp/tmp.0311051552n.dtb
-990017 -9999 -99999 -999 UNKNOWN MARINE 1948 1990 -999 -999.00
crua6[/cru/cruts] head -1 rerun1/data/cruts/rerun_tmp/database/+norm/_old/tmp.0310311715.dtb
-176000 3520 3330 220 NICOSIA CYPRUS 1932 1974 -999 nocode
crua6[/cru/cruts] head -1 rerun1/data/cruts/rerun_tmp/database/+norm/tmp.0311051552.dtb
-990017 -9999 -99999 -999 UNKNOWN MARINE 1948 1990 -999 -999.00
crua6[/cru/cruts] head -1 rerun1/data/cruts/rerun_tmp/database/tmp.0311051552.dtb
-990017 -9999 -99999 -999 UNKNOWN MARINE 1948 1990 -999 -999.00
crua6[/cru/cruts] head -1 version_3_0/primaries/temp/tmp.0702091122.dtb
10010 709 87 10 Jan Mayen NORWAY 1921 1990 341921 -999.00
crua6[/cru/cruts] head -1 version_3_0/primaries/temp/tmp.0704300053.dtb
10010 709 87 10 Jan Mayen NORWAY 1921 2006 341921 -999.00
crua6[/cru/cruts] head -1 version_3_0/db/testmergedb/tmp.0702091122.dtb
10010 709 87 10 Jan Mayen NORWAY 1921 1990 341921 -999.00
crua6[/cru/cruts] head -1 version_3_0/db/testmergedb/tmp.0704292355.dtb
10010 709 87 10 Jan Mayen NORWAY 1921 2006 341921 -999.00
crua6[/cru/cruts] head -1 version_3_0/db/testmergedb/badtimeline/tmp.0704251819.dtb
10010 709 87 10 Jan Mayen NORWAY 1921 2006 341921 -999.00
crua6[/cru/cruts] head -1 version_3_0/db/testmergedb/badtimeline/tmp.0704271015.dtb
10010 709 87 10 Jan Mayen NORWAY 1921 2006 341921 -999.00
crua6[/cru/cruts] head -1 version_3_0/db/testmergedb/badtimeline/tmp.0704292158.dtb
10010 709 87 10 Jan Mayen NORWAY 1921 2006 341921 -999.00
crua6[/cru/cruts] head -1 version_3_0/db/testmergedb/tmp.0704300053.dtb
10010 709 87 10 Jan Mayen NORWAY 1921 2006 341921 -999.00
crua6[/cru/cruts] head -1 version_3_0/db/tmp.0702091122.dtb
10010 709 87 10 Jan Mayen NORWAY 1921 1990 341921 -999.00
crua6[/cru/cruts] head -1 version_3_0/db/tmp.0704300053.dtb
10010 709 87 10 Jan Mayen NORWAY 1921 2006 341921 -999.00

Without going any further, it's obvious that LoadCTS is going to have to auto-
sense the lat and lon ranges. Missing value codes can then be derived - if it
always returns actual (unscaled) degrees (to one or two decimal places) then
any value lower than -998 will suffice for both parameters. However, this does
make me wonder why it wasn't done like that. Is there a likelihood of the
programs being used on a spatial subset of stations? Say, English? Then lon
would never get into double figures, though lat would.. well let's just hope
not! *laughs hollowly*

Okay.. so I wrote extra code into LoadCTS to detect Lat & Lon ranges. It excludes any
values for which the modulus of 100 is -99, so hopefully missing value codes do not
conribute. The factors are set accordingly (to 10 or 100). I had to default to 1 which
is a pity. Once you've got the factors, detection of missing values can be a simple
out-of-range test.

However *sigh* this led me to examine the detection of 'non-standard longitudes' - a
small section of code that converts PJ-style reversed longitudes, or 0-360 ones, to
regular -180 (W) to +180 (E). This code is switched on by the presence of the
'LongType' flag in the LoadCTS call - the trouble is, THAT FLAG IS NEVER SET BY
ANOMDTB. There is a declaration 'integer :: QLongType' but that is never referred to
again. Just another thing I cannot understand, and another reason why this should all
have been rewritten from scratch a year ago!

So, I wrote 'revlons.for' - a proglet to reverse all longitude values in a database
file. Ran it on the temperature database (final):


crua6[/cru/cruts/version_3_0/db/testmergedb] ./revlons
REVLONS - Reverse All Longitudes!

This nifty little proglet will fix all of your
longitudes so that they point the right way, ie,
positive = East of Greenwich, negative = West.

..of course, if they are already fixed, this will
UNfix them. I am not that smart! So be careful!!

Please enter the database to be fixed: tmp.0704300053.dtb

Output file will be: tmp.0705101334.dtb
Confirm this filename (Y/N): Y

Log file will be: tmp.0705101334.log

5065 stations written to tmp.0705101334.dtb



Thus the 'final' temperature database is now tmp.0705101334.dtb.

Re-ran anomdtb - with working lat/lon detection and missing lat/lon value
detection - for both precip and temperature. This should ensure that all
WMO codes are present and all lats and lons are correct.

Temp:

> ***** AnomDTB: converts .dtb to anom .txt for gridding *****

> Enter the suffix of the variable required:
.tmp
> Select the .cts or .dtb file to load:
tmp.0705101334.dtb

> Specify the start,end of the normals period:
1961,1990
> Specify the missing percentage permitted:
25
> Data required for a normal: 23
> Specify the no. of stdevs at which to reject data:
3
> Select outputs (1=.cts,2=.ann,3=.txt,4=.stn):
3
> Check for duplicate stns after anomalising? (0=no,>0=km range)
0
> Select the generic .txt file to save (yy.mm=auto):
tmp.txt
> Select the first,last years AD to save:
1901,2006
> Operating...

> NORMALS MEAN percent STDEV percent
> .dtb 3323823 81.3
> .cts 91963 2.2 3415786 83.5
> PROCESS DECISION percent %of-chk
> no lat/lon 1993 0.0 0.0
> no normal 673044 16.5 16.5
> out-of-range 744 0.0 0.0
> accepted 3415042 83.5
> Dumping years 1901-2006 to .txt files...


Precip:

crua6[/cru/cruts/version_3_0/primaries/precip] ./anomdtb

> ***** AnomDTB: converts .dtb to anom .txt for gridding *****

> Enter the suffix of the variable required:
.pre
> Will calculate percentage anomalies.
> Select the .cts or .dtb file to load:
pre.0612181221.dtb

> Specify the start,end of the normals period:
1961,1990
> Specify the missing percentage permitted:
25
> Data required for a normal: 23
> Specify the no. of stdevs at which to reject data:
4
> Select outputs (1=.cts,2=.ann,3=.txt,4=.stn):
3
> Check for duplicate stns after anomalising? (0=no,>0=km range)
0
> Select the generic .txt file to save (yy.mm=auto):
pre.txt
> Select the first,last years AD to save:
1901,2006
> Operating...

> NORMALS MEAN percent STDEV percent
> .dtb 7315040 73.8
> .cts 299359 3.0 7613600 76.8
> PROCESS DECISION percent %of-chk
> no lat/lon 17911 0.2 0.2
> no normal 2355275 23.8 23.8
> out-of-range 13253 0.1 0.2
> accepted 7521013 75.9
> Dumping years 1901-2006 to .txt files...


Note that precip accepted values is up to 75.9%, I honestly don't
think we'll get higher.

Decided to process temperature all the way. Ran IDL:

IDL> quick_interp_tdm2,1901,2006,'tmpglo/tmpgrid.',1200,gs=0.5,dumpglo='dumpglo',pts_prefix='tmp0km0705101334txt/tmp.'

then glo2abs, then mergegrids, to produce monthly output grids. It apparently worked:

-rw------- 1 f098 cru 138964083 May 13 20:42 cru_ts_3_00.1901.2006.tmp.dat.gz
-rw------- 1 f098 cru 7852589 May 13 20:42 cru_ts_3_00.2001.2006.tmp.dat.gz
-rw------- 1 f098 cru 13108065 May 13 20:39 cru_ts_3_00.1991.2000.tmp.dat.gz
-rw------- 1 f098 cru 13106515 May 13 20:36 cru_ts_3_00.1981.1990.tmp.dat.gz
-rw------- 1 f098 cru 13106963 May 13 20:33 cru_ts_3_00.1971.1980.tmp.dat.gz
-rw------- 1 f098 cru 13123939 May 13 20:30 cru_ts_3_00.1961.1970.tmp.dat.gz
-rw------- 1 f098 cru 13120586 May 13 20:26 cru_ts_3_00.1951.1960.tmp.dat.gz
-rw------- 1 f098 cru 13120691 May 13 20:23 cru_ts_3_00.1941.1950.tmp.dat.gz
-rw------- 1 f098 cru 13130077 May 13 20:20 cru_ts_3_00.1931.1940.tmp.dat.gz
-rw------- 1 f098 cru 13104881 May 13 20:16 cru_ts_3_00.1921.1930.tmp.dat.gz
-rw------- 1 f098 cru 13094948 May 13 20:13 cru_ts_3_00.1911.1920.tmp.dat.gz
-rw------- 1 f098 cru 13085509 May 13 17:08 cru_ts_3_00.1901.1910.tmp.dat.gz

As a reminder, these output grids are based on the tmp.0705101334.dtb database, with no
merging of neighbourly stations and a limit of 3 standard deviations on anomalies.

Decided to (re-) process precip all the way, in the hope that I was in the zone or
something. Started with IDL:

IDL> quick_interp_tdm2,1901,2006,'preglo/pregrid.',450,gs=0.5,dumpglo='dumpglo',pts_prefix='pre0km0612181221txt/pre.'

Then glo2abs, then mergegrids.. all went fine, apparently.


Go on to part 31, back to index or Email search