The HARRY_READ_ME.txt file

*************************************************************************************
* PRIORITY INTERRUPT * PRIORITY INTERRUPT * PRIORITY INTERRUPT * PRIORITY INTERRUPT *
*************************************************************************************

After an email enquiry from Wladimir J. Alonso (alonsow@xxxxx), in which unusual behaviour of CRU TS 2.10
Vapour Pressure data was observed, I discovered that some of the Wet Days and Vepour Pressure datasets had been
swapped!!

The files I was looking at were decadal, 1981-1990.

Vapour Pressure, January: Min 0 Max 310
Vapour Pressure, February: Min 0 Max 280

Wet Days, January: Min 0 Max 3220
Wet days, February: Min 0 Max 3240

So I wrote crutsstats.for, whioch returns monthly and annual minima, maxima and means for any gridded output file.

Tried it on the full runs, and they look OK:

crua6[/cru/cruts/vap_wet_investigation] head -90 cru_ts_2_10.1901-2002.vap.grid.stats |tail -10
1981 0 322 82 0 324 84 0 320 90 0 335 99 0 352 111 0 356 130 0 349 144 0 344 143 0 360 124 0 323 105 0 320 90 0 321 83 0 360 107
1982 0 312 80 0 323 83 0 318 88 0 329 98 0 348 111 0 357 126 0 365 143 0 364 140 0 355 124 0 318 105 0 321 90 0 318 84 0 365 106
1983 0 348 82 0 340 85 0 330 90 0 505 99 0 348 112 0 364 130 0 360 145 0 362 143 0 368 126 0 323 105 0 318 91 0 317 82 0 505 108
1984 0 312 80 0 320 82 0 315 89 0 329 97 0 347 112 0 359 130 0 353 144 0 343 140 0 353 122 0 324 105 0 318 89 0 316 81 0 359 106
1985 0 314 80 0 320 81 0 319 88 0 359 98 0 352 111 0 367 128 0 358 141 0 355 141 0 353 123 0 323 105 0 322 90 0 323 82 0 367 106
1986 0 312 81 0 330 83 0 316 89 0 321 99 0 366 112 0 394 129 0 371 143 0 342 139 0 354 122 0 323 104 0 318 90 0 316 82 0 394 106
1987 0 320 81 0 318 85 0 318 88 0 335 98 0 363 112 0 366 130 0 397 147 0 356 142 0 354 126 0 345 105 0 325 91 0 365 84 0 397 107
1988 0 413 83 0 324 84 0 352 90 0 323 99 0 346 113 0 363 131 0 367 148 0 358 144 0 387 126 0 342 105 0 320 89 0 315 83 0 413 108
1989 0 336 80 0 320 83 0 327 90 0 324 98 0 343 112 0 366 130 0 365 145 0 349 142 0 353 124 0 323 105 0 324 90 0 332 84 0 366 107
1990 0 320 83 0 323 85 0 476 92 0 413 101 0 361 113 0 363 132 0 371 146 0 353 143 0 371 124 0 327 106 0 318 93 0 317 84 0 476 108

crua6[/cru/cruts/vap_wet_investigation] head -90 cru_ts_2_10.1901-2002.wet.grid.stats | tail -10
1981 0 3100 1018 0 2800 919 0 3100 980 0 3000 911 0 3100 945 0 3000 1010 0 3100 1051 0 3100 1040 0 3000 981 0 3100 1017 0 3000 1021 0 3100 1003 0 3100 992
1982 0 3100 983 0 2800 894 0 3100 967 0 3000 925 0 3100 927 0 3000 941 0 3100 979 0 3100 1054 0 3000 1007 0 3100 1055 0 3000 996 0 3100 1044 0 3100 981
1983 0 3100 1035 0 2800 863 0 3100 941 0 3000 919 0 3100 929 0 3000 949 0 3100 990 0 3100 1039 0 3000 996 0 3100 1026 0 3000 1034 0 3100 1057 0 3100 982
1984 0 3100 981 0 2900 848 0 3100 920 0 3000 841 0 3100 932 0 3000 973 0 3100 1048 0 3100 1057 0 3000 1023 0 3100 1057 0 3000 992 0 3100 1016 0 3100 974
1985 0 3100 969 0 2800 896 0 3100 952 0 3000 896 0 3100 928 0 3000 938 0 3100 1057 0 3100 1043 0 3000 993 0 3100 1043 0 3000 1066 0 3100 1029 0 3100 984
1986 0 3100 988 0 2800 908 0 3100 950 0 3000 895 0 3100 922 0 3000 962 0 3100 1022 0 3100 1052 0 3000 1037 0 3100 1052 0 3000 1048 0 3100 986 0 3100 985
1987 0 3100 1011 0 2800 909 0 3100 930 0 3000 856 0 3100 954 0 3000 972 0 3100 1021 0 3100 1064 0 3000 978 0 3100 991 0 3000 1002 0 3100 1047 0 3100 978
1988 0 3100 1033 0 2900 924 0 3100 971 0 3000 903 0 3100 938 0 3000 980 0 3100 1039 0 3100 1101 0 3000 1014 0 3100 1017 0 3000 1007 0 3100 1054 0 3100 998
1989 0 3100 1019 0 2800 936 0 3100 1015 0 3000 892 0 3100 978 0 3000 1020 0 3100 1054 0 3100 1075 0 3000 1023 0 3100 1070 0 3000 1046 0 3100 1053 0 3100 1015
1990 0 3100 996 0 2800 959 0 3100 1011 0 3000 953 0 3100 928 0 3000 907 0 3100 983 0 3100 986 0 3000 915 0 3100 968 0 3000 949 0 3100 959 0 3100 960

So the monthly maxima are fine here. But for the decadal files?

crua6[/cru/cruts/vap_wet_investigation] cat cru_ts_2_10.1981-1990.vap.grid.stats0
1981 0 310 102 0 280 92 0 310 98 0 300 91 0 310 95 0 300 101 0 310 105 0 310 104 0 300 98 0 310 102 0 300 102 0 310 100 0 310 99
1982 0 310 98 0 280 89 0 310 97 0 300 93 0 310 93 0 300 94 0 310 98 0 310 105 0 300 101 0 310 106 0 300 100 0 310 104 0 310 98
1983 0 310 104 0 280 86 0 310 94 0 300 92 0 310 93 0 300 95 0 310 99 0 310 104 0 300 100 0 310 103 0 300 103 0 310 106 0 310 98
1984 0 310 98 0 290 85 0 310 92 0 300 84 0 310 93 0 300 97 0 310 105 0 310 106 0 300 102 0 310 106 0 300 99 0 310 102 0 310 97
1985 0 310 97 0 280 90 0 310 95 0 300 90 0 310 93 0 300 94 0 310 106 0 310 104 0 300 99 0 310 104 0 300 107 0 310 103 0 310 98
1986 0 310 99 0 280 91 0 310 95 0 300 90 0 310 92 0 300 96 0 310 102 0 310 105 0 300 104 0 310 105 0 300 105 0 310 99 0 310 99
1987 0 310 101 0 280 91 0 310 93 0 300 86 0 310 95 0 300 97 0 310 102 0 310 106 0 300 98 0 310 99 0 300 100 0 310 105 0 310 98
1988 0 310 103 0 290 92 0 310 97 0 300 90 0 310 94 0 300 98 0 310 104 0 310 110 0 300 101 0 310 102 0 300 101 0 310 105 0 310 100
1989 0 310 102 0 280 94 0 310 101 0 300 89 0 310 98 0 300 102 0 310 105 0 310 107 0 300 102 0 310 107 0 300 105 0 310 105 0 310 101
1990 0 310 100 0 280 96 0 310 101 0 300 95 0 310 93 0 300 91 0 310 98 0 310 99 0 300 91 0 310 97 0 300 95 0 310 96 0 310 96

crua6[/cru/cruts/vap_wet_investigation] cat cru_ts_2_10.1981-1990.wet.grid.stats
1981 0 3220 819 0 3240 842 0 3200 903 0 3350 992 0 3520 1113 0 3560 1304 0 3490 1440 0 3440 1427 0 3600 1236 0 3230 1048 0 3200 898 0 3210 833 0 3600 1071
1982 0 3120 801 0 3230 827 0 3180 881 0 3290 982 0 3480 1108 0 3570 1264 0 3650 1432 0 3640 1405 0 3550 1239 0 3180 1048 0 3210 901 0 3180 835 0 3650 1060
1983 0 3480 820 0 3400 850 0 3300 898 0 5050 993 0 3480 1125 0 3640 1295 0 3600 1451 0 3620 1428 0 3680 1259 0 3230 1050 0 3180 912 0 3170 822 0 5050 1075
1984 0 3120 803 0 3200 823 0 3150 887 0 3290 971 0 3470 1124 0 3590 1299 0 3530 1437 0 3430 1404 0 3530 1218 0 3240 1053 0 3180 894 0 3160 812 0 3590 1060
1985 0 3140 803 0 3200 815 0 3190 882 0 3590 978 0 3520 1113 0 3670 1277 0 3580 1405 0 3550 1411 0 3530 1233 0 3230 1048 0 3220 900 0 3230 821 0 3670 1057
1986 0 3120 809 0 3300 827 0 3160 889 0 3210 990 0 3660 1120 0 3940 1294 0 3710 1428 0 3420 1393 0 3540 1220 0 3230 1041 0 3180 895 0 3160 821 0 3940 1061
1987 0 3200 810 0 3180 849 0 3180 880 0 3350 980 0 3630 1124 0 3660 1296 0 3970 1466 0 3560 1423 0 3540 1260 0 3450 1054 0 3250 910 0 3650 844 0 3970 1075
1988 0 4130 829 0 3240 835 0 3520 902 0 3230 989 0 3460 1133 0 3630 1311 0 3670 1475 0 3580 1441 0 3870 1264 0 3420 1054 0 3200 889 0 3150 832 0 4130 1079
1989 0 3360 804 0 3200 825 0 3270 898 0 3240 978 0 3430 1120 0 3660 1301 0 3650 1447 0 3490 1421 0 3530 1240 0 3230 1052 0 3240 900 0 3320 836 0 3660 1069
1990 0 3200 827 0 3230 853 0 4760 918 0 4130 1005 0 3610 1127 0 3630 1322 0 3710 1462 0 3530 1428 0 3710 1236 0 3270 1062 0 3180 930 0 3170 844 0 4760 1084

Much confusion! The orders of magnitude have changed to reflect the expected ranges - but the data have clearly been swapped!

Another decade:

crua6[/cru/cruts/vap_wet_investigation]cat cru_ts_2_10.1921-1930.vap.grid.stats
1921 0 310 102 0 280 89 0 310 100 0 300 88 0 310 95 0 300 97 0 310 101 0 310 104 0 300 102 0 310 104 0 300 97 0 310 101 0 310 98
1922 0 310 95 0 280 93 0 310 97 0 300 89 0 310 95 0 300 98 0 310 105 0 310 107 0 300 98 0 310 104 0 300 102 0 310 103 0 310 99
1923 0 310 100 0 280 88 0 310 97 0 300 90 0 310 97 0 300 98 0 310 101 0 310 101 0 300 100 0 310 104 0 300 101 0 310 103 0 310 98
1924 0 310 97 0 290 89 0 310 95 0 300 90 0 310 91 0 300 97 0 310 100 0 310 102 0 300 101 0 310 102 0 300 102 0 310 100 0 310 97
1925 0 310 98 0 280 89 0 310 98 0 300 87 0 310 90 0 300 96 0 310 101 0 310 103 0 300 103 0 310 101 0 300 103 0 310 100 0 310 97
1926 0 310 99 0 280 87 0 310 95 0 300 87 0 310 95 0 300 93 0 310 103 0 310 104 0 300 99 0 310 102 0 300 102 0 310 101 0 310 97
1927 0 310 96 0 280 87 0 310 96 0 300 89 0 310 94 0 300 97 0 310 103 0 310 104 0 300 102 0 310 103 0 300 102 0 310 99 0 310 98
1928 0 310 97 0 290 89 0 310 91 0 300 88 0 310 90 0 300 96 0 310 101 0 310 104 0 300 97 0 310 99 0 300 99 0 310 96 0 310 96
1929 0 310 95 0 280 84 0 310 95 0 300 86 0 310 91 0 300 95 0 310 100 0 310 102 0 300 98 0 310 102 0 300 98 0 310 98 0 310 95
1930 0 310 98 0 280 88 0 310 97 0 300 88 0 310 93 0 300 93 0 310 99 0 310 103 0 300 99 0 310 105 0 300 101 0 310 97 0 310 97
crua6[/cru/cruts/vap_wet_investigation] cat cru_ts_2_10.1921-1930.wet.grid.stats
1921 0 3120 805 0 3190 814 0 3140 874 0 3210 969 0 3800 1106 0 3590 1289 0 3600 1439 0 3440 1390 0 3530 1220 0 3230 1032 0 3180 877 0 3160 824 0 3800 1053
1922 0 3120 794 0 3220 813 0 3140 874 0 3210 971 0 3470 1104 0 3590 1280 0 3560 1420 0 3440 1387 0 3530 1211 0 3230 1025 0 3180 896 0 3140 812 0 3590 1049
1923 0 3070 799 0 3140 808 0 3140 871 0 3210 947 0 3460 1082 0 3660 1276 0 3560 1410 0 3440 1392 0 3530 1222 0 3230 1048 0 3180 907 0 3160 826 0 3660 1049
1924 0 3270 792 0 3230 817 0 3160 879 0 3340 955 0 3460 1094 0 3710 1264 0 3560 1415 0 3440 1386 0 3530 1228 0 3160 1034 0 3180 892 0 3140 806 0 3710 1047
1925 0 3110 786 0 3190 815 0 3140 873 0 3210 966 0 3470 1084 0 3590 1253 0 3560 1408 0 3460 1397 0 3530 1231 0 3230 1025 0 3160 896 0 3220 828 0 3590 1047
1926 0 3260 815 0 3290 842 0 3310 889 0 3310 957 0 3460 1085 0 3950 1266 0 3560 1406 0 3450 1402 0 3530 1237 0 3230 1042 0 3250 899 0 3150 811 0 3950 1054
1927 0 3120 795 0 3300 822 0 3170 873 0 3360 959 0 3540 1096 0 3610 1271 0 3550 1424 0 3450 1390 0 3530 1233 0 3230 1053 0 3180 897 0 3280 814 0 3610 1052
1928 0 3200 809 0 3240 823 0 3140 875 0 3400 963 0 3470 1095 0 3590 1263 0 3560 1425 0 3450 1397 0 3530 1228 0 3230 1039 0 3180 902 0 3160 824 0 3590 1054
1929 0 3150 794 0 3190 802 0 3160 867 0 3310 950 0 3600 1084 0 3580 1250 0 3550 1399 0 3440 1385 0 3530 1218 0 3230 1049 0 3180 897 0 3160 806 0 3600 1042
1930 0 3190 798 0 3190 824 0 3150 881 0 3210 965 0 3470 1099 0 3590 1276 0 3530 1424 0 3440 1409 0 3540 1220 0 3200 1042 0 3300 907 0 3280 829 0 3590 1056

The same story. And the final two years:

crua6[/cru/cruts/vap_wet_investigation] cat cru_ts_2_10.2001-2002.vap.grid.stats
2001 0 310 87 0 280 84 0 310 90 0 300 81 0 310 87 0 300 93 0 310 95 0 310 95 0 300 89 0 310 95 0 300 95 0 310 87 0 310 90
2002 0 310 91 0 280 85 0 310 92 0 300 83 0 310 88 0 300 89 0 310 93 0 310 94 0 300 92 0 310 93 0 300 88 0 310 86 0 310 90
crua6[/cru/cruts/vap_wet_investigation] cat cru_ts_2_10.2001-2002.wet.grid.stats
2001 0 3320 834 0 3250 841 0 3180 913 0 3490 1010 0 3490 1147 0 4380 1323 0 3660 1487 0 5120 1466 0 3530 1266 0 3460 1088 0 3620 932 0 3410 843 0 5120 1096
2002 0 3310 837 0 3390 863 0 3270 918 0 3370 1012 0 3930 1151 0 4140 1339 0 3750 1503 0 5110 1453 0 3530 1261 0 3310 1067 0 3470 922 0 3300 833 0 5110 1096

It looks like a consistent problem: all the decadal VAp and WET files should be discarded, and only the 'full run' 1901-2002
files used. But my theory that the error occurred when the 1901-2002 files were converted to decadal doesn't sound true now,
because why would the precision levels change? Surely, if the decadal files are derived from the 1901-2002 files, it's just
a case of copying data across?

Let's look at *just* 1981, to try and assess this issue:

FULL 1901-2002 FILE
VAP:
1981 0 322 82 0 324 84 0 320 90 0 335 99 0 352 111 0 356 130 0 349 144 0 344 143 0 360 124 0 323 105 0 320 90 0 321 83 0 360 107
WET:
1981 0 3100 1018 0 2800 919 0 3100 980 0 3000 911 0 3100 945 0 3000 1010 0 3100 1051 0 3100 1040 0 3000 981 0 3100 1017 0 3000 1021 0 3100 1003 0 3100 992

DECADAL 1981-1990 FILE
VAP:
1981 0 310 102 0 280 92 0 310 98 0 300 91 0 310 95 0 300 101 0 310 105 0 310 104 0 300 98 0 310 102 0 300 102 0 310 100 0 310 99
WET:
1981 0 3220 819 0 3240 842 0 3200 903 0 3350 992 0 3520 1113 0 3560 1304 0 3490 1440 0 3440 1427 0 3600 1236 0 3230 1048 0 3200 898 0 3210 833 0 3600 1071

It's evident that the data have not only been swapped - they've been scaled too. Aaaarrrgghhhhhh!!!!!

*******************************************************************************
* PRIORITY INTERRUPT ENDS * PRIORITY INTERRUPT ENDS * PRIORITY INTERRUPT ENDS *
*******************************************************************************

Now, where were we.. ah yes, Vapour Pressure. So far:

Original: vap.0311181410.dtb
+
MCDW: vap.0709111032.dtb
v
v
Intermediate: vap.0710241541.dtb
+
CLIMAT: vap.0710151817.dtb
v
v
Final: vap.0710241549.dtb

Produce anomalies:

<BEGIN_QUOTE>
crua6[/cru/cruts/version_3_0/secondaries/vap] ./anomdtb

> ***** AnomDTB: converts .dtb to anom .txt for gridding *****

> Enter the suffix of the variable required:
.vap
> Select the .cts or .dtb file to load:
vap.0710241549.dtb
> Specify the start,end of the normals period:
1961,1990
> Specify the missing percentage permitted:
25
> Data required for a normal: 23
> Specify the no. of stdevs at which to reject data:
3
> Select outputs (1=.cts,2=.ann,3=.txt,4=.stn):
3
> Check for duplicate stns after anomalising? (0=no,>0=km range)
0
> Select the generic .txt file to save (yy.mm=auto):
vap.txt
> Select the first,last years AD to save:
1901,2006
> Operating...

> NORMALS MEAN percent STDEV percent
> .dtb 908812 45.2
> .cts 35390 1.8 944202 47.0
> PROCESS DECISION percent %of-chk
> no lat/lon 105 0.0 0.0
> no normal 1064261 53.0 53.0
> out-of-range 49 0.0 0.0
> accepted 944153 47.0
> Dumping years 1901-2006 to .txt files...

crua6[/cru/cruts/version_3_0/secondaries/vap]
<END_QUOTE>

Well.. 47% accepted, 53% no normals.. pretty much as expected, and unlikely to improve no matter how many new CLIMAT
and MCDW updates there are. We need back data for 1961-1990.

Synthetic production:

<BEGIN_QUOTE>
IDL> vap_gts_anom,dtr_prefix='../dtrbin/dtrbin',tmp_prefix='../tmpbin/tmpbin',1901,2006,outprefix='vapsyn/vapsyn',dumpbin=1
% Compiled module: VAP_GTS_ANOM.
% Compiled module: RDBIN.
% Compiled module: STRIP.
% Compiled module: DEFXYZ.
Land,sea: 56016 68400
Calculating tmn normal
% Compiled module: TVAP.
Calculating synthetic vap normal
% Compiled module: ESAT.
Calculating synthetic anomalies
% Compiled module: MOMENT.
1901 vap (x,s2,<<,>>): 1.61250e-05 6.15570e-06 -0.160607 0.222689
% Compiled module: WRBIN.
1902 vap (x,s2,<<,>>): -0.000123188 3.46116e-05 -0.268891 0.0261283
1903 vap (x,s2,<<,>>): 6.86689e-05 4.52675e-06 -0.121429 0.123995
(etc)
<END_QUOTE>

(also produced, vapsyn/vapsyn1901 .. vapsyn/vapsyn2006)

Gridding with both observed and synthetic data:

IDL> quick_interp_tdm2,1901,2006,'vapglo/vap.',1000,gs=0.5,dumpglo='dumpglo',synth_prefix='vapsyn/vapsyn',pts_prefix='vaptxt/vap.'

Create absolute grids from anomaly grids:

<BEGIN_QUOTE>
crua6[/cru/cruts/version_3_0/secondaries/vap] ./glo2abs
Welcome! This is the GLO2ABS program.
I will create a set of absolute grids from
a set of anomaly grids (in .glo format), also
a gridded version of the climatology.
Enter the path and name of the normals file: clim.6190.lan.vap
Enter a name for the gridded climatology file: clim.6190.lan.vap.grid
Enter the path and stem of the .glo files: vapglo/vap.
Enter the starting year: 1901
Enter the ending year: 2006
Enter the path (if any) for the output files: vapabs/
Now, CONCENTRATE. Addition or Percentage (A/P)? A
Do you wish to limit the output values? (Y/N): Y
1. Set minimum to zero
2. Set a single minimum and maximum
3. Set monthly minima and maxima (for wet/rd0)

Choose: 1
Right, erm.. off I jolly well go!
vap.01.1901.glo
vap.02.1901.glo
(etc)
<END_QUOTE>

and finally, create the output files:

<BEGIN_QUOTE>
crua6[/cru/cruts/version_3_0/secondaries/vap] ./mergegrids
Welcome! This is the MERGEGRIDS program.
I will create decadal and full gridded files
from the output files of (eg) glo2abs.for.

Enter a gridfile with YYYY for year and MM for month: vapabs/vap.MM.YYYY.glo.abs
Enter Start Year: 1901
Enter Start Month: 01
Enter End Year: 2006
Enter End Month: 12

Please enter a sample OUTPUT filename, replacing
start year with SSSS and end year with EEEE: cru_ts_3_00.SSSS.YYYY.vap.dat
Try again.. read instructions this time?

Please enter a sample OUTPUT filename, replacing
start year with SSSS and end year with EEEE: cru_ts_3_00.SSSS.EEEE.vap.dat
Writing cru_ts_3_00.1901.1910.vap.dat
Writing cru_ts_3_00.1911.1920.vap.dat
Writing cru_ts_3_00.1921.1930.vap.dat
Writing cru_ts_3_00.1931.1940.vap.dat
Writing cru_ts_3_00.1941.1950.vap.dat
Writing cru_ts_3_00.1951.1960.vap.dat
Writing cru_ts_3_00.1961.1970.vap.dat
Writing cru_ts_3_00.1971.1980.vap.dat
Writing cru_ts_3_00.1981.1990.vap.dat
Writing cru_ts_3_00.1991.2000.vap.dat
Writing cru_ts_3_00.2001.2006.vap.dat
<END_QUOTE>

Ah - and I was really hoping this time that it would just WORK. But of course not - nothing works first
time in this project. I ran crutsstats on cru_ts_3_00.1901.2006.vap.dat, and:

<BEGIN_QUOTE>
crua6[/cru/cruts/version_3_0/secondaries/vap] ./crutsstats

CRUTSSTATS: Stats for CRU TS gridded files

Enter the monthly gridded data file: cru_ts_3_00.1901.2006.vap.dat

Please enter the start year: 1901

106 years from 1901 to 2006

Output file is cru_ts_3_00.1901.2006.vap.dat.stats
1901 1 358 106
1902 1 358 106
1903 1 358 106
1904 1 358 106
1905 1 358 106
(etc)
2002 1 358 106
2003 1 358 106
2004 1 358 106
2005 1 358 106
2006 1 358 106
<END_QUOTE>

What?! Every year has the same min (fine, VAP of 0 is probably impossible), max (I can just about believe,
if there's a cell with no stations inside the cdd and the normal for it happens to be the highest value, and
MEAN (oh no, NO WAY!). What's odder - the .glo files are different:

crua6[/cru/cruts/version_3_0/secondaries/vap/vapabs] diff vap.06.1974.glo.abs.nh vap.06.1975.glo.abs.nh |wc -l
56

Admittedly, 56 lines different out of 360 isn't hugely different. And looking, they are only slight and
infrequent differences. But the monthly stats are all cloned as well:

1901 1 311 80 1 320 83 1 315 89 1 320 98 1 346 111 1 358 128 1 356 143 1 342 140 1 354 123 1 323 104 1 318 90 1 315 82 1 358 106
1902 1 311 80 1 320 83 1 315 89 1 320 98 1 346 111 1 358 128 1 356 143 1 342 140 1 354 123 1 323 104 1 318 90 1 315 82 1 358 106
1903 1 311 80 1 320 83 1 315 89 1 320 98 1 346 111 1 358 128 1 356 143 1 342 140 1 354 123 1 323 104 1 318 90 1 315 82 1 358 106
1904 1 311 80 1 320 83 1 315 89 1 320 98 1 346 111 1 358 128 1 356 143 1 342 140 1 354 123 1 323 104 1 318 90 1 315 82 1 358 106
1905 1 311 80 1 320 83 1 315 89 1 320 98 1 346 111 1 358 128 1 356 143 1 342 140 1 354 123 1 323 104 1 318 90 1 315 82 1 358 106
1906 1 311 80 1 320 83 1 315 89 1 320 98 1 346 111 1 358 128 1 356 143 1 342 140 1 354 123 1 323 104 1 318 90 1 315 82 1 358 106
1907 1 311 80 1 320 83 1 315 89 1 320 98 1 346 111 1 358 128 1 356 143 1 342 140 1 354 123 1 323 104 1 318 90 1 315 82 1 358 106


Well the first thing to do, after the inevitable wailing and gnashing of teeth, is to re-run glo2abs
without the 'zero minimum' flag (just in case I coded that badly, I was in a hurry):

<BEGIN_QUOTE>
crua6[/cru/cruts/version_3_0/secondaries/vap] ./glo2abs
Welcome! This is the GLO2ABS program.
I will create a set of absolute grids from
a set of anomaly grids (in .glo format), also
a gridded version of the climatology.
Enter the path and name of the normals file: clim.6190.lan.vap
Enter a name for the gridded climatology file: clim.6190.lan.vap.grid2
Enter the path and stem of the .glo files: vapglo/vap.
Enter the starting year: 1901
Enter the ending year: 2006
Enter the path (if any) for the output files: vapabs/
Now, CONCENTRATE. Addition or Percentage (A/P)? A
Do you wish to limit the output values? (Y/N): N
Right, erm.. off I jolly well go!
vap.01.1901.glo
vap.02.1901.glo
(etc)
<END_QUOTE>

<BEGIN_QUOTE>
crua6[/cru/cruts/version_3_0/secondaries/vap] ./mergegrids
Welcome! This is the MERGEGRIDS program.
I will create decadal and full gridded files
from the output files of (eg) glo2abs.for.

Enter a gridfile with YYYY for year and MM for month: vapabs/vap.MM.YYYY.glo.abs
Enter Start Year: 1901
Enter Start Month: 01
Enter End Year: 2006
Enter End Month: 12

Please enter a sample OUTPUT filename, replacing
start year with SSSS and end year with EEEE: cru_ts_3_00.SSSS.EEEE.vap.dat
Writing cru_ts_3_00.1901.1910.vap.dat
Writing cru_ts_3_00.1911.1920.vap.dat
Writing cru_ts_3_00.1921.1930.vap.dat
Writing cru_ts_3_00.1931.1940.vap.dat
Writing cru_ts_3_00.1941.1950.vap.dat
Writing cru_ts_3_00.1951.1960.vap.dat
Writing cru_ts_3_00.1961.1970.vap.dat
Writing cru_ts_3_00.1971.1980.vap.dat
Writing cru_ts_3_00.1981.1990.vap.dat
Writing cru_ts_3_00.1991.2000.vap.dat
Writing cru_ts_3_00.2001.2006.vap.dat
<END_QUOTE>

Sadly, that gave the same result. So what of the published (v2.10) VAP dataset? That looks ~ok:

<BEGIN_QUOTE>
crua6[/cru/cruts/version_3_0/secondaries/vap] ./crutsstats

CRUTSSTATS: Stats for CRU TS gridded files

Enter the monthly gridded data file: cru_ts_2_10.1901-2002.vap.grid

Please enter the start year: 1901

102 years from 1901 to 2002

Output file is cru_ts_2_10.1901-2002.vap.grid.stats
1901 0 411 105
1902 0 413 104
1903 0 465 104
1904 0 359 104
1905 0 383 104
1906 0 376 105
1907 0 387 104
(etc)
<END_QUOTE>

Not good at all. Or, rather, good that it must be a solvable problem. Except that it's 10 to 5 on a Sunday
afternoon and it's me that's got to solve it.

Where to start? Well, retrace your steps, that's how you get out of a minefield. So first up, to compare
similar months in the anomaly files. Though I already know what I'm going to find, don't I? Because glo2abs
isn't going to do anything unusual, it just adds the normal and there you go. So if the absolutes are very
similar, the anomalies will be, too.. hmm. Well, I *suppose* I could try producing two more copies of the
output files - one with just synthetic data and one with just observed data? It's only a couple of re-runs
of the quick_interp_tdm2.pro IDL routine..

Started with the synthetic-only run:

<BEGIN_QUOTE>
IDL> quick_interp_tdm2,1901,2006,'vapsynglo/vapsyn.',1000,gs=0.5,dumpglo='dumpglo',nostn=1,synth_prefix='vapsyn/vapsyn'

crua6[/cru/cruts/version_3_0/secondaries/vap/syn_only] ./glo2abs
Welcome! This is the GLO2ABS program.
I will create a set of absolute grids from
a set of anomaly grids (in .glo format), also
a gridded version of the climatology.
Enter the path and name of the normals file: ../clim.6190.lan.vap
Enter a name for the gridded climatology file: clim.6190.lan.vap.grid
Enter the path and stem of the .glo files: vapsynglo/vapsyn.
Enter the starting year: 1901
Enter the ending year: 2006
Enter the path (if any) for the output files: vapsynabs/
Now, CONCENTRATE. Addition or Percentage (A/P)? A
Do you wish to limit the output values? (Y/N): N
Right, erm.. off I jolly well go!
vapsyn.01.1901.glo
vapsyn.02.1901.glo
(etc)

crua6[/cru/cruts/version_3_0/secondaries/vap/syn_only] ./mergegrids
Welcome! This is the MERGEGRIDS program.
I will create decadal and full gridded files
from the output files of (eg) glo2abs.for.

Enter a gridfile with YYYY for year and MM for month: vapsynabs/vapsyn.MM.YYYY.glo.abs
Enter Start Year: 1901
Enter Start Month: 01
Enter End Year: 2006
Enter End Month: 12

Please enter a sample OUTPUT filename, replacing
start year with SSSS and end year with EEEE: cru_ts_3_00.SSSS.EEEE.vap.syn.dat
Writing cru_ts_3_00.1901.1910.vap.syn.dat
Writing cru_ts_3_00.1911.1920.vap.syn.dat
Writing cru_ts_3_00.1921.1930.vap.syn.dat
Writing cru_ts_3_00.1931.1940.vap.syn.dat
Writing cru_ts_3_00.1941.1950.vap.syn.dat
Writing cru_ts_3_00.1951.1960.vap.syn.dat
Writing cru_ts_3_00.1961.1970.vap.syn.dat
Writing cru_ts_3_00.1971.1980.vap.syn.dat
Writing cru_ts_3_00.1981.1990.vap.syn.dat
Writing cru_ts_3_00.1991.2000.vap.syn.dat
Writing cru_ts_3_00.2001.2006.vap.syn.dat
<END_QUOTE>

And then the observed-only:

<BEGIN_QUOTE>
IDL> quick_interp_tdm2,1901,2006,'vapobsglo/vapobs.',1000,gs=0.5,dumpglo='dumpglo',pts_prefix='vaptxt/vap.'

crua6[/cru/cruts/version_3_0/secondaries/vap/obs_only] ./glo2abs
Welcome! This is the GLO2ABS program.
I will create a set of absolute grids from
a set of anomaly grids (in .glo format), also
a gridded version of the climatology.
Enter the path and name of the normals file: ../clim.6190.lan.vap
Enter a name for the gridded climatology file: clim.6190.lan.vap.grid
Enter the path and stem of the .glo files: vapobsglo/vapobs.
Enter the starting year: 1901
Enter the ending year: 2006
Enter the path (if any) for the output files: vapobsabs/
Now, CONCENTRATE. Addition or Percentage (A/P)? A
Do you wish to limit the output values? (Y/N): N
Right, erm.. off I jolly well go!
vapobs.01.1901.glo
vapobs.02.1901.glo
(etc)

crua6[/cru/cruts/version_3_0/secondaries/vap/obs_only] ./mergegrids
Welcome! This is the MERGEGRIDS program.
I will create decadal and full gridded files
from the output files of (eg) glo2abs.for.

Enter a gridfile with YYYY for year and MM for month: vapobsabs/vapobs.MM.YYYY.glo.abs
Enter Start Year: 1901
Enter Start Month: 01
Enter End Year: 2006
Enter End Month: 12

Please enter a sample OUTPUT filename, replacing
start year with SSSS and end year with EEEE: cru_ts_3_00.SSSS.EEEE.vap.obs.dat
Writing cru_ts_3_00.1901.1910.vap.obs.dat
Writing cru_ts_3_00.1911.1920.vap.obs.dat
Writing cru_ts_3_00.1921.1930.vap.obs.dat
Writing cru_ts_3_00.1931.1940.vap.obs.dat
Writing cru_ts_3_00.1941.1950.vap.obs.dat
Writing cru_ts_3_00.1951.1960.vap.obs.dat
Writing cru_ts_3_00.1961.1970.vap.obs.dat
Writing cru_ts_3_00.1971.1980.vap.obs.dat
Writing cru_ts_3_00.1981.1990.vap.obs.dat
Writing cru_ts_3_00.1991.2000.vap.obs.dat
Writing cru_ts_3_00.2001.2006.vap.obs.dat
<END_QUOTE>

So.. how do the stats look for these two datasets?

Synthetic-only:

<BEGIN_QUOTE>
crua6[/cru/cruts/version_3_0/secondaries/vap/syn_only] ./crutsstats

CRUTSSTATS: Stats for CRU TS gridded files

Enter the monthly gridded data file: cru_ts_3_00.1901.2006.vap.syn.dat

Please enter the start year: 1901

106 years from 1901 to 2006

Output file is cru_ts_3_00.1901.2006.vap.syn.dat.stats
1901 1 358 106
1902 1 358 106
1903 1 358 106
1904 1 358 106
1905 1 358 106
1906 1 358 106
(etc)
<END_QUOTE>

Observed-only:

<BEGIN_QUOTE>
crua6[/cru/cruts/version_3_0/secondaries/vap/obs_only] ./crutsstats

CRUTSSTATS: Stats for CRU TS gridded files

Enter the monthly gridded data file: cru_ts_3_00.1901.2006.vap.obs.dat

Please enter the start year: 1901

106 years from 1901 to 2006

Output file is cru_ts_3_00.1901.2006.vap.obs.dat.stats
1901 1 358 106
1902 1 358 106
1903 1 358 106
1904 1 358 106
1905 1 358 106
1906 1 358 106
(etc)
<END_QUOTE>

Oh, GOD. What is going on? Are we data sparse and just looking at the climatology? How can a synthetic
dataset derived from tmp and dtr produce the same statistics as an 'real' dataset derived from observations?

Let's be logical. Here are the two 'separated' gridding runs:

IDL> quick_interp_tdm2,1901,2006,'vapsynglo/vapsyn.',1000,gs=0.5,dumpglo='dumpglo',nostn=1,synth_prefix='vapsyn/vapsyn'
IDL> quick_interp_tdm2,1901,2006,'vapobsglo/vapobs.',1000,gs=0.5,dumpglo='dumpglo',pts_prefix='vaptxt/vap.'

Well they look fine. The synthetic run has no other data inputs ('nostn=1'), and the observed run has no references to
the synthetic data. So.. either quick_interp_tdm2.pro is doing something 'unusual', or, or.. hang on, let's try the
climatology for stats:

1961 1 311 80 1 320 83 1 315 89 1 320 98 1 346 111 1 358 128 1 356 143 1 342 140 1 354 123 1 323 104 1 318 90 1 315 82 1 358 106

Ah, Bingo was his name-o! as I was hoping (well OK it's a bad kind of hope), the reason it's all the same is that it is
by and large defaulting to the climatology. Which means that not much (any?) data is getting through, no matter if we
use synthetic, observed, or both together. What's odd about that conclusion is that the synthetic data is derived from
TMP and DTR - two very well-populated datasets! So synthetics alone should pretty much fill the.. hang on, just though
of something horrendous.. oh, okay, probably not that. I was wondering if glo2abs.for was factoring the normals so that
the anomalies were insignificant, but the equation is:

absgrid(ilon(i),ilat(i)) =
* nint(anoms(ilon(i),ilat(i))*10) + normals(i,imo)

..so the anomaly is getting the weight! But still - - not a wise thing to leave to automatics. So glo2abs should prompt
the user.. but with what? Just one anomaly and normal? Several? The same one from different timesteps? Eeek. Let's look
at this actual case.

January 1961, lines 11103, 11104 in the glo file (11099, 11100 without header, putting it on about 33.5 degs N)
0.0000E+00 0.0000E+00 0.0000E+00 0.0000E+00 0.0000E+00 0.0000E+00 4.7173E-04 4.7224E-03
5.4273E-03 6.1323E-03 6.8372E-03 7.5422E-03 8.2472E-03 1.9677E-03 0.0000E+00 0.0000E+00

Those anomalies are mighty tiny, given that the absolutes are three-digit integers! Hardly surprising they're not really
appearing on the radar when added to normals typically two orders of magnitude higher! Even with the *10 in the glo2abs
prog, we're still looking at values around 0.06.

Looked at the observed anomalies (output from anomdtb.f90) - here the anomalies are larger! Between -5 and +5, roughly,
which is what I'm used to seeing in .txt files.

To investigate the synthetics, I needed to look at re-run vap_gts_tdm.pro. It says,

; Note that anomalies are in hPa*10 (bin) or hPa (glo)

So the binary file anomaly units - the ones we're using - are in hPa*10. Let's get one o' them synthetic glo files:

IDL> vap_gts_anom,dtr_prefix='../dtrbin/dtrbin',tmp_prefix='../tmpbin/tmpbin',1961,1961,outprefix='vapsynglo/vapsyn.',dumpglo=1
Land,sea: 56016 68400
Calculating tmn normal
% Compiled module: TVAP.
Calculating synthetic vap normal
% Compiled module: ESAT.
Calculating synthetic anomalies
% Compiled module: MOMENT.
1961 vap (x,s2,<<,>>): 5.72571e-05 9.01807e-07 -0.0653905 0.0261283
% Compiled module: SAVEGLO.
% Compiled module: SELECTMODEL.

For Jan 1961 (may as well stick with it), -999 is the missing value code. The range is -0.0149 to +0.0222 (remember this is
an anomaly in hPa according to the program comment). So if it's telling the truth, the binary anomalies presented to
quick_interp_tdm2.pro will range from roughly -0.3 to +0.3. still nt going to impinge on normals between 1 and 358, is it?

So, what are the normals in? Well according to clim.6190.lan.vap:

crua6[/cru/cruts/version_3_0/secondaries/vap] head -11 clim.6190.lan.vap
Tyndall Centre grim file created on 12.01.2004 at 11:47 by Dr. Tim Mitchell
.vap = vapour pressure (hPa)
0.5deg lan clim:1961-90 MarkNew
[Long=-180.00, 180.00] [Lati= -90.00, 90.00] [Grid X,Y= 720, 360]
[Boxes= 67420] [Years=1975-1975] [Multi= 0.1000] [Missing=-999]
Grid-ref= 1, 148
291 294 296 293 287 279 265 262 271 279 286 287
Grid-ref= 1, 311
14 11 13 21 44 69 92 90 65 37 22 14
Grid-ref= 1, 312
13 10 12 20 43 67 90 87 63 35 21 13

That's what I've been missing! D'oh. That '[Multi= 0.1000]'. That would still only give a range of 0.1 to 35.8 hPa, and
my anomalies are still around 0.006 (or 0.3 for synthetics).

Two things, then. Firstly to get glo2abs to read the multiplicative factor from the climatology header and impose it on the
output. Secondly to work out why all the anomalies have different magnitudes! Or is vapour pressure really so teeny?

Working on glo2abs. Well my theory for additive anomalies is this: I read in the normals, and apply the multiplicative factor
in the header (for VAP it's 0.1). I assume the anomalies are already in the relevant units (ie require no factoring). This
looks to be the case for .txt files anyway. So I can add the anomaly to the adjusted normal. Then (because I need integer
output) I can DIVIDE by the factor (because that got us from integer to real before). Fine in theory but it all depends on
the anomalies being in regular 'units' (why wouldn't they be? They're reals!). OK, check from the beginning, obs first:

Database: hPa*10 (typically 3-digit integers)

anomdtb.for calls subroutine CheckVariSuffix, which contains:

<BEGIN_QUOTE>
else if (Suffix.EQ.".vap") then
Variable="vapour pressure (hPa)"
Factor = 0.1
<END_QUOTE>

And how does anomdtb.f90 use the Factor? well in the original version:

<BEGIN_QUOTE>
crua6[/cru/cruts/untouched/code/linux/cruts] grep 'Factor' anomdtb.f90
real :: MissThresh,StdevThresh,DistanceThresh,Factor, ExeSpace,WyeSpace
call CheckVariSuffix (LoadSuffix,Variable,Factor)
OpTot = OpTot + (real(DataA(XAYear,XMonth,XAStn))/Factor)
OpTotSq = OpTotSq + ((real(DataA(XAYear,XMonth,XAStn))/Factor) ** 2)
NormMean (XMonth,XAStn) = Factor*OpTot/OpEn
if (OpTotSq.GT.0) NormStdev (XMonth,XAStn) = Factor*sqrt((OpTotSq/OpEn)-((OpTot/OpEn)**2))
OpTot = OpTot + (real(DataA(XAYear,XMonth,XAStn))/Factor)
OpTotSq = OpTotSq + ((real(DataA(XAYear,XMonth,XAStn))/Factor) ** 2)
NormMean (XMonth,XAStn) = Factor*OpTot/OpEn
NormStdev (XMonth,XAStn) = Factor*sqrt((OpEn/(OpEn-1))*((OpTotSq/OpEn)-((OpTot/OpEn)**2)))
OpTot = OpTot + (DataA(XAYear,XMonth,XAStn)/Factor)
OpTotSq = OpTotSq + (DataA(XAYear,XMonth,XAStn)/Factor) ** 2
OpStDev = Factor*sqrt((OpEn/(OpEn-1))*((OpTotSq/OpEn)-((OpTot/OpEn)**2)))
OpMean = Factor*(OpTot/OpEn)
ALat(XAStn),ALon(XAStn),AElv(XAStn),real(DataA(XAYear,XMonth,XAStn))*Factor,AStn(XAStn)
<END_QUOTE>

I *think* the factor is being used multiplicatively. I don't understand why it's being used as a divisor though.. I must
have understood last December because I managed to rewrite the 'standard deviation' section, also using it as a divisor!

One obvious thing to try is to use the revised glo2abs. That should now be working in 'units' (but saving in whatever
range the normals are in). After that I could try comparing the old and 'new' (ie modded by me) versions of anomdtb.f90
to ensure I didn't break something (sure I didn't, but still..)

So, I revised glo2abs. It now reads the 'Multi' factor from the climatology header, and applies it to the normals before
they're used.

So, re-ran quick_interp+tdm2.pro:

IDL> quick_interp_tdm2,1901,2006,'vapglo/vap.',1000,gs=0.5,dumpglo='dumpglo',synth_prefix='vapsyn/vapsyn',pts_prefix='vaptxt/vap.'

A sample of the outputs, vap.12.1962.glo, had a range of values from -2.3006 to +1.8388, with the majority being 0. A total
of 56387 cells were nonzero, which given that there are 67420 land cells, isn't too bad. It's a pretty gaussian distribution,
too. It still seems like a small variation (typically +/- 0.5). For the cell where I live (Norwich, 363,286), the normals are:

Grid-ref= 363, 286
71 69 76 86 107 129 147 149 135 115 88 77

Or in hPa:

Grid-ref= 363, 286
7.1 6.9 7.6 8.6 10.7 12.9 14.7 14.9 13.5 11.5 8.8 7.7

The nearest station (well based on a quick search) is LOWESTOFT. Taking 1962 and 1963 and scaling:

62 7.6 6.9 6.5 9.2 10.9 12.6 14.4 15.0 13.6 12.3 8.9 6.5
63 5.4 5.5 7.9 9.9 11.1 14.8 15.8 15.1 14.6 11.7 10.3 6.9

The ranges:
2.2 1.4 1.4 0.7 0.2 2.2 1.4 0.1 1.0 0.6 1.4 0.4

Well our sample December 1962 range of anomalies was -2.3006 to +1.8388, and the January range is -3.3640 to +2.1250. So, I
have to admit, that's the same order of magnitude for our particular cell, year and month(s).

So, assuming these .glo files are OK, we'll try glo2abs again:

<BEGIN_QUOTE>
crua6[/cru/cruts/version_3_0/secondaries/vap] ./glo2abs
Welcome! This is the GLO2ABS program.
I will create a set of absolute grids from
a set of anomaly grids (in .glo format), also
a gridded version of the climatology.
Enter the path and name of the normals file: clim.6190.lan.vap
Enter a name for the gridded climatology file: deleteme1
Enter the path and stem of the .glo files: vapglo/vap.
Enter the starting year: 1901
Enter the ending year: 2006
Enter the path (if any) for the output files: vapabs/
Now, CONCENTRATE. Addition or Percentage (A/P)? A
Do you wish to limit the output values? (Y/N): Y
1. Set minimum to zero
2. Set a single minimum and maximum
3. Set monthly minima and maxima (for wet/rd0)
Choose: 1
Right, erm.. off I jolly well go!
vap.01.1901.glo
vap.02.1901.glo
(etc)
<END_QUOTE>

..and the result.. look good! For (again) December 1962:

Min 0 (well I did set that, see above)
Max 315

Number of zeros: 1078, perfectly respectable although I do wonder if VAP=0 is illegal.. hmm.. OK, added an option in glo2abs:

<BEGIN_QUOTE>
crua6[/cru/cruts/version_3_0/secondaries/vap] ./glo2abs
Welcome! This is the GLO2ABS program.
I will create a set of absolute grids from
a set of anomaly grids (in .glo format), also
a gridded version of the climatology.
Enter the path and name of the normals file: clim.6190.lan.vap
Enter a name for the gridded climatology file: deleteme3
Enter the path and stem of the .glo files: vapglo/vap.
Enter the starting year: 1901
Enter the ending year: 2006
Enter the path (if any) for the output files: vapabs/
Now, CONCENTRATE. Addition or Percentage (A/P)? A
Do you wish to limit the output values? (Y/N): Y
1. Set minimum to zero
2. Set a single minimum and maximum
3. Set monthly minima and maxima (for wet/rd0)
4. Set all values >0, (ie, positive)
Choose: 4
Right, erm.. off I jolly well go!
vap.01.1901.glo
vap.02.1901.glo
(etc)
<END_QUOTE>

Result for December 1962: Min 1, Max 315. A good spread of values, without a disproportionate number of '1's, I'm please
to say.

So, to generate the output files. Again.

<BEGIN_QUOTE>
crua6[/cru/cruts/version_3_0/secondaries/vap] ./mergegrids
Welcome! This is the MERGEGRIDS program.
I will create decadal and full gridded files
from the output files of (eg) glo2abs.for.

Enter a gridfile with YYYY for year and MM for month: vapabs/vap.MM.YYYY.glo.abs
Enter Start Year: 1901
Enter Start Month: 01
Enter End Year: 2006
Enter End Month: 12

Please enter a sample OUTPUT filename, replacing
start year with SSSS and end year with EEEE: cru_ts_3_00.SSSS.EEEE.vap.dat
Writing cru_ts_3_00.1901.1910.vap.dat
Writing cru_ts_3_00.1911.1920.vap.dat
Writing cru_ts_3_00.1921.1930.vap.dat
Writing cru_ts_3_00.1931.1940.vap.dat
Writing cru_ts_3_00.1941.1950.vap.dat
Writing cru_ts_3_00.1951.1960.vap.dat
Writing cru_ts_3_00.1961.1970.vap.dat
Writing cru_ts_3_00.1971.1980.vap.dat
Writing cru_ts_3_00.1981.1990.vap.dat
Writing cru_ts_3_00.1991.2000.vap.dat
Writing cru_ts_3_00.2001.2006.vap.dat
<END_QUOTE>

And what of the statistics. Well by now I've realised that we don't have complete coverage! So the normals are
bound to poke through quite a bit. In fact, the story is as it was in the beginning! *cries*

<BEGIN_QUOTE>
crua6[/cru/cruts/version_3_0/secondaries/vap] ./crutsstats

CRUTSSTATS: Stats for CRU TS gridded files

Enter the monthly gridded data file: cru_ts_3_00.1901.2006.vap.dat

Please enter the start year: 1901

106 years from 1901 to 2006

Output file is cru_ts_3_00.1901.2006.vap.dat.stats
1901 1 358 106
1902 1 358 106
1903 1 358 106
1904 1 358 106
1905 1 358 106
1906 1 358 106
1907 1 358 106
1908 1 358 106
(etc)
<END_QUOTE>

Now admittedly, the 106 mean does vary.. it hioits the dizzying heights of 107 on occasion! With a couple of 105s
thrown in to balance the books. Had a look at the stats in detail, compared to those for CRU TS 2.10. And guess
what? Yes.. the old stats are better! Here's the first decade:

CRU TS 2.10
1901 0 324 79 0 338 82 0 314 88 0 321 97 0 411 110 0 378 128 0 358 143 0 343 140 0 353 122 0 332 103 0 318 88 0 314 81 0 411 105
1902 0 312 80 0 319 82 0 314 87 0 321 96 0 413 109 0 366 125 0 356 141 0 343 138 0 353 122 0 323 102 0 318 88 0 315 80 0 413 104
1903 0 314 79 0 331 82 0 315 88 0 334 95 0 465 109 0 359 125 0 371 141 0 359 139 0 353 122 0 323 102 0 318 88 0 315 80 0 465 104
1904 0 310 78 0 319 81 0 312 86 0 321 95 0 347 109 0 359 126 0 355 140 0 344 138 0 354 121 0 323 103 0 318 89 0 316 81 0 359 104
1905 0 314 79 0 319 79 0 321 86 0 326 95 0 346 109 0 383 127 0 356 142 0 344 139 0 353 122 0 330 103 0 318 90 0 321 82 0 383 104
1906 0 328 80 0 330 81 0 323 87 0 335 98 0 376 111 0 359 128 0 356 142 0 343 140 0 353 122 0 323 103 0 318 89 0 316 82 0 376 105
1907 0 312 79 0 327 80 0 314 87 0 321 94 0 387 106 0 359 125 0 379 140 0 343 139 0 353 122 0 323 104 0 318 87 0 316 81 0 387 104
1908 0 312 79 0 323 81 0 330 86 0 338 95 0 346 109 0 359 127 0 353 142 0 343 138 0 353 122 0 316 102 0 318 87 0 316 81 0 359 104
1909 0 312 79 0 319 81 0 323 87 0 321 94 0 346 107 0 359 125 0 355 141 0 343 140 0 354 122 0 320 103 0 318 90 0 316 81 0 359 104
1910 0 312 80 0 319 82 0 315 86 0 321 95 0 347 109 0 359 126 0 383 142 0 343 139 0 353 122 0 318 102 0 318 87 0 316 80 0 383 104

CRU TS 3.00
1901 1 311 80 1 320 83 1 315 89 1 320 98 1 346 111 1 358 128 1 356 143 1 342 140 1 354 123 1 323 104 1 318 90 1 315 82 1 358 106
1902 1 311 80 1 320 83 1 315 89 1 320 98 1 346 111 1 358 128 1 356 143 1 342 140 1 354 123 1 323 104 1 318 90 1 315 82 1 358 106
1903 1 311 80 1 320 83 1 315 89 1 320 98 1 346 111 1 358 128 1 356 143 1 342 140 1 354 123 1 323 104 1 318 90 1 315 82 1 358 106
1904 1 311 80 1 320 82 1 315 89 1 320 98 1 346 111 1 358 128 1 356 143 1 342 140 1 354 123 1 323 104 1 318 90 1 315 82 1 358 106
1905 1 311 80 1 320 83 1 315 88 1 320 98 1 346 111 1 358 128 1 356 143 1 342 140 1 354 123 1 323 104 1 318 90 1 315 82 1 358 106
1906 1 311 80 1 320 83 1 315 89 1 320 98 1 346 111 1 358 128 1 356 143 1 342 140 1 354 123 1 323 104 1 318 90 1 315 82 1 358 106
1907 1 311 80 1 320 83 1 315 89 1 320 98 1 346 111 1 358 128 1 356 143 1 342 141 1 354 123 1 323 104 1 318 90 1 315 82 1 358 106
1908 1 311 80 1 320 83 1 315 89 1 320 98 1 346 111 1 358 129 1 356 143 1 342 140 1 354 123 1 323 104 1 318 90 1 315 82 1 358 106
1909 1 311 80 1 320 83 1 315 89 1 320 98 1 346 111 1 358 128 1 356 143 1 342 140 1 354 123 1 323 104 1 318 90 1 315 82 1 358 106
1910 1 311 80 1 320 83 1 315 89 1 320 98 1 346 111 1 358 128 1 356 143 1 342 140 1 354 123 1 323 104 1 318 90 1 315 82 1 358 106

..and here's a more recent decade:

CRU TS 2.10
1991 0 314 82 0 322 84 0 331 90 0 672 100 0 523 113 0 540 134 0 607 147 0 424 143 0 353 125 0 328 106 0 386 91 0 350 83 0 672 108
1992 0 337 82 0 383 84 0 450 90 0 613 98 0 347 112 0 359 128 0 373 140 0 345 140 0 353 122 0 347 103 0 414 89 0 384 83 0 613 106
1993 0 324 81 0 403 83 0 449 90 0 622 98 0 518 113 0 534 131 0 652 147 0 398 143 0 353 122 0 333 105 0 408 89 0 339 84 0 652 107
1994 0 346 82 0 396 82 0 457 90 0 626 100 0 524 113 0 507 132 0 605 146 0 416 143 0 349 125 0 332 107 0 397 93 0 341 84 0 626 108
1995 0 369 83 0 406 86 0 461 90 0 686 100 0 505 114 0 565 134 0 673 146 0 492 147 0 364 127 0 342 108 0 427 91 0 339 82 0 686 109
1996 0 334 81 0 431 83 0 548 88 0 634 97 0 524 113 0 530 131 0 645 147 0 422 142 0 366 124 0 337 106 0 413 91 0 344 84 0 645 107
1997 0 367 82 0 322 84 0 348 90 0 323 99 0 344 113 0 484 133 0 426 147 0 523 145 0 353 126 0 348 108 0 345 93 0 370 86 0 523 109
1998 0 339 84 0 345 89 0 338 92 0 355 104 0 361 116 1 531 137 1 356 152 0 560 149 0 370 128 0 347 108 0 369 92 0 334 85 0 560 111
1999 0 323 83 0 334 86 0 324 90 0 336 100 0 362 113 0 487 132 0 362 148 0 357 143 1 353 127 0 331 107 0 337 91 0 316 85 0 487 109
2000 0 319 82 0 319 85 0 319 91 0 328 102 0 356 114 0 476 133 0 358 146 0 520 146 0 353 124 0 333 107 0 335 91 0 334 84 0 520 109

CRU TS 3.00
1991 1 311 81 1 320 83 1 320 90 1 320 100 1 346 113 1 358 132 1 356 146 1 342 143 1 354 125 1 323 105 1 318 91 1 315 82 1 358 108
1992 1 311 82 1 319 84 1 315 90 1 320 97 1 346 111 1 358 127 1 356 141 1 342 140 1 354 122 1 323 102 1 317 89 1 315 83 1 358 106
1993 1 313 81 1 315 83 1 315 89 1 320 98 1 346 112 1 358 131 1 356 146 1 342 142 1 354 122 1 323 103 1 323 88 1 317 83 1 358 106
1994 1 311 82 1 322 82 1 315 89 1 320 99 1 346 112 1 358 131 1 356 146 1 346 142 1 354 125 1 323 106 1 318 92 1 315 83 1 358 107
1995 1 311 82 1 318 85 1 320 90 1 324 99 1 346 112 1 358 131 1 356 146 1 345 144 1 354 124 1 323 107 1 321 90 1 315 81 1 358 108
1996 1 311 80 1 321 82 1 320 87 1 320 96 1 346 111 1 358 130 1 356 145 1 343 141 1 354 122 1 323 105 1 318 90 1 319 82 1 358 106
1997 1 311 81 1 320 84 1 315 90 1 320 99 1 346 113 1 358 131 1 356 145 1 342 143 1 354 123 1 323 106 1 318 90 1 315 83 1 358 107
1998 1 311 81 1 334 85 1 326 89 1 338 100 1 346 114 1 358 134 1 356 148 1 342 145 1 354 125 1 323 105 1 318 89 1 315 84 1 358 108
1999 1 316 82 1 320 85 1 322 88 1 320 99 1 346 112 1 358 131 1 356 148 1 342 142 1 354 125 1 323 106 1 318 91 1 315 84 1 358 108
2000 1 317 82 1 320 84 1 315 90 1 320 100 1 346 113 1 358 131 1 356 146 1 342 144 1 354 123 1 323 105 1 318 90 1 315 83 1 358 108

I DON'T UNDERSTAND!!!!!

Well, OK - I see that a VAP of zero is acceptable. Though as it's a pressure, I don't believe it! I'll stick with 1.

The issue is that the earlier dataset has a variability (in the maximum) that we just don't have in the new one. And
I feel that I've been through every bloody phase of the process and checked we're doing it right!!!

~~~

Right. Let's look at the distributions of values in each dataset. We'll take Jan 1910 and Jun 2000. And as this is
a textual document, I'll have to describe the results.

Offsets. Well each month has 360 lines, so each year has 4320 lines. So for Jan 1910 we need to skip nine years,
or 38880 lines, then take the next 360. For Jun 2000 we need to skip 99 years, or 427680 lines, then another five
months, or 1800 lines, then take the next 360. So:

head -39240 cru_ts_2.10.1901-2002.vap.dat |tail -360 > cru_ts_2.10.Jan.1910.vap.dat
head -39240 cru_ts_3.00.1901.2006.vap.dat |tail -360 > cru_ts_3.00.Jan.1910.vap.dat

head -428040 cru_ts_2.10.1901-2002.vap.dat |tail -360 > cru_ts_2.10.Jun.2000.vap.dat
head -428040 cru_ts_3_00.1901.2006.vap.dat |tail -360 > cru_ts_3_00.Jun.2000.vap.dat

I loaded the resultant monthly files into Matlab, and played with them mercilessly.

Well to start with, they all look the same. Truly. I've got a 4-plot page with TS 2.10 in the left-hand column,
and TS 3.00 on the right. January 1910 on the top, June 2000 on the bottom. and they look pretty much inseparable,
though if I had to Spot The Difference, the TS 2.10 June 2000 distribution is a little flatter (that is, the
massive spike at the low end is a little shorter, and the rest of the entourage are a little taller.

What are particularly worthy of note are the maximums. Because they don't match those produced by crutsstats.for.

Month Model Max (Matlab) Max (crutsstats)
Jan 1910 TS 2.10 312 312
Jan 1910 TS 3.00 311 311
Jun 2000 TS 2.10 319 476
Jun 2000 TS 3.00 317 358

Not entirely sure why the latter ones would be wrong. But I suspect crutsstats - because otherwise I miscounted
the line numbers to extract June 2000 with! Actually, OK, that does seem more likely.

Let's try it from the 1991-2000 files. The offset will be 9*4320 + 5*360 + 360 = 41040.

gunzip -c /cru/cruts/fromtyn1/data/cru_ts_2.10/newly_gridded/data_dec/cru_ts_2_10.1991-2000.vap.grid.gz | head -41040 | tail -360 > cru_ts_2_10.Jun.2000.vap.dat
gunzip -c cru_ts_3_00.1991.2000.vap.dat.gz | head -41040 | tail -360 > cru_ts_3_00.Jun.2000.vap.dat

Well - looks like I did miscount, because the new files are different! And so are the Maxima:

Month Model Max (Matlab) Max (crutsstats)
Jun 2000 TS 2.10 300 476
Jun 2000 TS 3.00 358 358

..so almost perfect. At least the stats for the file I'm creating match.

And now the June 2000 histograms are much more interesting! And of course (for this is THIS project), much
more worrying. The June 2000 plot for the new data (3.00) shows a fall at VAP ->0. This is in contrast to the
other three, which show a more expotential decline from a high near 0 (though admittedly the 2.10 version does have a second
peak at around 120). In fact, the June 2000 3.00 series has peaks at ~90 and ~300! Oh, help.

The big question must be, why does it have so little representation in the low numbers? Especially given that I'm rounding
erroneous negatives up to 1!!

Oh, sod it. It'll do. I don't think I can justify spending any longer on a dataset, the previous version of which was
completely wrong (misnamed) and nobody noticed for five years.