Having drawn out the process flowchart, I wondered if quick_interp_tdm2.pro would be kind
enough to output both .glo and binary gridded files, simultaneously? This would simplify
and speed things up a bit. So, with absolutely no alarm bells ringing at all, I decided
to make a sample run for DTR, just for 2006, to compare simultaneous outputs with the
original ones. You idiot.
IDL> quick_interp_tdm2,2006,2006,'testdtrglo/dtr.',750,gs=0.5,pts_prefix='dtrtxt/dtr.',dumpglo='dumpglo',dumpbin='dumpbin'
% Compiled module: QUICK_INTERP_TDM2.
% Compiled module: GLIMIT.
Defaults set
2006
% Compiled module: MAP_SET.
% Compiled module: CROSSP.
% Compiled module: STRIP.
% Compiled module: SAVEGLO.
% Compiled module: SELECTMODEL.
no stations found in: dtrtxt/dtr.2006.09.txt
no stations found in: dtrtxt/dtr.2006.10.txt
no stations found in: dtrtxt/dtr.2006.11.txt
no stations found in: dtrtxt/dtr.2006.12.txt
% Compiled module: MEAN.
% Compiled module: MOMENT.
% Compiled module: STDDEV.
grid 2006 non-zero 0.1125 2.1122 2.9219 cells= 202010
% Compiled module: WRBIN.
IDL> exit
crua6[/cru/cruts/version_3_0/primaries/dtr] ls -l testdtrglo/
total 43048
-rw------- 1 f098 cru 6220800 Feb 23 11:06 dtr.2006
-rw------- 1 f098 cru 3142986 Feb 23 11:06 dtr.2006.01.glo
-rw------- 1 f098 cru 3142986 Feb 23 11:06 dtr.2006.02.glo
-rw------- 1 f098 cru 3142986 Feb 23 11:06 dtr.2006.03.glo
-rw------- 1 f098 cru 3142986 Feb 23 11:06 dtr.2006.04.glo
-rw------- 1 f098 cru 3142986 Feb 23 11:06 dtr.2006.05.glo
-rw------- 1 f098 cru 3142986 Feb 23 11:06 dtr.2006.06.glo
-rw------- 1 f098 cru 3142986 Feb 23 11:06 dtr.2006.07.glo
-rw------- 1 f098 cru 3142986 Feb 23 11:06 dtr.2006.08.glo
-rw------- 1 f098 cru 3142986 Feb 23 11:06 dtr.2006.09.glo
-rw------- 1 f098 cru 3142986 Feb 23 11:06 dtr.2006.10.glo
-rw------- 1 f098 cru 3142986 Feb 23 11:06 dtr.2006.11.glo
-rw------- 1 f098 cru 3142986 Feb 23 11:06 dtr.2006.12.glo
crua6[/cru/cruts/version_3_0/primaries/dtr]
So there, as hoped-for, binary and text output files. BUT. Comparisons with earlier
versions from the same database.. are depressingly awful:
Different again! Can this just be the random seed used in the gridding algorithm? If so, why aren't
we seeing a consistent pattern of 0.0 vs non-0.0 values? Another reason - if one were needed - why
we should dump this gridding approach altogether. But, er, not yet! No time to finish and test the
fortran gridder, which will doubtless sink to some depth and never be seen again, we'll carry on
with this mediocre approach.
Spent a whole day knocking up an anomaly program - as I felt anomdtb was vastly overweight and
supremely complicated to compile. Unfortunately, I got stuck trying to work out data and latlon
factors for different parameters, (argh! why?), and what percentage anomalies really were, and in
the end GAVE UP and now I have to modify anomdtb after all. Actually - that looked even worse, so
went back to anomauto and finished it off. And.. it works. Actually, a bit too well. For example,
when deriving anomalies from the CLD database, this was the original (a few weeks ago!):
OK. So we read in 987. Then we multiply by the factor, which should be 0.1, giving us 98.7.
Then we subtract the mean, giving us 98.7-90.14 = 8.56, which is what we're getting. So no
mismatches between data, time, and metadata. Good. and the 95/02 mean is right, too (90.1375).
So, er. AH! solved it. Looking at the wrong 'old' cloud text files. tadaa:
Hurrah. Now I need to know why I'm producing too many. It's not as bad, though:
OLD BUT CORRECT:
crua6[/cru/cruts/version_3_0/update_top] wc -l ../secondaries/cld/cldupdatetxt/cldupdate.2000.11.txt
760 ../secondaries/cld/cldupdatetxt/cldupdate.2000.11.txt
Now, our limit for a valid normal is 75%, which for 1995-2002 should mean 6.
BODO VI has five valid values in November. So our limit is either wrong, or not being applied.
..yup:
uealogin1[/cru/cruts/version_3_0/update_top] ./anomauto
minn calculated as 7
Ho hum. Recalculated it to 6 (whilst checking that 1961-1990 still gave 23). Re-ran.
To my horror - if not surprise - that let EVEN MORE IN! Well of course it did you silly sausage.
This still doesn't explain how BODO VI gets in with 5 values:
Aha. I wonder if I'm initialising the onestn() array in the wrong place? Because data is
only added if not -9999, so it has to be prefilled with -9999 *every time*.. dammit. If
I fix that, I get:
It's not going to be easy to find 14 missing stations, is it? Since the anomalies aren't exactly the same.
Should I be worried about 14 lost series? Less than 2%. Actually, I noticed something interesting.. look
at the anomalies. The anomdtb ones aren't *rounded* to 1dp, they're *truncated*! So, er - wrong!
So let's say, anomalies are done. Hurrah. Onwards, plenty more to do!
Got the gridding working, I think. IDL of course. I modified quick_interp_tdm2.pro to accept
start and end months, otherwise it just produces whole years with files of zeros for months with
no anomaly file. And errors. And since this is likely to be a six-month update..
Re-planned the program layout. Not a major exercise, just putting different loops in to speed up and
simplify operations. It now runs as follows (note this is simplified!!):
1. User chooses update databases or update datasets. Dates, parameters, etc.
2. Update Databases
2.1 Convert any MCDW bulletins to CRU format; merge into existing databases
2.2 Convert any CLIMAT bulletins to CRU format; merge into databases from 2.1
2.3 Convert any BOM bulletins to CRU format; merge into databases from 2.2
3. Update datasets
3.1 Convert databases to anomalies
3.2 Grid primary parameters
3.3 Generate synthetic secondary parameters
3.4 Grid secondary parameters
3.5 Convert gridded anomalies to actuals
3.6 Produce final datasets
1876 lines including subrotuines and notes. Ten Fortran and four IDL programs (plus indirect ones). All
Fortran programs are mine, now. Top-level listing:
drwx------ 10 f098 cru 4096 Feb 19 20:55 db
drwx------ 3 f098 cru 4096 Feb 28 17:01 reference
drwx------ 3 f098 cru 4096 Mar 1 15:41 runs
drwx------ 4 f098 cru 4096 Feb 23 12:15 gridded_finals
drwx------ 4 f098 cru 4096 Feb 27 17:56 results
drwx------ 5 f098 cru 4096 Mar 1 15:40 logs
drwx------ 6 f098 cru 4096 Dec 18 11:00 updates
drwx------ 8 f098 cru 4096 Feb 28 16:15 interim_data
-rw------- 1 f098 cru 11 Feb 27 17:48 newdata.latest.date
-rwxr-xr-x 1 f098 cru 132425 Mar 1 14:41 update
-rwxr-xr-x 1 f098 cru 16465 Mar 1 14:41 dtr2cldauto
-rwxr-xr-x 1 f098 cru 17990 Mar 1 14:55 tmnx2dtrauto
-rwxr-xr-x 1 f098 cru 19427 Mar 1 15:43 glo2absauto
-rwxr-xr-x 1 f098 cru 20929 Mar 1 14:42 movenormsuato
-rwxr-xr-x 1 f098 cru 23350 Mar 1 15:42 anomauto
-rwxr-xr-x 1 f098 cru 29076 Mar 1 14:50 climat2cruauto
-rwxr-xr-x 1 f098 cru 29481 Mar 1 14:50 bom2cruauto
-rwxr-xr-x 1 f098 cru 29867 Mar 1 14:49 mcdw2cruauto
-rwxr-xr-x 1 f098 cru 323870 Mar 1 15:52 makegridsauto
-rwxr-xr-x 1 f098 cru 89515 Mar 1 16:10 newmergedbauto
So, to station counts. These will have to mirror section 3 above. Coverage of secondary parameters is
particularly difficult - what is the best approach? To include synthetic coverage, when it's only at
2.5-degree?
No. I'm going to back my previous decision - all station count files reflect actualy obs for that
parameter only. So for secondaries, you get actual obs of that parameter (ie naff all for FRS). You
get the info about synthetics that enables you to use the relevant primary counts if you want to. Of
course, I'm going to have to provide a combined TMP and DTR station count to satisfy VAP & FRS users.
The problem is that the synthetics are incorporated at 2.5-degrees, NO IDEA why, so saying they affect
particular 0.5-degree cells is harder than it should be. So we'll just gloss over that entirely ;0)
ARGH. Just went back to check on synthetic production. Apparently - I have no memory of this at all -
we're not doing observed rain days! It's all synthetic from 1990 onwards. So I'm going to need
conditionals in the update program to handle that. And separate gridding before 1989. And what TF
happens to station counts?
OH FUCK THIS. It's Sunday evening, I've worked all weekend, and just when I thought it was done I'm
hitting yet another problem that's based on the hopeless state of our databases. There is no uniform
data integrity, it's just a catalogue of issues that continues to grow as they're found.
rd0_gts_anom_05 will produce half-degree .glo files from gridded pre anoms. So if we call that, we
can use it, and stncounts for PRE will be authentic (as it's the sole input). Final decision: coded
update.for to produce WET from obs+syn until 12/1989, syn only thereafter. WET station counts only
produced until 1989, PRE must be used (with caveats) after that point.
Wrote tmpdtrstnsauto.for to produce tmp.and.dtr station counts (ie you only get a count when both
parameters have a count, and even then it's the min()). The resulting counts are the effective FRS
counts, and the synthetic VAP counts.
Onto PET. Tracked down the PET program from Dimitrios, way back in 2007! It uses TMP, TMN, TMX, VAP,
CLD and WND (the latter as 61-90 normals from IPCC). Converted to f77 'automatic' (makepetauto.for).