The HARRY_READ_ME.txt file

Part 35r

Back to get CLD sorted out. Need a break from the updater! Though much the same
difficulties, trying to work out the process (it's anything but straightforward for
cloud, seeing as the incoming updates are in Sun Hours, and we have to apply our
own regressions to DTR). Knocked up a subroutine, 'sh2cp', to convert sun hours to
cloud percentage on the fly, and added it to mcdw2cruauto and climat2cruauto. Of course,
one of the problems is that you need a latitude value to perform the conversion - so
the CLIMAT bulletins lose the value if they can't be matched in the WMO list! Not much
I can do about that, and let's face it those stations are going to end up as 'new'
stations with no possibility of a 61-90 normal.

So.. using the new converters (which are built to be driven by the update program), I
set about converting MCDW and CLIMAT bulletins:

uealogin1[/cru/cruts/version_3_0/update_top] ./mcdw2cruauto
uealogin1[/cru/cruts/version_3_0/update_top] cat results/results.0901101032/mcdw.0901101032.res
OK
uealogin1[/cru/cruts/version_3_0/update_top]

uealogin1[/cru/cruts/version_3_0/update_top] ./climat2cruauto
uealogin1[/cru/cruts/version_3_0/update_top] cat results/results.0901101032/climat.0901101032.res
OK
uealogin1[/cru/cruts/version_3_0/update_top]

Gotta love silent running :=)

The output cld databases both look OK, and pretty much equivalent except that MCDW goes back
further (to 1994). CLIMAT is 2000 onwards because that's what's on Phil Brohan's website.

Now we have to merge. This is where it gets hairy.

Looking at existing cld databases, we have the originals (cld.0301081434.dtb and
cld.0312181428.dtb), and the 2007 version (cld.0711272230.dtb). Looking back through
the notes, this was the product of processing the MCDW and CLIMAT bulletins into
sun hours databases (sun.0711272156.dtb and sun.0711272219.dtb respectiverly), then
merging those to form sun.0711272225.dtb, then converting to cloud using Hsp2cld.for,
giving us cld.0711272230.dtb. So the new cloud databases I've just produced should
be, if not identical, very similar? Oh, dear. There is a passing similarity, though
this seems to break down in Winter. I don't have time to do detailed comparisons, of
course, so we'll just run with the new one. After all, I have tested the conversion
for the latest programs, I'm not sure how much testing was done last time.

The procedure last time - that is, when I was trying to re-produce TS 2.10, we have
no idea what the procedure was for its initial production! - was to incorporate the
sun percent data from the bulletins into the existing sun percent db (spc.0312221624.dtb).
The trouble is, the existing cloud dbs are bigger. They stop at 1996, but that's no
problem, since we have MCDW from then.

228936 cld.0301081434.dtb
104448 cld.0312181428.dtb
111989 combo.cld.dtb
57395 spc.0301201628.dtb
51551 spc.0312221624.dtb
51551 spc.94-00.0312221624.dtb

So, how about merging our new MCDW cloud database into cld.0312181428.dtb, then merging
the CLIMAT one into that? The logic here is that the cloud must be from observations,
because the sun databases are much smaller. Well, the ones we know about! It might be
worth checking the station numbers for each year though. Unfortunately, we don't have a
lot of luck merging MCDW updates into the Dec 2003 CLD database:

uealogin1[/cru/cruts/version_3_0/db/cld] ./newmergedb

WELCOME TO THE DATABASE UPDATER

Before we get started, an important question:
If you are merging an update - CLIMAT, MCDW, Australian - do
you want the quick and dirty approach? This will blindly match
on WMO codes alone, ignoring data/metadata checks, and making any
unmatched updates into new stations (metadata permitting)?

Enter 'B' for blind merging, or <ret>: B
Please enter the Master Database name: cld.0312181428.dtb
Please enter the Update Database name: mcdw.cld.0901101032.dtb

Reading in both databases..
Master database stations: 3605
Update database stations: 2204

Looking for WMO code matches..
5 reject(s) from update process 0902101404

Writing cld.0902101404.dtb

+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+
+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+

OUTPUT(S) WRITTEN

New master database: cld.0902101404.dtb

Update database stations: 2204
> Matched with Master stations: 858
(automatically: 858)
(by operator: 0)
> Added as new Master stations: 1341
> Rejected: 5
Rejects file: mcdw.cld.0901101032.dtb.rejected
uealogin1[/cru/cruts/version_3_0/db/cld]

Of course, as we are only generating from 1996 onwards, this probably isn't
of much significance. Luckily.

Next, merge CLIMAT into the new database. well of course this is much more
satisfactory, because it matches with the MCDW stations:

uealogin1[/cru/cruts/version_3_0/db/cld] ./newmergedb

WELCOME TO THE DATABASE UPDATER

Before we get started, an important question:
If you are merging an update - CLIMAT, MCDW, Australian - do
you want the quick and dirty approach? This will blindly match
on WMO codes alone, ignoring data/metadata checks, and making any
unmatched updates into new stations (metadata permitting)?

Enter 'B' for blind merging, or <ret>: B
Please enter the Master Database name: cld.0902101404.dtb
Please enter the Update Database name: climat.cld.0901101032.dtb

Reading in both databases..
Master database stations: 4946
Update database stations: 2038

Looking for WMO code matches..
2 reject(s) from update process 0902101409

Writing cld.0902101409.dtb

+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+
+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+

OUTPUT(S) WRITTEN

New master database: cld.0902101409.dtb

Update database stations: 2038
> Matched with Master stations: 1858
(automatically: 1858)
(by operator: 0)
> Added as new Master stations: 178
> Rejected: 2
Rejects file: climat.cld.0901101032.dtb.rejected
uealogin1[/cru/cruts/version_3_0/db/cld]

So we now have cld.0902101409.dtb, a database consisting of cld.0312181428.dtb,
updated first with derived-cloud data from MCDW (1994-2008), then with
derived-cloud data from CLIMAT (2000-2008). This should be anomalised, then
fed into quick_interp_tdm2.pro along with synthetic, DTR-derived cloud. So that
is the next hurdle.

Well, we have the program. And we've played with it, but forgot to c&p those runs
into here (well they were only a few days ago!) so here they are now:

crua6[/cru/cruts/version_3_0/secondaries/cld/cldfromdtrtxt] ./dtr2cld
Please enter the path/file of the FIRST dtr txt file: /cru/cruts/version_3_0/primaries/dtr/dtrtxt/dtr.1901.01.txt
Please enter the path/file of the LAST dtr txt file: /cru/cruts/version_3_0/primaries/dtr/dtrtxt/dtr.2006.12.txt
crua6[/cru/cruts/version_3_0/secondaries/cld/cldfromdtrtxt]

Then an experimental IDL gridding using half degree and glo output. It was late at night,
I think I had an idea of visualising or comparing.. no such luck.

crua6[/cru/cruts/version_3_0/secondaries/cld] idl
IDL Version 5.4 (OSF alpha). (c) 2000, Research Systems, Inc.
Installation number: 66286.
Licensed for use by: Climatic Research Unit

IDL> quick_interp_tdm2,1995,2006,'cld.',750,gs=0.5,dumpglo='dumpglo',pts_prefix='cldfromdtrtxt/cld.'
% Compiled module: QUICK_INTERP_TDM2.
% Compiled module: GLIMIT.
Defaults set
1995
% Compiled module: MAP_SET.
% Compiled module: CROSSP.
% Compiled module: STRIP.
% Compiled module: SAVEGLO.
% Compiled module: SELECTMODEL.
1996
1997
1998
1999
2000
2001
2002
2003
2004
2005
2006
no stations found in: cldfromdtrtxt/cld.2006.09.txt
no stations found in: cldfromdtrtxt/cld.2006.10.txt
no stations found in: cldfromdtrtxt/cld.2006.11.txt
no stations found in: cldfromdtrtxt/cld.2006.12.txt
IDL>

So now we need to try that last step again - this time going for 2.5-degree binary
outputs, suitable for feeding back into it for the full cloud gridding. Oh, my.

IDL> quick_interp_tdm2,1996,2006,'cldfromdtr25bin/cld.',750,gs=2.5,dumpbin='dumpbin',pts_prefix='cldfromdtrtxt/cld.'

<output removed as re-done below with CDD=600>

Okay, that's the synthetic binary cloud ready. Now we need to run anomdtb
on the 'observed' cloud database:

crua6[/cru/cruts/version_3_0/secondaries/cld] ./anomdtb

> ***** AnomDTB: converts .dtb to anom .txt for gridding *****

> Enter the suffix of the variable required (eg, .tmp):
.cld
> Select the .dtb file to load:
cld.0902101409.dtb
cld.0902101409.dtb


/tmp_mnt/cru-auto/cruts/version_3_0/secondaries/cld/cld.0902101409.dtb


> Specify the start,end of the normals period:
1961,1990
> Specify the missing percentage permitted:
25
> Data required for a normal: 23
> Specify the no. of stdevs at which to reject data:
3
The output selection is tied to 3 (ungridded anomalies)
> Check for duplicate stns after anomalising? (0=no,>0=km range)
0
> Select the generic .txt file to save (yy.mm=auto):
cldnew.txt
> Select the first,last years AD to save:
1996,2006
> Operating...
/tmp_mnt/cru-auto/cruts/version_3_0/secondaries/cld/cld.0902101409.dtb


/tmp_mnt/cru-auto/cruts/version_3_0/secondaries/cld/cld.0902101409.dtb


> NORMALS MEAN percent STDEV percent
> .dtb 0 0.0
> .cts 288328 22.2 288328 22.2
> PROCESS DECISION percent %of-chk
> no lat/lon 0 0.0 0.0
> no normal 1010030 77.8 77.8
> out-of-range 24 0.0 0.0
> accepted 288304 22.2
> Dumping years 1996-2006 to .txt files...

crua6[/cru/cruts/version_3_0/secondaries/cld]

Unfortunately, that isn't working. Too many stations outside the usual normals
period (1961-1990). My notes from the last attempt are less than inspiring.. it
looks as though we need the program 'normshift.for', and normalise 95-02. So:

crua6[/cru/cruts/version_3_0/secondaries/cld] ./anomdtb

> ***** AnomDTB: converts .dtb to anom .txt for gridding *****

> Enter the suffix of the variable required (eg, .tmp):
.cld
> Select the .dtb file to load:
cld.0902101409.dtb
cld.0902101409.dtb


/tmp_mnt/cru-auto/cruts/version_3_0/secondaries/cld/cld.0902101409.dtb


> Specify the start,end of the normals period:
1995,2002
> Specify the missing percentage permitted:
25
> Data required for a normal: 6
> Specify the no. of stdevs at which to reject data:
3
The output selection is tied to 3 (ungridded anomalies)
> Check for duplicate stns after anomalising? (0=no,>0=km range)
0
> Select the generic .txt file to save (yy.mm=auto):
cldupdate.txt
> Select the first,last years AD to save:
1996,2006
> Operating...
/tmp_mnt/cru-auto/cruts/version_3_0/secondaries/cld/cld.0902101409.dtb


/tmp_mnt/cru-auto/cruts/version_3_0/secondaries/cld/cld.0902101409.dtb


> NORMALS MEAN percent STDEV percent
> .dtb 0 0.0
> .cts 271495 20.9 271495 20.9
> PROCESS DECISION percent %of-chk
> no lat/lon 0 0.0 0.0
> no normal 1026863 79.1 79.1
> out-of-range 474 0.0 0.2
> accepted 271021 20.9
> Dumping years 1996-2006 to .txt files...

crua6[/cru/cruts/version_3_0/secondaries/cld]

Hmm.. that's giving us between 670 and 790 stations per month.. not too bad
I suppose seeing as it's a secondary parameter. Now for normshift, which has
already been run (search back in this file), producing clim.9502.to.6190.grid.cld,
which is your standard 12-grids-360rx720c, giving the diffs between 1995-2002
normals and 1961-1990 normals. So after gridding we could add these.. except that
after gridding we'll have incorporated the DTR_derived synthetic cloud, which is
of course based on the 1961-1990 normals as it's derived from DTR!! Arrrrggghh.

So.. {sigh}.. another problem. Well we can't change the updates side, that has to
use 1995-2002 normals. But maybe we'll have to adjust the station anomalies, prior
to gridding? I don't see an alternative.

Wrote movenorms.for, using the engine of dtr2cld (as it's processing the same kind
of files and also needs to map stations to cells). However we quickly hit a problem:

crua6[/cru/cruts/version_3_0/secondaries/cld] ./movenorms

Please enter the adjustment file: clim.9502.to.6190.grid.cld

Please enter a generic source file with MM for month and YYYY for year: cldupdatetxt/cldupdate.YYYY.MM.txt
Start YEAR: 1996
Start MONTH: 01
End YEAR: 2006
End MONTH: 12

Please enter a generic destination file with MM for month and YYYY for year: cldupdate6190/cldupdate6190.YYYY.MM.txt

ERROR. Station in sea:
File: cldupdate6190/cldupdate6190.1996.01.txt
Offending line: 18.54 72.49 11.0 -6.600004305700
Resulting indices ilat,ilon: 218 505
crua6[/cru/cruts/version_3_0/secondaries/cld]

This is a station on the West coast of India; probably Mumbai. Unfortunately, as a coastal
station it runs the risk of missing the nearest land cell. The simple movenorms program is
about to become less simple.. but was do-able. The log file was empty at the end, indicating
that all 'damp' stations had found dry land:

crua6[/cru/cruts/version_3_0/secondaries/cld] ./movenorms

Please enter the adjustment file: clim.9502.to.6190.grid.cld

Please enter a generic source file with MM for month and YYYY for year: cldupdatetxt/cldupdate.YYYY.MM.txt
Start YEAR: 1996
Start MONTH: 1
End YEAR: 2006
End MONTH: 12

Please enter a generic destination file with MM for month and YYYY for year: cldupdate6190/cldupdate6190.YYYY.MM.txt
crua6[/cru/cruts/version_3_0/secondaries/cld] wc -l movenorms.log
0 movenorms.log
crua6[/cru/cruts/version_3_0/secondaries/cld]

So.. now I should be able to do the final gridding of cloud for 1996-2006.

IDL> quick_interp_tdm2,1996,2006,'cloudcomboglo/cld.',750,gs=0.5,dumpglo='dumpglo',synth_prefix='cldfromdtr25bin/cld.',pts_prefix='cldupdate6190/cldupdate6190.'

<output removed as re-done below with CDD=600>

crua6[/cru/cruts/version_3_0/secondaries/cld] ./glo2abs

<output removed as re-done below with CDD=600>

uealogin1[/cru/cruts/version_3_0/secondaries/cld] ./makegrids

<output removed as re-done below with CDD=600>

All files look alright. BUT. The NetCDF attributes (which are still bad) do say that
the CDD for cloud is 600. If it is, I will eat my screen, because I'll have to do all
the gridding ops again :-((( and.. it is :-o

So.. binary re-production:


IDL> quick_interp_tdm2,1996,2006,'cldfromdtr25bin/cld.',600,gs=2.5,dumpbin='dumpbin',pts_prefix='cldfromdtrtxt/cld.'
% Compiled module: QUICK_INTERP_TDM2.
% Compiled module: GLIMIT.
Defaults set
1996
% Compiled module: MAP_SET.
% Compiled module: CROSSP.
% Compiled module: MEAN.
% Compiled module: MOMENT.
% Compiled module: STDDEV.
grid 1996 non-zero -22.2967 46.5119 134.6274 cells= 53787
% Compiled module: STRIP.
% Compiled module: WRBIN.
1997
grid 1997 non-zero -22.7474 47.1535 131.6472 cells= 53374
1998
grid 1998 non-zero -24.3090 50.5343 155.2392 cells= 53557
1999
grid 1999 non-zero -21.0658 46.2280 127.9565 cells= 52391
2000
grid 2000 non-zero -23.3182 50.4612 154.3142 cells= 51948
2001
grid 2001 non-zero -25.4712 50.3292 147.5332 cells= 50464
2002
grid 2002 non-zero -22.6252 49.8823 153.3252 cells= 46980
2003
grid 2003 non-zero -21.8382 48.3537 136.5305 cells= 48279
2004
grid 2004 non-zero -23.9221 47.9721 145.4819 cells= 48179
2005
grid 2005 non-zero -26.0049 48.8422 145.3165 cells= 44448
2006
no stations found in: cldfromdtrtxt/cld.2006.09.txt
no stations found in: cldfromdtrtxt/cld.2006.10.txt
no stations found in: cldfromdtrtxt/cld.2006.11.txt
no stations found in: cldfromdtrtxt/cld.2006.12.txt
grid 2006 non-zero -17.3353 45.5259 135.7803 cells= 29194
IDL>

..and finals re-produced:

IDL> quick_interp_tdm2,1996,2006,'cloudcomboglo/cld.',600,gs=0.5,dumpglo='dumpglo',synth_prefix='cldfromdtr25bin/cld.',pts_prefix='cldupdate6190/cldupdate6190.'
Defaults set
1996
% Compiled module: RDBIN.
% Compiled module: DEFXYZ.
% Compiled module: SAVEGLO.
% Compiled module: SELECTMODEL.
1997
1998
1999
2000
2001
2002
2003
2004
2005
2006
IDL>

crua6[/cru/cruts/version_3_0/secondaries/cld] ./glo2abs
Welcome! This is the GLO2ABS program.
I will create a set of absolute grids from
a set of anomaly grids (in .glo format), also
a gridded version of the climatology.
Enter the path and name of the normals file: clim.6190.lan.cld
Enter a name for the gridded climatology file: clim.6190.lan.cld.toothbrush
Enter the path and stem of the .glo files: cloudcomboglo/cld.
Enter the starting year: 1996
Enter the ending year: 2006
Enter the path (if any) for the output files: cloudcomboabs/
Now, CONCENTRATE. Addition or Percentage (A/P)? A
Do you wish to limit the output values? (Y/N): Y
1. Set minimum to zero
2. Set single minimum and maximum values
3. Set minima and maxima based on days in month
4. Set integer values >=1, (ie, positive)
5. Changed my mind, no limits
Choose: 2

Enter minimum value: 0

Enter maximum value: 1000
Right, erm.. off I jolly well go!
cloudcomboglo/cld.01.1996.glo
cloudcomboglo/cld.1996.01.glo
cld.1996.01.glo
cld.1996.02.glo

(etc)


cld.2006.11.glo
cld.2006.12.glo
crua6[/cru/cruts/version_3_0/secondaries/cld]

uealogin1[/cru/cruts/version_3_0/secondaries/cld] ./makegrids
Welcome! This is the MAKEGRIDS program.
I will create decadal and full gridded files,
in both ASCII text and NetCDF formats, from
the output files of (eg) glo2abs.for.

Enter a gridfile with YYYY for year and MM for month: cloudcomboabs/cld.YYYY.MM.glo.abs
Enter Start Year: 1996
Enter Start Month: 01
Enter End Year: 2006
Enter End Month: 12

Please enter a sample OUTPUT filename, replacing
start year with SSSS and end year with EEEE, and
ending with '.dat', eg: cru_ts_3_00.SSSS.EEEE.tmp.dat : cru_ts_3_00.SSSS.EEEE.cld.dat

Now please enter the 3-ch parameter code: cld
Enter a generic title for this dataset, eg:
CRU TS 3.00 Mean Temperature : CRU TS 3.00 Percentage Cloud Cover
Writing: cru_ts_3_00.1996.2000.cld.dat
cru_ts_3_00.1996.2000.cld.nc
Writing: cru_ts_3_00.2001.2006.cld.dat
cru_ts_3_00.2001.2006.cld.nc
uealogin1[/cru/cruts/version_3_0/secondaries/cld]

The question is, IS THIS ANY GOOD? Well, we currently have published cloud data
to 2002. So we can make comparisons between 1996 and 2002. Oh, my. I am sure
I've written plenty of comparison routines, but as to their location or name..ah.
There is cmpmgrids.m, which I modified away from its precipitation-only mentality.
I used mmeangrid.for to calculate monthly mean fields (1996-2002) for both 2.10 and
3.00 cloud. The resulting mean files, cru_ts_2_10.1996.2002.cld.dat.mmeans and
cru_ts_3_00.1996.2002.cld.dat.mmeans, were fed into cmpmgrids.m. The results were
less than ideal, though they could have been much worse. Essentially, North America
is totally different - cloudier in Feb/Mar/Apr, sunnier the rest of the year. There
are other differences, particularly in Northern Asia, but these are oatchier and
don't extend throughout the year. So.. the obvious cause would be the inclusion of
DTR-derived cloud, since that would have significant station counts in North America
compared to CLD? Also, there seems to be horizontal banding.. not a good sign given
the nature of the DTR-to-CLD conversion! Naturally, the way to test this is to make
comparisons between FIVE different datasets:

1. CRU TS 2.10 1996-2002 monthly means
2. CRU TS 3.00 1996-2002 monthly means
3. CRU TS 3.00 1996-2002 (synthetic only) monthly means
4. CRU TS 3.00 1996-2002 (observed only) monthly means
5. CLD Climatology

The inclusion of 5 will show the extent of missing data, perhaps.. so I'm suggesting
the following tests:

2-1 Basic comparison of old and new (already done)
3-1 How the DTR-derived synthetic CLD relates to 2.1
4-1 How the sun-hours-derived observed CLD relates to 2.1
3-2 How the DTR-derived synthetic CLD relates to the 'combo' CLD
1-5 How 2.1 relates to the climatology
2-5 How 3.0 relates to the climatology

So, to making datasets 3 and 4:

IDL> quick_interp_tdm2,1996,2002,'cldfromdtrglo05/cld.',600,gs=0.5,dumpglo='dumpglo',pts_prefix='cldfromdtrtxt/cld.'
% Compiled module: QUICK_INTERP_TDM2.
% Compiled module: GLIMIT.
Defaults set
1996
% Compiled module: MAP_SET.
% Compiled module: CROSSP.
% Compiled module: STRIP.
% Compiled module: SAVEGLO.
% Compiled module: SELECTMODEL.
1997
1998
1999
2000
2001
2002
2003
2004
2005
2006
no stations found in: cldfromdtrtxt/cld.2006.09.txt
no stations found in: cldfromdtrtxt/cld.2006.10.txt
no stations found in: cldfromdtrtxt/cld.2006.11.txt
no stations found in: cldfromdtrtxt/cld.2006.12.txt
IDL> quick_interp_tdm2,1996,2002,'cldfromupdate6190glo/cld.',600,gs=0.5,dumpglo='dumpglo',pts_prefix='cldupdate6190/cldupdate6190.'
Defaults set
1996
1997
1998
1999
2000
2001
2002
2003
2004
2005
2006
IDL>

crua6[/cru/cruts/version_3_0/secondaries/cld] ./glo2abs
Welcome! This is the GLO2ABS program.
I will create a set of absolute grids from
a set of anomaly grids (in .glo format), also
a gridded version of the climatology.
Enter the path and name of the normals file: clim.6190.lan.cld
Enter a name for the gridded climatology file: clim.6190.lan.cld.dunconvertin
Enter the path and stem of the .glo files: cldfromdtrglo05/cld.
Enter the starting year: 1996
Enter the ending year: 2002
Enter the path (if any) for the output files: cldfromdtrglo05abs/
Now, CONCENTRATE. Addition or Percentage (A/P)? A
Do you wish to limit the output values? (Y/N): Y
1. Set minimum to zero
2. Set single minimum and maximum values
3. Set minima and maxima based on days in month
4. Set integer values >=1, (ie, positive)
5. Changed my mind, no limits
Choose: 2

Enter minimum value: 0

Enter maximum value: 1000
Right, erm.. off I jolly well go!
cldfromdtrglo05/cld.01.1996.glo
cldfromdtrglo05/cld.1996.01.glo
cld.1996.01.glo
cld.1996.02.glo

(etc)

cld.2002.11.glo
cld.2002.12.glo
crua6[/cru/cruts/version_3_0/secondaries/cld] ./glo2abs
Welcome! This is the GLO2ABS program.
I will create a set of absolute grids from
a set of anomaly grids (in .glo format), also
a gridded version of the climatology.
Enter the path and name of the normals file: clim.6190.lan.cld
Enter a name for the gridded climatology file: clim.6190.lan.cld.notuagain
Enter the path and stem of the .glo files: cldfromupdate6190glo/cld.
Enter the starting year: 1996
Enter the ending year: 2002
Enter the path (if any) for the output files: cldfromupdate6190gloabs/
Now, CONCENTRATE. Addition or Percentage (A/P)? A
Do you wish to limit the output values? (Y/N): Y
1. Set minimum to zero
2. Set single minimum and maximum values
3. Set minima and maxima based on days in month
4. Set integer values >=1, (ie, positive)
5. Changed my mind, no limits
Choose: 2

Enter minimum value: 0

Enter maximum value: 1000
Right, erm.. off I jolly well go!
cldfromupdate6190glo/cld.01.1996.glo
cldfromupdate6190glo/cld.1996.01.glo
cld.1996.01.glo
cld.1996.02.glo

(etc)

cld.2002.11.glo
cld.2002.12.glo
crua6[/cru/cruts/version_3_0/secondaries/cld]

uealogin1[/cru/cruts/version_3_0/secondaries/cld] ./makegrids
Welcome! This is the MAKEGRIDS program.
I will create decadal and full gridded files,
in both ASCII text and NetCDF formats, from
the output files of (eg) glo2abs.for.

Enter a gridfile with YYYY for year and MM for month: cldfromdtrglo05abs/cld.YYYY.MM.glo.abs
Enter Start Year: 1996
Enter Start Month: 01
Enter End Year: 2002
Enter End Month: 12

Please enter a sample OUTPUT filename, replacing
start year with SSSS and end year with EEEE, and
ending with '.dat', eg: cru_ts_3_00.SSSS.EEEE.tmp.dat : cru_ts_3_00.SSSS.EEEE.cld_from_dtr_only.dat

Now please enter the 3-ch parameter code: cld
Enter a generic title for this dataset, eg:
CRU TS 3.00 Mean Temperature : CRU TS 3.00 Percentage Cloud Cover from DTR only
Writing: cru_ts_3_00.1996.2000.cld_from_dtr_only.dat
cru_ts_3_00.1996.2000.cld_from_dtr_only.nc
Writing: cru_ts_3_00.2001.2002.cld_from_dtr_only.dat
cru_ts_3_00.2001.2002.cld_from_dtr_only.nc
uealogin1[/cru/cruts/version_3_0/secondaries/cld] ./makegrids
Welcome! This is the MAKEGRIDS program.
I will create decadal and full gridded files,
in both ASCII text and NetCDF formats, from
the output files of (eg) glo2abs.for.

Enter a gridfile with YYYY for year and MM for month: cldfromupdate6190gloabs/cld.YYYY.MM.glo.abs
Enter Start Year: 1996
Enter Start Month: 01
Enter End Year: 2002
Enter End Month: 12

Please enter a sample OUTPUT filename, replacing
start year with SSSS and end year with EEEE, and
ending with '.dat', eg: cru_ts_3_00.SSSS.EEEE.tmp.dat : cru_ts_3_00.SSSS.EEEE.cld_from_sunobs_only.dat

Now please enter the 3-ch parameter code: cld
Enter a generic title for this dataset, eg:
CRU TS 3.00 Mean Temperature : CRU TS 3.00 Percentage Cloud Cover from SUN obs only
Writing: cru_ts_3_00.1996.2000.cld_from_sunobs_only.dat
cru_ts_3_00.1996.2000.cld_from_sunobs_only.nc
Writing: cru_ts_3_00.2001.2002.cld_from_sunobs_only.dat
cru_ts_3_00.2001.2002.cld_from_sunobs_only.nc
uealogin1[/cru/cruts/version_3_0/secondaries/cld] ./mmeangrid

MMEANGRID - Calculate monthly means for CRU_TS grids

Please enter the gridded data filename: cru_ts_3_00.1996.2002.cld_from_dtr_only.dat

Writing monthly mean grids to: cru_ts_3_00.1996.2002.cld_from_dtr_only.dat.mmeans

uealogin1[/cru/cruts/version_3_0/secondaries/cld] ./mmeangrid

MMEANGRID - Calculate monthly means for CRU_TS grids

Please enter the gridded data filename: cru_ts_3_00.1996.2002.cld_from_sunobs_only.dat

Writing monthly mean grids to: cru_ts_3_00.1996.2002.cld_from_sunobs_only.dat.mmeans

uealogin1[/cru/cruts/version_3_0/secondaries/cld]

Giving us (numbering from before):

1. cru_ts_2_10.1996.2002.cld.dat.mmeans
2. cru_ts_3_00.1996.2002.cld.dat.mmeans
3. cru_ts_3_00.1996.2002.cld_from_dtr_only.dat.mmeans
4. cru_ts_3_00.1996.2002.cld_from_sunobs_only.dat.mmeans
5. clim.6190.lan.cld.grid

And here are our target comparisons again, this time with notes:

2-1 Basic comparison of old and new (already done)

-> Major diffs in N America, all months (lat. striping)

3-1 How the DTR-derived synthetic CLD relates to 2.1

-> major diffs globally, all months (lat. striping)

4-1 How the sun-hours-derived observed CLD relates to 2.1

-> minor patchy diffs globally, all months

3-2 How the DTR-derived synthetic CLD relates to the climatology

-> major diffs globally, all months (lat. striping) c/w 3-1

1-5 How 2.1 relates to the climatology

-> minor patchy diffs globally - equiv for

2-5 How 3.0 relates to the climatology

-> Pretty much as for 2-1

The deduction so far is that the DTR-derived CLD is waaay off. The DTR looks OK, well
OK in the sense that it doesn;t have prominent bands! So it's either the factors and
offsets from the regression, or the way they've been applied in dtr2cld.

Well, dtr2cld is not the world's most complicated program. Wheras cloudreg is, and I
immediately found a mistake! Scanning forward to 1951 was done with a loop that, for
completely unfathomable reasons, didn't include months! So we read 50 grids instead
of 600!!! That may have had something to do with it. I also noticed, as I was correcting
THAT, that I reopened the DTR and CLD data files when I should have been opening the
bloody station files!! I can only assume that I was being interrupted continually when
I was writing this thing. Running with those bits fixed improved matters somewhat,
though now there's a problem in that one 5-degree band (10S to 5S) has no stations! This
will be due to low station counts in that region, plus removal of duplicate values.

Had a think. Phil advised averaging the bands either side to fill the gap, but yuk! And
also the band to the North (ie, 5S to equator) is noticeably lower (extreme, even). So
after some investigation I found that, well, here's the email:

<MAIL QUOTE>
Phil,

I've looked at why we're getting low counts for valid cloud cells in certain 5-degree latitude bands.

The filtering algorithm omits any cell values where the station count is zero, for either CLD or DTR. In general, it's the CLD counts that are zero and losing us the data.

However, in many cases, the cloud value in that cell on that month is not equal to the climatology. And there is plenty of DTR data. So I'm wondering how accurate the station counts are for secondary variables, given that they have to reflect observed and synthetic inputs. Here's a brief example:

(all values are x10)
CLD------------------- DTR-------------------
val stn anom val stn anom
553.00 0.00 -10.00 134.00 20.00 -1.00
558.00 0.00 -17.00 139.00 20.00 2.00
565.00 0.00 -23.00 137.00 20.00 5.00
581.00 0.00 -32.00 139.00 16.00 8.00
587.00 0.00 -38.00 137.00 16.00 9.00
567.00 0.00 -46.00 127.00 15.00 6.00
564.00 0.00 -49.00 120.00 14.00 3.00
552.00 0.00 -48.00 111.00 12.00 0.00
543.00 0.00 -45.00 105.00 12.00 -1.00
535.00 0.00 -40.00 99.00 10.00 -1.00

So, I'm proposing to filter on only the DTR counts, on the assumption that PRE was probably available if DTR was, so synthesis of CLD was likely to have happened, just not shown in the station counts which are probably 'conservative'?
<END MAIL QUOTE>

I didn't get an email back but he did verbally consent. So away we go!

Running with a DTR-station-only screening gives us lots of station values, even with
duplicate filtering turned back on. Niiice. It's still not exactly smooth, but it
might be enough to 'fix' the synthetic cloud.

So, moved all existing directories to <name>_old_badcoeffs, and..

..reproduced the synthetic cloud:

crua6[/cru/cruts/version_3_0/secondaries/cld] ./dtr2cld
Please enter the path/file of the FIRST dtr txt file: /cru/cruts/version_3_0/primaries/dtr/dtrtxt/dtr.1996.01.txt
Please enter the path/file of the LAST dtr txt file: /cru/cruts/version_3_0/primaries/dtr/dtrtxt/dtr.2006.12.txt
crua6[/cru/cruts/version_3_0/secondaries/cld]

Binary 2.5 grid production:

IDL> quick_interp_tdm2,1996,2006,'cldfromdtr25bin/cld.',600,gs=2.5,dumpbin='dumpbin',pts_prefix='cldfromdtrtxt/cld.'
Defaults set
1996
grid 1996 non-zero -28.0336 35.2211 155.5659 cells= 35824
1997
grid 1997 non-zero -28.7817 35.5387 155.2398 cells= 36027
1998
grid 1998 non-zero -31.6090 41.7502 182.8359 cells= 36481
1999
grid 1999 non-zero -26.8934 35.6123 151.3076 cells= 34127
2000
grid 2000 non-zero -28.3765 41.1417 180.7935 cells= 34846
2001
grid 2001 non-zero -31.7763 40.9542 172.0433 cells= 33724
2002
grid 2002 non-zero -29.6498 42.6404 182.7383 cells= 31683
2003
grid 2003 non-zero -27.8828 38.8903 161.9822 cells= 32227
2004
grid 2004 non-zero -30.5593 38.4767 174.0231 cells= 32315
2005
grid 2005 non-zero -33.2088 40.1841 170.8421 cells= 30951
2006
no stations found in: cldfromdtrtxt/cld.2006.09.txt
no stations found in: cldfromdtrtxt/cld.2006.10.txt
no stations found in: cldfromdtrtxt/cld.2006.11.txt
no stations found in: cldfromdtrtxt/cld.2006.12.txt
grid 2006 non-zero -27.2999 36.2585 161.9338 cells= 20383
IDL>

Final gridding with obs as well:

IDL> quick_interp_tdm2,1996,2006,'cloudcomboglo/cld.',600,gs=0.5,dumpglo='dumpglo',synth_prefix='cldfromdtr25bin/cld.',pts_prefix='cldupdate6190/cldupdate6190.'
Defaults set
1996
1997
1998
1999
2000
2001
2002
2003
2004
2005
2006
IDL>

crua6[/cru/cruts/version_3_0/secondaries/cld] ./glo2abs
Welcome! This is the GLO2ABS program.
I will create a set of absolute grids from
a set of anomaly grids (in .glo format), also
a gridded version of the climatology.
Enter the path and name of the normals file: clim.6190.lan.cld
Enter a name for the gridded climatology file: clim.6190.lan.cld.speeeeew
Enter the path and stem of the .glo files: cloudcomboglo/cld.
Enter the starting year: 1996
Enter the ending year: 2006
Enter the path (if any) for the output files: cloudcomboabs/
Now, CONCENTRATE. Addition or Percentage (A/P)? A
Do you wish to limit the output values? (Y/N): Y
1. Set minimum to zero
2. Set single minimum and maximum values
3. Set minima and maxima based on days in month
4. Set integer values >=1, (ie, positive)
5. Changed my mind, no limits
Choose: 2

Enter minimum value: 0

Enter maximum value: 1000
Right, erm.. off I jolly well go!
cloudcomboglo/cld.01.1996.glo
cloudcomboglo/cld.1996.01.glo
cld.1996.01.glo
cld.1996.02.glo
(etc)
cld.2006.11.glo
cld.2006.12.glo
crua6[/cru/cruts/version_3_0/secondaries/cld]

uealogin1[/cru/cruts/version_3_0/secondaries/cld] ./makegrids
Welcome! This is the MAKEGRIDS program.
I will create decadal and full gridded files,
in both ASCII text and NetCDF formats, from
the output files of (eg) glo2abs.for.

Enter a gridfile with YYYY for year and MM for month: cloudcomboabs/cld.YYYY.MM.glo.abs
Enter Start Year: 1996
Enter Start Month: 01
Enter End Year: 2006
Enter End Month: 12

Please enter a sample OUTPUT filename, replacing
start year with SSSS and end year with EEEE, and
ending with '.dat', eg: cru_ts_3_00.SSSS.EEEE.tmp.dat : cru_ts_3_00.SSSS.EEEE.cld.dat

Now please enter the 3-ch parameter code: cld
Enter a generic title for this dataset, eg:
CRU TS 3.00 Mean Temperature : CRU TS 3.00 Percentage Cloud Cover
Writing: cru_ts_3_00.1996.2000.cld.dat
cru_ts_3_00.1996.2000.cld.nc
Writing: cru_ts_3_00.2001.2006.cld.dat
cru_ts_3_00.2001.2006.cld.nc
uealogin1[/cru/cruts/version_3_0/secondaries/cld]

uealogin1[/cru/cruts/version_3_0/secondaries/cld] head -30240 cru_ts_3_00.1996.2006.cld.dat >cru_ts_3_00.1996.2002.cld.dat
uealogin1[/cru/cruts/version_3_0/secondaries/cld] ./mmeangrid

MMEANGRID - Calculate monthly means for CRU_TS grids

Please enter the gridded data filename: cru_ts_3_00.1996.2002.cld.dat

Writing monthly mean grids to: cru_ts_3_00.1996.2002.cld.dat.mmeans

uealogin1[/cru/cruts/version_3_0/secondaries/cld]

Back with cmpmgrids.m.. and things look MUCH better. Differences with the climatology,
or with the 2.10 release, are patchy and generally below 30%. Of course it would be
nice if the differences with the 2.10 release were negligable, since our regression
coefficients were based on 2.10 DTR and CLD.. though of course the sun hours component
is an unknown there, as is the fact that 2.10 used PRE as well as DTR for the synthetics.
Anyway it gets the thumbs-up. The strategy will be to just produce it for 2003-2006.06,
to tie in with the rest of the 3.00 release. So I just need to.. argh. I don't have any
way to create NetCDF files 1901-2006 without the .glo.abs files to work from! I'd have
to specially code a version that swallowed the existing 1901-2002 then added ours. Meh.
Well it's no problem to release a concatenated ASCII file, so I'll do that:

crua6[/cru/cruts/version_3_0/secondaries/cld/cld_final] ls -l
total 2414884
-rw------- 1 f098 cru 1904005440 Feb 13 10:35 cru_ts_2_10.1901-2002.cld.grid
-rw------- 1 f098 cru 7542905 Feb 12 17:50 cru_ts_3_00.1996.2000.cld.dat.gz
-rw------- 1 f098 cru 62217668 Feb 12 17:50 cru_ts_3_00.1996.2000.cld.nc
-rw------- 1 f098 cru 273762720 Feb 12 17:51 cru_ts_3_00.1996.2006.cld.dat
-rw------- 1 f098 cru 136867556 Feb 12 17:51 cru_ts_3_00.1996.2006.cld.nc
-rw------- 1 f098 cru 8893264 Feb 12 17:51 cru_ts_3_00.2001.2006.cld.dat.gz
-rw------- 1 f098 cru 74659316 Feb 12 17:51 cru_ts_3_00.2001.2006.cld.nc
crua6[/cru/cruts/version_3_0/secondaries/cld/cld_final] gunzip cru_ts_3_00.2001.2006.cld.dat.gz
crua6[/cru/cruts/version_3_0/secondaries/cld/cld_final] wc -l cru_ts_3_00.2001.2006.cld.dat
25920 cru_ts_3_00.2001.2006.cld.dat
crua6[/cru/cruts/version_3_0/secondaries/cld/cld_final] tail -17280 cru_ts_3_00.2001.2006.cld.dat >cru_ts_3_00.2003.2006.cld.dat
crua6[/cru/cruts/version_3_0/secondaries/cld/cld_final] wc -l cru_ts_3_00.2003.2006.cld.dat
17280 cru_ts_3_00.2003.2006.cld.dat
crua6[/cru/cruts/version_3_0/secondaries/cld/cld_final] mv cru_ts_2_10.1901-2002.cld.grid cru_ts_3.00.1901.2006.cld.dat
crua6[/cru/cruts/version_3_0/secondaries/cld/cld_final] cat cru_ts_3_00.2003.2006.cld.dat >>cru_ts_3.00.1901.2006.cld.dat
crua6[/cru/cruts/version_3_0/secondaries/cld/cld_final] wc -l cru_ts_3.00.1901.2006.cld.dat
457920 cru_ts_3.00.1901.2006.cld.dat
crua6[/cru/cruts/version_3_0/secondaries/cld/cld_final] cmp ../cru_ts_2_10.1901-2002.cld.grid cru_ts_3.00.1901.2006.cld.dat
cmp: EOF on ../cru_ts_2_10.1901-2002.cld.grid
crua6[/cru/cruts/version_3_0/secondaries/cld/cld_final]

Done and tested.



Go on to part 35s, back to index or Email search