The HARRY_READ_ME.txt file

So to CLOUD. For over a year, rumours have been circulating that money had been found
to pay somebody for a month to recreate Mark New's coefficients. But it never quite
gelled. Now, at last, someone's producing them! Unfortunately.. it's me.

The idea is to derive the coefficients (for the regressing of cloud against DTR) using
the published 2.10 data. We'll use 5-degree blocks and years 1951-2002, then produce
coefficients for each 5-degree latitude band and month. Finally, we'll interpolate to
get half-degree coefficients. Apparently.

Lots of 'issues'. We need to exclude 'background' stations - those that were relaxed to
the climatology. This is hard to detect because the climatology consists of valid values,
so testing for equivalence isn't enough. It might have to be the station files *shudder*.

Using station files was OK, actually. A bigger problem was the inclusion of strings of
consecutive, identical values (for cloud and/or dtr). Not sure what the source is, as they
are not == to the climatology (ie the anoms are not 0). Discussed with Phil - decided to
try excluding any cell with a string like that of >10 values. Cloud only for now. The result
of that was, unfortunately, the loss of several output values, ie:

lat band: 19 month: 7
3.00 -38.00
3.00 -38.00
3.00 -38.00
3.00 -36.00
3.00 -38.00
3.00 -38.00
3.00 -38.00
3.00 -38.00
3.00 -37.00
3.00 -38.00
3.00 -38.00
3.00 -38.00
3.00 -38.00
3.00 -38.00
3.00 -38.00
3.00 -38.00
3.00 -38.00
3.00 -38.00
3.00 -41.00
3.00 -38.00
3.00 -38.00
3.00 -38.00
3.00 -38.00
3.00 -38.00
3.00 -38.00
3.00 -38.00
3.00 -38.00
3.00 -43.00
3.00 -38.00
3.00 -38.00
3.00 -41.00
3.00 -38.00
3.00 -39.00
3.00 -38.00
3.00 -38.00
3.00 -38.00
3.00 -38.00
3.00 -38.00
3.00 -43.00
3.00 -38.00
3.00 -38.00
3.00 -38.00
3.00 -38.00
3.00 -38.00
3.00 -38.00
3.00 -44.00
3.00 -38.00
3.00 -38.00
3.00 -38.00
3.00 -38.00
3.00 -38.00
3.00 -38.00

Results (n= 52): nan nan

As can be seen, neither the dtr (left) nor the cloud (right) look 'sensible', even as
anomalies. Several other months in lat band #19 are either nan or -999 (count=0).

However, if we push the duplicates limit up to, say, 20, we get:

lat band: 19 month: 7
13.50 -50.00
13.50 -50.00
13.50 -50.00
13.50 -50.00
13.50 -50.00
13.50 -50.00
13.50 -50.00
13.50 -50.00
13.50 -53.50
13.50 -50.00
13.50 -50.00
13.50 -50.00
13.50 -50.00
13.50 -50.00
13.50 -50.00
13.50 -50.00
13.50 -50.00
13.50 -50.00
13.50 -50.00
13.50 -50.00
13.50 -50.00
13.50 -50.00
13.50 -50.00
13.50 -50.00
13.50 -50.00
17.00 -65.00
8.50 -42.00
11.50 -49.00
18.00 -71.00

////////////////////

1.00 33.50
1.00 40.00
1.00 32.00
1.00 42.50
1.00 38.00
1.00 38.00
1.00 32.50
1.00 52.50
1.00 44.00
1.00 36.50
1.00 41.00
1.00 30.50
1.00 38.00
1.00 36.00
1.00 38.00
1.00 38.50
1.00 39.00
1.00 31.50
1.00 40.00
1.00 38.00
1.00 31.00
1.00 44.00
1.00 43.00
1.00 37.00
1.00 31.00
1.00 31.00
1.00 30.50

Results (n= 988): 37.59707 -6.05338

So, we can have a proper result, but only by including a load of garbage! In fact,
I might print out this cell as an example. Let's see:

limit nvals factor intercept
10 52 nan nan
20 728 -7.68581 33.30551
none 1716 -8.32450 34.28972

Hmm.. also tried just removing duplicate strings (rather than whole cells):

limit nvals factor intercept
10 1160 -6.99748 26.31960

This 'looks' better - not so steep, and the intercept is a shade closer to 0. The
Matlab script plotcld.m allows comparison of scatter diagrams, these are fed from
example data files manually extracted from the cloudreg.log file after varying the
duplicate limit and/or strategy.

Showed Phil - and now sidetracked into producing global mean series from the 3.0
parameters (DTR first).