01 December 2009
Blog Home : December 2009 : Permalink
As for the code review, I've seen a ton of awful FORTRAN when I was designing death for bux at the missile factory 
There is always a tension between The Scientists, who tend to be pathetic coders, and the programmers who don't understand the theory. And in many cases - ie, probably this one - academic environments don't have professional programmers.
In well-done scientific programming, you have developers write a framework and APIs and let the scientists implement The Magic Algorithm in an environment where they don't have to do things like file I/O, sorting, etc.
This indeed is the problem here. The CRU folks have, for various reasons, not outsourced much if any of the development to actual professional programmers and the result is therefore nasty.Inserted debug statements into anomdtb.f90, discovered that a sum-of-squared variable is becoming very, very negative! [...]
DataA val = 49920, OpTotSq=-1799984256.00
[...]
..so the data value is unbfeasibly large, but why does the sum-of-squares parameter OpTotSq go negative?!!
Probable answer: the high value is pushing beyond the single-precision default for Fortran reals?
Now the code for this is fairly easy to find* and when one looks at it one discovers that the bug has not been fixed. The code still says:integer, pointer, dimension (:,:,:) :: Data,DataA,DataB,DataC
[...]
if (DataA(XAYear,XMonth,XAStn).NE.DataMissVal) then
OpEn=OpEn+1
OpTot=OpTot+DataA(XAYear,XMonth,XAStn)
OpTotSq=OpTotSq+(DataA(XAYear,XMonth,XAStn)**2)
end if
catch,error_value
; avoids a bug in IDL that throws out an occasional
; plot error in virtual window
if error_value ne 0 then begin
error_value=0
i=i+1
goto,skip_poly
endif
[...]The first bug appears to be in IDL itself. Sometimes the polyfill function will throw an error. This error is caught by the catch part and enters the little if there.
Inside the if there's a bug, it's the line i=i+1. This is adding 1 to the loop counter i whenever there's an error. This means that when an error occurs one set of data is not plotted (because the polyfill failed) and then another one is skipped because of the i=i+1.
Given the presence of two bugs in that code (one which was known about and ignored), I wonder how much other crud there is in the code.
To test that I was right about the bug I wrote a simple IDL program in IDL Workbench. Here's a screen shot of the (overly commented!) code and output. It should have output 100, 102, 103 but the bug caused it to skip 102.
Also, and this is a really small thing, the error_value=0 is not necessary because the catch resets the error_value.
In fact the whole error handling code could be reduced to the "goto" line. John ends his post with a question "BTW Does anyone know if these guys use source code management tools?" and I'm 99.99% sure the answer to his question is that they do not use SCM tools.Code this bad is the equivalent of witchcraft. There is essentially no empirical test to distinguish its output from nonsense. Sad to say, I've seen things like this before. Multi-author, non-software engineer-written codebases tend to have these sorts of hair-raising betises liberally sprinkled throughout (although this an extreme example - I wouldn't want to go into that code without a pump-action shotgun and a torch). Ian Harris certainly deserves our sympathy. Trying to hack your way through this utter balderdash must still have him sitting bolt upright in the middle of the night with a look of horror on his face.
The problem all this hits is the difference between "Science" and "Computer Science".Details that could throw doubt on your interpretation must be given, if you know them. You must do the best you can--if you know anything at all wrong, or possibly wrong--to explain it. If you make a theory, for example, and advertise it, or put it out, then you must also put down all the facts that disagree with it, as well as those that agree with it. There is also a more subtle problem. When you have put a lot of ideas together to make an elaborate theory, you want to make sure, when explaining what it fits, that those things it fits are not just the things that gave you the idea for the theory; but that the finished theory makes something else come out right, in addition.
In summary, the idea is to try to give all of the information to help others to judge the value of your contribution; not just the information that leads to judgment in one particular direction or another.
The easiest way to explain this idea is to contrast it, for example, with advertising. Last night I heard that Wesson oil doesn't soak through food. Well, that's true. It's not dishonest; but the thing I'm talking about is not just a matter of not being dishonest, it's a matter of scientific integrity, which is another level. The fact that should be added to that advertising statement is that no oils soak through food, if operated at a certain temperature. If operated at another temperature, they all will-- including Wesson oil. So it's the implication which has been conveyed, not the fact, which is true, and the difference is what we have to deal with.
We've learned from experience that the truth will come out. Other experimenters will repeat your experiment and find out whether you were wrong or right. Nature's phenomena will agree or they'll disagree with your theory. And, although you may gain some temporary fame and excitement, you will not gain a good reputation as a scientist if you haven't tried to be very careful in this kind of work. And it's this type of integrity, this kind of care not to fool yourself, that is missing to a large extent in much of the research in cargo cult science.
For science that involves computer programs it seems blindingly obvious that the code must be included in the supplemental information. Even more, in cases like this where the data sources are mixed, it is vital that the actual raw data be available as well. And the excuses proffered by the CRU and its apologists (e.g. the CRU record largely agrees with GISS or a desire to not confuse the non-scientists with confusing detail) are neatly skewered in subsequent passages of the essay/speech.One of the students told me she wanted to do an experiment that went something like this--it had been found by others that under certain circumstances, X, rats did something, A. She was curious as to whether, if she changed the circumstances to Y, they would still do A. So her proposal was to do the experiment under circumstances Y and see if they still did A.
I explained to her that it was necessary first to repeat in her laboratory the experiment of the other person--to do it under condition X to see if she could also get result A, and then change to Y and see if A changed. Then she would know that the real difference was the thing she thought she had under control.
She was very delighted with this new idea, and went to her professor. And his reply was, no, you cannot do that, because the experiment has already been done and you would be wasting time. This was in about 1947 or so, and it seems to have been the general policy then to not try to repeat psychological experiments, but only to change the conditions and see what happens.
Nowadays there's a certain danger of the same thing happening, even in the famous (?) field of physics. I was shocked to hear of an experiment done at the big accelerator at the National Accelerator Laboratory, where a person used deuterium. In order to compare his heavy hydrogen results to what might happen with light hydrogen" he had to use data from someone else's experiment on light hydrogen, which was done on different apparatus.
This, I think, explains why the climate science "community" has such a hard time with people like Steve McIntyre. They simply are not used to people actually trying to replicate their results instead of taking them on trust.
As a result they have never really thought about data and code archiving policies and all the other techniques that are required if someone is to replicate a complex piece of data analysis. They aren't helped by the rise of the internet and the power of modern computers. These days you can spend about €400 and get a netbook (and a terabyte external harddrive) with more processing power and more data storage than a mainframe 25 years ago. 