{"id":2039,"date":"2009-11-25T14:23:52","date_gmt":"2009-11-25T18:23:52","guid":{"rendered":"http:\/\/www.schollnick.net\/wordpress\/?p=2039"},"modified":"2009-11-26T07:43:11","modified_gmt":"2009-11-26T11:43:11","slug":"hide-the-decline-i-think-hide-the-mistakes","status":"publish","type":"post","link":"http:\/\/www.schollnick.net\/wordpress\/2009\/11\/hide-the-decline-i-think-hide-the-mistakes\/","title":{"rendered":"Hide the Decline?  I think hide the mistakes.."},"content":{"rendered":"<p><span style=\"font-family: 'Lucida Grande'; line-height: normal; font-size: small;\"> <\/span><\/p>\n<div class=\"newsItemTitle\">I went through the CRU code below, and while I try to stay out of politics, this appears to be a programming issue with Politics embedded in it.<\/div>\n<div class=\"newsItemTitle\">I would hazzard a guess that this started as a cover-up from issues arising from bad programming choices, and unvetted data being used for a while&#8230;. \u00c2\u00a0And then when someone noticed &#8220;it was too late to fix&#8221; without embarassing people&#8230; \u00c2\u00a0Probably the wrong people.<\/div>\n<div class=\"newsItemTitle\">But never the less, if you are curious, take a look&#8230;<!--more--><\/div>\n<div class=\"newsItemTitle\"><strong>The original information is from\u00c2\u00a0<a href=\"http:\/\/wattsupwiththat.com\/2009\/11\/25\/climategate-hide-the-decline-codified\/\">Climategate: hide the decline \u00e2\u20ac\u201c codified<\/a><\/strong><\/div>\n<div class=\"newsItemDescription\">\n<div>\n<blockquote><p>WUWT blogging ally\u00c2\u00a0<a href=\"http:\/\/ecotretas.blogspot.com\/\"><strong>Ecotretas<\/strong><\/a> writes in to say that he has made a compendium of programming code segments that show comments by the programmer that suggest places where data may be corrected, modified, adjusted, or busted. Some the\u00c2\u00a0 HARRY_READ_ME comments are quite revealing. For those that don\u00e2\u20ac\u2122t understand computer programming, don\u00e2\u20ac\u2122t fret, the comments by the programmer tell the story quite well even if the code itself makes no sense to you.<\/p>\n<p>To say that the CRU code might be \u00e2\u20ac\u0153buggy\u00e2\u20ac\u009d would be\u00e2\u20ac\u00a6well I\u00e2\u20ac\u2122ll just let CRU\u00e2\u20ac\u2122s programmer tell you in his own words.<\/p><\/blockquote>\n<ul>\n<li><span style=\"text-decoration: underline;\">FOIAdocumentsosborn-tree6mannoldprogmaps12.pro<\/span><span style=\"text-decoration: underline;\">FOIAdocumentsosborn-tree6mannoldprogmaps15.pro<\/span><span style=\"text-decoration: underline;\">FOIAdocumentsosborn-tree6mannoldprogmaps24.pro<\/span><code>; Plots 24 yearly maps of calibrated (PCR-infilled or not) MXD reconstructions<br \/>\n; of growing season temperatures.\u00c2\u00a0<strong>Uses \"corrected\" MXD - but shouldn't usually<br \/>\n; plot past 1960 because these will be artificially adjusted to look closer to<br \/>\n; the real temperatures.<span> <\/span><\/strong><\/code><\/li>\n<li><span style=\"text-decoration: underline;\">FOIAdocumentsharris-treerecon_esper.pro<\/span><code>; Computes regressions on full, high and low pass Esper et al. (2002) series,<br \/>\n; anomalies against full NH temperatures and other series.<br \/>\n; CALIBRATES IT AGAINST THE LAND-ONLY TEMPERATURES NORTH OF 20 N<br \/>\n;<br \/>\n; Specify period over which to compute the regressions (<strong>stop in 1960 to avoid<br \/>\n; the decline<\/strong><\/code><\/li>\n<li><span style=\"text-decoration: underline;\">FOIAdocumentsharris-treecalibrate_nhrecon.pro<\/span><code>;<br \/>\n; Specify period over which to compute the regressions (<strong>stop in 1960 to avoid<\/strong><br \/>\n;\u00c2\u00a0<strong>the decline that affects tree-ring density records<\/strong>)<br \/>\n;<\/code><\/li>\n<li><span style=\"text-decoration: underline;\">FOIAdocumentsharris-treerecon1.pro<\/span><br \/>\n<span style=\"text-decoration: underline;\">FOIAdocumentsharris-treerecon2.pro<\/span><span style=\"text-decoration: underline;\">FOIAdocumentsharris-treerecon_jones.pro<\/span><code>;<br \/>\n; Specify period over which to compute the regressions (<strong>stop in 1940 to avoid<\/strong><br \/>\n;\u00c2\u00a0<strong>the decline<\/strong><br \/>\n;<\/code><\/li>\n<li><a href=\"http:\/\/www.anenglishmanscastle.com\/HARRY_READ_ME.txt\">FOIAdocumentsHARRY_READ_ME.txt<\/a><code>17. Inserted debug statements into anomdtb.f90, discovered that<br \/>\na\u00c2\u00a0<strong>sum-of-squared variable is becoming very, very negative<\/strong>! Key<br \/>\noutput from the debug statements:<br \/>\n(..)<br \/>\nforrtl: error (75): floating point exception<br \/>\nIOT trap (core dumped)<br \/>\n..so the data value is unbfeasibly large,\u00c2\u00a0<strong>but why does the<br \/>\nsum-of-squares parameter OpTotSq go negative?!!<\/strong><\/code><\/li>\n<li><a href=\"http:\/\/www.anenglishmanscastle.com\/HARRY_READ_ME.txt\">FOIAdocumentsHARRY_READ_ME.txt<\/a><code>22. Right, time to stop pussyfooting around the niceties of Tim's labyrinthine software<br \/>\nsuites - let's have a go at producing CRU TS 3.0! since failing to do that will be the<br \/>\ndefinitive failure of the entire project..<\/code><\/li>\n<li><a href=\"http:\/\/www.anenglishmanscastle.com\/HARRY_READ_ME.txt\">FOIAdocumentsHARRY_READ_ME.txt<\/a><code><strong>getting seriously fed up with the state of the Australian data. so many new stations have been<br \/>\nintroduced, so many false references.. so many changes that aren't documented.<\/strong> Every time a<br \/>\ncloud forms I'm presented with a bewildering selection of similar-sounding sites, some with<br \/>\nreferences, some with WMO codes, and some with both. And if I look up the station metadata with<br \/>\none of the local references, chances are the WMO code will be wrong (another station will have<br \/>\nit) and the lat\/lon will be wrong too.<\/code><\/li>\n<li><a href=\"http:\/\/www.anenglishmanscastle.com\/HARRY_READ_ME.txt\">FOIAdocumentsHARRY_READ_ME.txt<\/a><code>I am very sorry to report that the rest of the databases seem to be in nearly as poor a state as<br \/>\nAustralia was.\u00c2\u00a0<strong>There are hundreds if not thousands of pairs of dummy stations<\/strong>, one with no WMO<br \/>\nand one with, usually overlapping and with the same station name and very similar coordinates. I<br \/>\nknow it could be old and new stations, but why such large overlaps if that's the case? Aarrggghhh!<br \/>\nThere truly is no end in sight.<\/code><\/li>\n<li><a href=\"http:\/\/www.anenglishmanscastle.com\/HARRY_READ_ME.txt\">FOIAdocumentsHARRY_READ_ME.txt<\/a><code>28. With huge reluctance, I have dived into 'anomdtb' -\u00c2\u00a0<strong>and already I have<br \/>\nthat familiar Twilight Zone sensation.<\/strong><\/code><\/li>\n<li><a href=\"http:\/\/www.anenglishmanscastle.com\/HARRY_READ_ME.txt\">FOIAdocumentsHARRY_READ_ME.txt<\/a><code>Wrote 'makedtr.for' to tackle the thorny problem of the tmin and tmax databases not<br \/>\nbeing kept in step. Sounds familiar, if worrying.\u00c2\u00a0<strong>am I the first person to attempt<br \/>\nto get the CRU databases in working order?!!<\/strong><\/code><\/li>\n<li><a href=\"http:\/\/www.anenglishmanscastle.com\/HARRY_READ_ME.txt\">FOIAdocumentsHARRY_READ_ME.txt<\/a><code>Well, dtr2cld is not the world's most complicated program. Wheras cloudreg is, and\u00c2\u00a0<strong>I<br \/>\nimmediately found a mistake!<\/strong> Scanning forward to 1951 was done with a loop that,\u00c2\u00a0<strong>for<br \/>\ncompletely unfathomable reasons, didn't include months! So we read 50 grids instead<br \/>\nof 600!!!<\/strong> That may have had something to do with it. I also noticed, as I was correcting<br \/>\nTHAT, that I reopened the DTR and CLD data files when I should have been opening the<br \/>\nbloody station files!!<\/code><\/li>\n<li><a href=\"http:\/\/www.anenglishmanscastle.com\/HARRY_READ_ME.txt\">FOIAdocumentsHARRY_READ_ME.txt<\/a><code>Back to the gridding. I am seriously worried that our flagship gridded data product is produced by<br \/>\nDelaunay triangulation - apparently linear as well. As far as I can see,\u00c2\u00a0<strong>this renders the station<br \/>\ncounts totally meaningless<\/strong>. It also means that we cannot say exactly how the gridded data is arrived<br \/>\nat from a statistical perspective - since we're using an off-the-shelf product that isn't documented<br \/>\nsufficiently to say that. Why this wasn't coded up in Fortran I don't know - time pressures perhaps?<br \/>\nWas too much effort expended on homogenisation, that there wasn't enough time to write a gridding<br \/>\nprocedure?\u00c2\u00a0<strong>Of course, it's too late for me to fix it too.<\/strong> Meh.<\/code><\/li>\n<li><a href=\"http:\/\/www.anenglishmanscastle.com\/HARRY_READ_ME.txt\">FOIAdocumentsHARRY_READ_ME.txt<\/a><code>Here, the expected 1990-2003 period is MISSING - so the correlations aren't so hot! Yet<br \/>\nthe WMO codes and station names \/locations are identical (or close). What the hell is<br \/>\nsupposed to happen here?\u00c2\u00a0<strong>Oh yeah - there is no 'supposed', I can make it up. So I have :-)<\/strong><\/code><\/li>\n<li><a href=\"http:\/\/www.anenglishmanscastle.com\/HARRY_READ_ME.txt\">FOIAdocumentsHARRY_READ_ME.txt<\/a><code>Well,\u00c2\u00a0<strong>it's been a real day of revelations, never mind the week<\/strong>. This morning I<br \/>\ndiscovered that proper angular weighted interpolation was coded into the IDL<br \/>\nroutine, but that its use was discouraged because it was slow! Aaarrrgghh.<br \/>\nThere is even an option to tri-grid at 0.1 degree resolution and then 'rebin'<br \/>\nto 720x360 - also deprecated! And now, just before midnight (so it counts!),<br \/>\nhaving gone back to the tmin\/tmax work,\u00c2\u00a0<strong>I've found that most if not all of the<br \/>\nAustralian bulletin stations have been unceremoniously dumped into the files<br \/>\nwithout the briefest check for existing stations.<\/strong><\/code><\/li>\n<li><a href=\"http:\/\/www.anenglishmanscastle.com\/HARRY_READ_ME.txt\">FOIAdocumentsHARRY_READ_ME.txt<\/a><code><strong>As we can see, even I'm cocking it up!<\/strong> Though recoverably. DTR, TMN and TMX need to be written as (i7.7).\/code&gt;<\/code><\/li>\n<li><a href=\"http:\/\/www.anenglishmanscastle.com\/HARRY_READ_ME.txt\">FOIAdocumentsHARRY_READ_ME.txt<\/a><code><strong>OH FUCK THIS.<\/strong> It's Sunday evening, I've worked all weekend, and just when I thought it was done I'm<br \/>\nhitting yet\u00c2\u00a0<strong>another problem that's based on the hopeless state of our databases.<\/strong> There is no uniform<br \/>\ndata integrity, it's just a catalogue of issues that continues to grow as they're found.<\/code><\/li>\n<li><span style=\"text-decoration: underline;\">FOIAdocumentsosborn-tree6mannmxdgrid2ascii.pro<\/span><code>printf,1,\u00e2\u20ac\u2122Osborn et al. (2004) gridded reconstruction of warm-season\u00e2\u20ac\u2122<br \/>\nprintf,1,\u00e2\u20ac\u2122(April-September) temperature anomalies (from the 1961-1990 mean).\u00e2\u20ac\u2122<br \/>\nprintf,1,\u00e2\u20ac\u2122Reconstruction is based on tree-ring density records.\u00e2\u20ac\u2122<br \/>\nprintf,1<br \/>\nprintf,1,\u00e2\u20ac\u2122NOTE:\u00c2\u00a0<strong>recent decline in tree-ring density has been ARTIFICIALLY<\/strong>\u00e2\u20ac\u2122<br \/>\nprintf,1,\u00e2\u20ac\u2122<strong>REMOVED<\/strong> to facilitate calibration. THEREFORE, post-1960 values\u00e2\u20ac\u2122<br \/>\nprintf,1,\u00e2\u20ac\u2122will be much closer to observed temperatures then they should be,\u00e2\u20ac\u2122<br \/>\nprintf,1,\u00e2\u20ac\u2122which will incorrectly imply the reconstruction is more skilful\u00e2\u20ac\u2122<br \/>\nprintf,1,\u00e2\u20ac\u2122than it actually is. See Osborn et al. (2004).\u00e2\u20ac\u2122<\/code><\/li>\n<li><span style=\"text-decoration: underline;\">FOIAdocumentsosborn-tree6summer_modesdata4sweden.pro<\/span><br \/>\nFOIAdocumentsosborn-tree6summer_modesdata4sweden.pro<code>printf,1,'IMPORTANT NOTE:'<br \/>\nprintf,1,'The data after 1960 should not be used. The tree-ring density'<br \/>\nprintf,1,'records tend to show a decline after 1960 relative to the summer'<br \/>\nprintf,1,'temperature in many high-latitude locations. In this data set'<br \/>\nprintf,1,'this \"decline\" has been artificially removed in an ad-hoc way, and'<br \/>\nprintf,1,'this means that\u00c2\u00a0<strong>data after 1960 no longer represent tree-ring<\/strong><br \/>\nprintf,1,'<strong>density variations, but have been modified to look more like the<\/strong><br \/>\nprintf,1,'<strong>observed temperatures.<\/strong>'<\/code><\/li>\n<li><span style=\"text-decoration: underline;\">FOIAdocumentsosborn-tree6combined_wavelet_col.pro<\/span><code>;<br \/>\n; Remove missing data from start &amp; end (<strong>end in 1960 due to decline<\/strong>)<br \/>\n;<br \/>\nkl=where((yrmxd ge 1402) and (yrmxd le 1960),n)<br \/>\nsst=prednh(kl)<\/code><\/li>\n<li><span style=\"text-decoration: underline;\">FOIAdocumentsosborn-tree6mannmxd_pcr_localtemp.pro<\/span><code>; Tries to reconstruct Apr-Sep temperatures, on a box-by-box basis, from the<br \/>\n; EOFs of the MXD data set. This is PCR, although PCs are used as predictors<br \/>\n; but not as predictands. This PCR-infilling must be done for a number of<br \/>\n; periods, with different EOFs for each period (due to different spatial<br \/>\n; coverage). *BUT* don\u00e2\u20ac\u2122t do special PCR for the modern period (post-1976),<br \/>\n; since they won\u00e2\u20ac\u2122t be used\u00c2\u00a0<strong>due to the decline\/correction problem<\/strong>.<br \/>\n; Certain boxes that appear to reconstruct well are \u00e2\u20ac\u0153manually\u00e2\u20ac\u009d removed because<br \/>\n; they are isolated and away from any trees.<\/code><\/li>\n<li><span style=\"text-decoration: underline;\">FOIAdocumentsosborn-tree6briffa_sep98_d.pro<\/span><code>;mknormal,yyy,timey,refperiod=[1881,1940]<br \/>\n;<br \/>\n;\u00c2\u00a0<strong>Apply a VERY ARTIFICAL correction for decline!!<\/strong><br \/>\n;<br \/>\nyrloc=[1400,findgen(19)*5.+1904]<br \/>\nvaladj=[0.,0.,0.,0.,0.,-0.1,-0.25,-0.3,0.,-0.1,0.3,0.8,1.2,1.7,2.5,2.6,2.6,$<br \/>\n2.6,2.6,2.6]*0.75 ; fudge factor<br \/>\n(...)<br \/>\n;<br \/>\n;\u00c2\u00a0<strong>APPLY ARTIFICIAL CORRECTION<\/strong><br \/>\n;<br \/>\nyearlyadj=interpol(valadj,yrloc,x)<br \/>\ndensall=densall+yearlyadj<\/code><\/li>\n<li><span style=\"text-decoration: underline;\">FOIAdocumentsosborn-tree6summer_modespl_decline.pro<\/span><code>;<br \/>\n; Plots density \u00e2\u20ac\u02dcdecline\u00e2\u20ac\u2122 as a time series of the difference between<br \/>\n; temperature and density averaged over the region north of 50N,<br \/>\n; and an associated pattern in the difference field.<br \/>\n; The difference data set is computed using only boxes and years with<br \/>\n; both temperature and density in them \u00e2\u20ac\u201c i.e., the grid changes in time.<br \/>\n; The pattern is computed by correlating and regressing the *filtered*<br \/>\n; time series against the unfiltered (or filtered) difference data set.<br \/>\n;<br \/>\n;<strong>*** MUST ALTER FUNCT_DECLINE.PRO TO MATCH THE COORDINATES OF THE<\/strong><br \/>\n;<strong> START OF THE DECLINE *** ALTER THIS EVERY TIME YOU CHANGE ANYTHING ***<\/strong><\/code><\/li>\n<li><span style=\"text-decoration: underline;\">FOIAdocumentsosborn-tree6mannoldprogmaps12.pro<\/span><code>;<br \/>\n; Plots 24 yearly maps of calibrated (PCR-infilled or not) MXD reconstructions<br \/>\n; of growing season temperatures.\u00c2\u00a0<strong>Uses \u00e2\u20ac\u0153corrected\u00e2\u20ac\u009d MXD<\/strong> \u00e2\u20ac\u201c but shouldn\u00e2\u20ac\u2122t usually<br \/>\n; plot past 1960\u00c2\u00a0<strong>because these will be artificially adjusted to look closer to<\/strong><br \/>\n;\u00c2\u00a0<strong>the real temperatures.<\/strong><br \/>\n;<\/code><\/li>\n<li><span style=\"text-decoration: underline;\">FOIAdocumentsosborn-tree6mannoldprogcalibrate_correctmxd.pro<\/span><code>; We have previously (calibrate_mxd.pro) calibrated the high-pass filtered<br \/>\n; MXD over 1911-1990, applied the calibration to unfiltered MXD data (which<br \/>\n; gives a zero mean over 1881-1960) after extending the calibration to boxes<br \/>\n; without temperature data (pl_calibmxd1.pro). We have identified and<br \/>\n;\u00c2\u00a0<strong>artificially removed (i.e. corrected) the decline<\/strong> in this calibrated<br \/>\n; data set. We now recalibrate this corrected calibrated dataset against<br \/>\n; the unfiltered 1911-1990 temperature data, and apply the same calibration<br \/>\n; to the corrected and uncorrected calibrated MXD data.<\/code><\/li>\n<li><span style=\"text-decoration: underline;\">FOIAdocumentsosborn-tree6summer_modescalibrate_correctmxd.pro<\/span><code>; No need to verify the correct and uncorrected versions, since these<br \/>\n; should be identical prior to 1920 or 1930 or\u00c2\u00a0<strong>whenever the decline<br \/>\n; was corrected onwards from.<\/strong><\/code><\/li>\n<li><span style=\"text-decoration: underline;\">FOIAdocumentsosborn-tree5densplus188119602netcdf.pro<\/span><code>;\u00c2\u00a0<strong>we know the file starts at yr 440, but we want nothing till 1400,<\/strong> so we<br \/>\n; can skill lines (1400-440)\/10 + 1 header line<br \/>\n; we now want all lines (10 yr per line) from 1400 to 1980, which is<br \/>\n; (1980-1400)\/10 + 1 lines<br \/>\n(...)<br \/>\n;\u00c2\u00a0<strong>we know the file starts at yr 1070, but we want nothing till 1400,<\/strong> so we<br \/>\n; can skill lines (1400-1070)\/10 + 1 header line<br \/>\n; we now want all lines (10 yr per line) from 1400 to 1991, which is<br \/>\n; (1990-1400)\/10 + 1 lines (since 1991 is on line beginning 1990)<\/code><\/li>\n<\/ul>\n<div style=\"overflow-x: hidden; overflow-y: hidden; width: 1px; height: 1px;\">\n<ul>\n<li><span style=\"text-decoration: underline;\">FOIAdocumentsosborn-tree6mannoldprogmaps12.pro<\/span><br \/>\n<span style=\"text-decoration: underline;\">FOIAdocumentsosborn-tree6mannoldprogmaps15.pro<\/span><br \/>\n<span style=\"text-decoration: underline;\">FOIAdocumentsosborn-tree6mannoldprogmaps24.pro<\/span><br \/>\n<code>; Plots 24 yearly maps of calibrated (PCR-infilled or not) MXD reconstructions<br \/>\n; of growing season temperatures.<strong>Uses \"corrected\" MXD - but shouldn't usually<br \/>\n; plot past 1960 because these will be artificially adjusted to look closer to<br \/>\n; the real temperatures.<\/strong><\/code><\/li>\n<li><span style=\"text-decoration: underline;\">FOIAdocumentsharris-treerecon_esper.pro<\/span><br \/>\n<code>; Computes regressions on full, high and low pass Esper et al. (2002) series,<br \/>\n; anomalies against full NH temperatures and other series.<br \/>\n; CALIBRATES IT AGAINST THE LAND-ONLY TEMPERATURES NORTH OF 20 N<br \/>\n;<br \/>\n; Specify period over which to compute the regressions (<strong>stop in 1960 to avoid<br \/>\n; the decline<\/strong><\/code><\/li>\n<li><span style=\"text-decoration: underline;\">FOIAdocumentsharris-treecalibrate_nhrecon.pro<\/span><br \/>\n<code>;<br \/>\n; Specify period over which to compute the regressions (<strong>stop in 1960 to avoid<\/strong><br \/>\n;<strong>the decline that affects tree-ring density records<\/strong>)<br \/>\n;<\/code><\/li>\n<li><span style=\"text-decoration: underline;\">FOIAdocumentsharris-treerecon1.pro<\/span><br \/>\n<span style=\"text-decoration: underline;\">FOIAdocumentsharris-treerecon2.pro<\/span><span style=\"text-decoration: underline;\">FOIAdocumentsharris-treerecon_jones.pro<\/span><br \/>\n<code>;<br \/>\n; Specify period over which to compute the regressions (<strong>stop in 1940 to avoid<\/strong><br \/>\n;<strong>the decline<\/strong><br \/>\n;<\/code><\/li>\n<li><a href=\"http:\/\/www.anenglishmanscastle.com\/HARRY_READ_ME.txt\">FOIAdocumentsHARRY_READ_ME.txt<\/a><br \/>\n<code>17. Inserted debug statements into anomdtb.f90, discovered that<br \/>\na<strong>sum-of-squared variable is becoming very, very negative<\/strong>! Key<br \/>\noutput from the debug statements:<br \/>\n(..)<br \/>\nforrtl: error (75): floating point exception<br \/>\nIOT trap (core dumped)<br \/>\n..so the data value is unbfeasibly large,<strong>but why does the<br \/>\nsum-of-squares parameter OpTotSq go negative?!!<\/strong><\/code><\/li>\n<li><a href=\"http:\/\/www.anenglishmanscastle.com\/HARRY_READ_ME.txt\">FOIAdocumentsHARRY_READ_ME.txt<\/a><br \/>\n<code>22. Right, time to stop pussyfooting around the niceties of Tim's labyrinthine software<br \/>\nsuites - let's have a go at producing CRU TS 3.0! since failing to do that will be the<br \/>\ndefinitive failure of the entire project..<\/code><\/li>\n<li><a href=\"http:\/\/www.anenglishmanscastle.com\/HARRY_READ_ME.txt\">FOIAdocumentsHARRY_READ_ME.txt<\/a><br \/>\n<code><strong>getting seriously fed up with the state of the Australian data. so many new stations have been<br \/>\nintroduced, so many false references.. so many changes that aren't documented.<\/strong>Every time a<br \/>\ncloud forms I'm presented with a bewildering selection of similar-sounding sites, some with<br \/>\nreferences, some with WMO codes, and some with both. And if I look up the station metadata with<br \/>\none of the local references, chances are the WMO code will be wrong (another station will have<br \/>\nit) and the lat\/lon will be wrong too.<\/code><\/li>\n<li><a href=\"http:\/\/www.anenglishmanscastle.com\/HARRY_READ_ME.txt\">FOIAdocumentsHARRY_READ_ME.txt<\/a><br \/>\n<code>I am very sorry to report that the rest of the databases seem to be in nearly as poor a state as<br \/>\nAustralia was.<strong>There are hundreds if not thousands of pairs of dummy stations<\/strong>, one with no WMO<br \/>\nand one with, usually overlapping and with the same station name and very similar coordinates. I<br \/>\nknow it could be old and new stations, but why such large overlaps if that's the case? Aarrggghhh!<br \/>\nThere truly is no end in sight.<\/code><\/li>\n<li><a href=\"http:\/\/www.anenglishmanscastle.com\/HARRY_READ_ME.txt\">FOIAdocumentsHARRY_READ_ME.txt<\/a><br \/>\n<code>28. With huge reluctance, I have dived into 'anomdtb' -<strong>and already I have<br \/>\nthat familiar Twilight Zone sensation.<\/strong><\/code><\/li>\n<li><a href=\"http:\/\/www.anenglishmanscastle.com\/HARRY_READ_ME.txt\">FOIAdocumentsHARRY_READ_ME.txt<\/a><br \/>\n<code>Wrote 'makedtr.for' to tackle the thorny problem of the tmin and tmax databases not<br \/>\nbeing kept in step. Sounds familiar, if worrying.<strong>am I the first person to attempt<br \/>\nto get the CRU databases in working order?!!<\/strong><\/code><\/li>\n<li><a href=\"http:\/\/www.anenglishmanscastle.com\/HARRY_READ_ME.txt\">FOIAdocumentsHARRY_READ_ME.txt<\/a><br \/>\n<code>Well, dtr2cld is not the world's most complicated program. Wheras cloudreg is, and<strong>I<br \/>\nimmediately found a mistake!<\/strong>Scanning forward to 1951 was done with a loop that,<strong>for<br \/>\ncompletely unfathomable reasons, didn't include months! So we read 50 grids instead<br \/>\nof 600!!!<\/strong>That may have had something to do with it. I also noticed, as I was correcting<br \/>\nTHAT, that I reopened the DTR and CLD data files when I should have been opening the<br \/>\nbloody station files!!<\/code><\/li>\n<li><a href=\"http:\/\/www.anenglishmanscastle.com\/HARRY_READ_ME.txt\">FOIAdocumentsHARRY_READ_ME.txt<\/a><br \/>\n<code>Back to the gridding. I am seriously worried that our flagship gridded data product is produced by<br \/>\nDelaunay triangulation - apparently linear as well. As far as I can see,<strong>this renders the station<br \/>\ncounts totally meaningless<\/strong>. It also means that we cannot say exactly how the gridded data is arrived<br \/>\nat from a statistical perspective - since we're using an off-the-shelf product that isn't documented<br \/>\nsufficiently to say that. Why this wasn't coded up in Fortran I don't know - time pressures perhaps?<br \/>\nWas too much effort expended on homogenisation, that there wasn't enough time to write a gridding<br \/>\nprocedure?<strong>Of course, it's too late for me to fix it too.<\/strong>Meh.<\/code><\/li>\n<li><a href=\"http:\/\/www.anenglishmanscastle.com\/HARRY_READ_ME.txt\">FOIAdocumentsHARRY_READ_ME.txt<\/a><br \/>\n<code>Here, the expected 1990-2003 period is MISSING - so the correlations aren't so hot! Yet<br \/>\nthe WMO codes and station names \/locations are identical (or close). What the hell is<br \/>\nsupposed to happen here?<strong>Oh yeah - there is no 'supposed', I can make it up. So I have :-)<\/strong><\/code><\/li>\n<li><a href=\"http:\/\/www.anenglishmanscastle.com\/HARRY_READ_ME.txt\">FOIAdocumentsHARRY_READ_ME.txt<\/a><br \/>\n<code>Well,<strong>it's been a real day of revelations, never mind the week<\/strong>. This morning I<br \/>\ndiscovered that proper angular weighted interpolation was coded into the IDL<br \/>\nroutine, but that its use was discouraged because it was slow! Aaarrrgghh.<br \/>\nThere is even an option to tri-grid at 0.1 degree resolution and then 'rebin'<br \/>\nto 720x360 - also deprecated! And now, just before midnight (so it counts!),<br \/>\nhaving gone back to the tmin\/tmax work,<strong>I've found that most if not all of the<br \/>\nAustralian bulletin stations have been unceremoniously dumped into the files<br \/>\nwithout the briefest check for existing stations.<\/strong><\/code><\/li>\n<li><a href=\"http:\/\/www.anenglishmanscastle.com\/HARRY_READ_ME.txt\">FOIAdocumentsHARRY_READ_ME.txt<\/a><br \/>\n<code><strong>As we can see, even I'm cocking it up!<\/strong>Though recoverably. DTR, TMN and TMX need to be written as (i7.7).\/code&gt;<\/code><\/li>\n<li><a href=\"http:\/\/www.anenglishmanscastle.com\/HARRY_READ_ME.txt\">FOIAdocumentsHARRY_READ_ME.txt<\/a><br \/>\n<code><strong>OH FUCK THIS.<\/strong>It's Sunday evening, I've worked all weekend, and just when I thought it was done I'm<br \/>\nhitting yet<strong>another problem that's based on the hopeless state of our databases.<\/strong>There is no uniform<br \/>\ndata integrity, it's just a catalogue of issues that continues to grow as they're found.<\/code><\/li>\n<li><span style=\"text-decoration: underline;\">FOIAdocumentsosborn-tree6mannmxdgrid2ascii.pro<\/span><br \/>\n<code>printf,1,\u00e2\u20ac\u2122Osborn et al. (2004) gridded reconstruction of warm-season\u00e2\u20ac\u2122<br \/>\nprintf,1,\u00e2\u20ac\u2122(April-September) temperature anomalies (from the 1961-1990 mean).\u00e2\u20ac\u2122<br \/>\nprintf,1,\u00e2\u20ac\u2122Reconstruction is based on tree-ring density records.\u00e2\u20ac\u2122<br \/>\nprintf,1<br \/>\nprintf,1,\u00e2\u20ac\u2122NOTE:<strong>recent decline in tree-ring density has been ARTIFICIALLY<\/strong>\u00e2\u20ac\u2122<br \/>\nprintf,1,\u00e2\u20ac\u2122<strong>REMOVED<\/strong>to facilitate calibration. THEREFORE, post-1960 values\u00e2\u20ac\u2122<br \/>\nprintf,1,\u00e2\u20ac\u2122will be much closer to observed temperatures then they should be,\u00e2\u20ac\u2122<br \/>\nprintf,1,\u00e2\u20ac\u2122which will incorrectly imply the reconstruction is more skilful\u00e2\u20ac\u2122<br \/>\nprintf,1,\u00e2\u20ac\u2122than it actually is. See Osborn et al. (2004).\u00e2\u20ac\u2122<\/code><\/li>\n<li><span style=\"text-decoration: underline;\">FOIAdocumentsosborn-tree6summer_modesdata4sweden.pro<\/span><br \/>\nFOIAdocumentsosborn-tree6summer_modesdata4sweden.pro<br \/>\n<code>printf,1,'IMPORTANT NOTE:'<br \/>\nprintf,1,'The data after 1960 should not be used. The tree-ring density'<br \/>\nprintf,1,'records tend to show a decline after 1960 relative to the summer'<br \/>\nprintf,1,'temperature in many high-latitude locations. In this data set'<br \/>\nprintf,1,'this \"decline\" has been artificially removed in an ad-hoc way, and'<br \/>\nprintf,1,'this means that<strong>data after 1960 no longer represent tree-ring<\/strong><br \/>\nprintf,1,'<strong>density variations, but have been modified to look more like the<\/strong><br \/>\nprintf,1,'<strong>observed temperatures.<\/strong>'<\/code><\/li>\n<li><span style=\"text-decoration: underline;\">FOIAdocumentsosborn-tree6combined_wavelet_col.pro<\/span><br \/>\n<code>;<br \/>\n; Remove missing data from start &amp; end (<strong>end in 1960 due to decline<\/strong>)<br \/>\n;<br \/>\nkl=where((yrmxd ge 1402) and (yrmxd le 1960),n)<br \/>\nsst=prednh(kl)<\/code><\/li>\n<li><span style=\"text-decoration: underline;\">FOIAdocumentsosborn-tree6mannmxd_pcr_localtemp.pro<\/span><br \/>\n<code>; Tries to reconstruct Apr-Sep temperatures, on a box-by-box basis, from the<br \/>\n; EOFs of the MXD data set. This is PCR, although PCs are used as predictors<br \/>\n; but not as predictands. This PCR-infilling must be done for a number of<br \/>\n; periods, with different EOFs for each period (due to different spatial<br \/>\n; coverage). *BUT* don\u00e2\u20ac\u2122t do special PCR for the modern period (post-1976),<br \/>\n; since they won\u00e2\u20ac\u2122t be used<strong>due to the decline\/correction problem<\/strong>.<br \/>\n; Certain boxes that appear to reconstruct well are \u00e2\u20ac\u0153manually\u00e2\u20ac\u009d removed because<br \/>\n; they are isolated and away from any trees.<\/code><\/li>\n<li><span style=\"text-decoration: underline;\">FOIAdocumentsosborn-tree6briffa_sep98_d.pro<\/span><code>;mknormal,yyy,timey,refperiod=[1881,1940]<br \/>\n;<br \/>\n;<strong>Apply a VERY ARTIFICAL correction for decline!!<\/strong><br \/>\n;<br \/>\nyrloc=[1400,findgen(19)*5.+1904]<br \/>\nvaladj=[0.,0.,0.,0.,0.,-0.1,-0.25,-0.3,0.,-0.1,0.3,0.8,1.2,1.7,2.5,2.6,2.6,$<br \/>\n2.6,2.6,2.6]*0.75 ; fudge factor<br \/>\n(...)<br \/>\n;<br \/>\n;<strong>APPLY ARTIFICIAL CORRECTION<\/strong><br \/>\n;<br \/>\nyearlyadj=interpol(valadj,yrloc,x)<br \/>\ndensall=densall+yearlyadj<\/code><\/li>\n<li><span style=\"text-decoration: underline;\">FOIAdocumentsosborn-tree6summer_modespl_decline.pro<\/span><br \/>\n<code>;<br \/>\n; Plots density \u00e2\u20ac\u02dcdecline\u00e2\u20ac\u2122 as a time series of the difference between<br \/>\n; temperature and density averaged over the region north of 50N,<br \/>\n; and an associated pattern in the difference field.<br \/>\n; The difference data set is computed using only boxes and years with<br \/>\n; both temperature and density in them \u00e2\u20ac\u201c i.e., the grid changes in time.<br \/>\n; The pattern is computed by correlating and regressing the *filtered*<br \/>\n; time series against the unfiltered (or filtered) difference data set.<br \/>\n;<br \/>\n;<strong>*** MUST ALTER FUNCT_DECLINE.PRO TO MATCH THE COORDINATES OF THE<\/strong><br \/>\n;<strong>START OF THE DECLINE *** ALTER THIS EVERY TIME YOU CHANGE ANYTHING ***<\/strong><\/code><\/li>\n<li><span style=\"text-decoration: underline;\">FOIAdocumentsosborn-tree6mannoldprogmaps12.pro<\/span><br \/>\n<code>;<br \/>\n; Plots 24 yearly maps of calibrated (PCR-infilled or not) MXD reconstructions<br \/>\n; of growing season temperatures.<strong>Uses \u00e2\u20ac\u0153corrected\u00e2\u20ac\u009d MXD<\/strong>\u00e2\u20ac\u201c but shouldn\u00e2\u20ac\u2122t usually<br \/>\n; plot past 1960<strong>because these will be artificially adjusted to look closer to<\/strong><br \/>\n;<strong>the real temperatures.<\/strong><br \/>\n;<\/code><\/li>\n<li><span style=\"text-decoration: underline;\">FOIAdocumentsosborn-tree6mannoldprogcalibrate_correctmxd.pro<\/span><br \/>\n<code>; We have previously (calibrate_mxd.pro) calibrated the high-pass filtered<br \/>\n; MXD over 1911-1990, applied the calibration to unfiltered MXD data (which<br \/>\n; gives a zero mean over 1881-1960) after extending the calibration to boxes<br \/>\n; without temperature data (pl_calibmxd1.pro). We have identified and<br \/>\n;<strong>artificially removed (i.e. corrected) the decline<\/strong>in this calibrated<br \/>\n; data set. We now recalibrate this corrected calibrated dataset against<br \/>\n; the unfiltered 1911-1990 temperature data, and apply the same calibration<br \/>\n; to the corrected and uncorrected calibrated MXD data.<\/code><\/li>\n<li><span style=\"text-decoration: underline;\">FOIAdocumentsosborn-tree6summer_modescalibrate_correctmxd.pro<\/span><br \/>\n<code>; No need to verify the correct and uncorrected versions, since these<br \/>\n; should be identical prior to 1920 or 1930 or<strong>whenever the decline<br \/>\n; was corrected onwards from.<\/strong><\/code><\/li>\n<li><span style=\"text-decoration: underline;\">FOIAdocumentsosborn-tree5densplus188119602netcdf.pro<\/span><br \/>\n<code>;<strong>we know the file starts at yr 440, but we want nothing till 1400,<\/strong>so we<br \/>\n; can skill lines (1400-440)\/10 + 1 header line<br \/>\n; we now want all lines (10 yr per line) from 1400 to 1980, which is<br \/>\n; (1980-1400)\/10 + 1 lines<br \/>\n(...)<br \/>\n;<strong>we know the file starts at yr 1070, but we want nothing till 1400,<\/strong>so we<br \/>\n; can skill lines (1400-1070)\/10 + 1 header line<br \/>\n; we now want all lines (10 yr per line) from 1400 to 1991, which is<br \/>\n; (1990-1400)\/10 + 1 lines (since 1991 is on line beginning 19<\/code><\/li>\n<\/ul>\n<\/div>\n<\/div>\n<\/div>\n","protected":false},"excerpt":{"rendered":"<p>I went through the CRU code below, and while I try to stay out of politics, this appears to be a programming issue with Politics embedded in it. I would hazzard a guess that this started as a cover-up from issues arising from bad programming choices, and unvetted data being used for a while&#8230;. \u00c2\u00a0And <a class=\"read-more\" href=\"http:\/\/www.schollnick.net\/wordpress\/2009\/11\/hide-the-decline-i-think-hide-the-mistakes\/\">[&hellip;]<\/a><\/p>\n","protected":false},"author":1,"featured_media":0,"comment_status":"open","ping_status":"open","sticky":false,"template":"","format":"standard","meta":{"_monsterinsights_skip_tracking":false,"_monsterinsights_sitenote_active":false,"_monsterinsights_sitenote_note":"","_monsterinsights_sitenote_category":0},"categories":[54,47,19,48],"tags":[91],"_links":{"self":[{"href":"http:\/\/www.schollnick.net\/wordpress\/wp-json\/wp\/v2\/posts\/2039"}],"collection":[{"href":"http:\/\/www.schollnick.net\/wordpress\/wp-json\/wp\/v2\/posts"}],"about":[{"href":"http:\/\/www.schollnick.net\/wordpress\/wp-json\/wp\/v2\/types\/post"}],"author":[{"embeddable":true,"href":"http:\/\/www.schollnick.net\/wordpress\/wp-json\/wp\/v2\/users\/1"}],"replies":[{"embeddable":true,"href":"http:\/\/www.schollnick.net\/wordpress\/wp-json\/wp\/v2\/comments?post=2039"}],"version-history":[{"count":0,"href":"http:\/\/www.schollnick.net\/wordpress\/wp-json\/wp\/v2\/posts\/2039\/revisions"}],"wp:attachment":[{"href":"http:\/\/www.schollnick.net\/wordpress\/wp-json\/wp\/v2\/media?parent=2039"}],"wp:term":[{"taxonomy":"category","embeddable":true,"href":"http:\/\/www.schollnick.net\/wordpress\/wp-json\/wp\/v2\/categories?post=2039"},{"taxonomy":"post_tag","embeddable":true,"href":"http:\/\/www.schollnick.net\/wordpress\/wp-json\/wp\/v2\/tags?post=2039"}],"curies":[{"name":"wp","href":"https:\/\/api.w.org\/{rel}","templated":true}]}}<!-- WP Super Cache is installed but broken. The constant WPCACHEHOME must be set in the file wp-config.php and point at the WP Super Cache plugin directory. -->