So far almost all of the attention has been focused on the email contents. Graphically, the file structure looks like this:
To date, almost all scrutiny has been placed on the smaller of the two main sub-directories, "mail." It's shown above in the upper right hand corner shaded in yellow. Graphically one can intuitively see that the "documents" section is much larger but so far largely ignored. However, in an attempt to keep some focus - we're going to look mostly into the "email" file.
First, lets look at the file format. There are 1073 individual "text" files. These have been widely described as "email files." They are not. Perhaps some will consider this a hyper-technical distinction, but it is critical when trying to determine the source of this information. An "email" resides on a "mail server" and has some very specific characteristics. The most common form of email servers do not create individual files for emails (although some do). None of these individual files found on the "hacked" "email" directory contain the "envelope" and "header" characteristics of an email file. Consider the following:
here is a sample of a "portion" of an "mbox" file found on a typical mail server:
As we can see in this sample of an "e
mail" file, this file has a Return Path, Delivered To, and a series of SMTP server addresses from which the email traveled to and from on it's route to it's final destination
Some other additional "header" information is included to describe the Mime version of the originating MUA (mail user agent such as Outlook, Eudora, or Thunderbird).
And finally the POP server ID that eventually served the email to the recipients MUA for reading.
These items are present in every email. Most MUA's by default strip out such information when a user "views" an email, but it is always there. If these "emails" were "hacked" from a mail server or even a client "mail box" this data would be in every email file. It is not.
All of the "emails" in the "hacked" email directory are "plain text files" and lack any envelope or header information. This does not mean that they do not represent the "bodies" of valid emails or are in some way fraudulent. It's quite likely that they do represent valid email content
By contrast, lets look at an example of one of the actual files from the "email" sub-directory:
This "email" file pictured to the right
is shown in it's entirety exactly as
rendered when opened with a
common text editor. (With the
exception of the yellow highlight
which was added to show one part-
ularly interesting sentence.)
It contains no "header" or "envelope"
information that would be in an
"email" file. It contains only the
plain text of the Body, To, From, and
subject lines of an email file.
This is of interest only to illustrate
that we can say with a high degree
of confidence that
these files did not
come from an any email server -
hacked or otherwise.
This poses the obvious question, "What then, if not emails - are these files?"
These files, in all likelihood are "archive files" generated by an "email archiving" software program. What such programs do, is extract the "body" of an email, create an individual file, generate a unique file id, and then index the email in a way that any words in the email can be searched. These archiving systems such as "MailArchiva" are widely used by large organizations - primarily so that emails can be maintained in a "compliant" format that meets the requirements of "Freedom of Information" laws. It appears a virtual certainty that the files found in the "hacked" "email" directory are archive files, generated by an email archiving system.
Why is this important? We'll get back to that momentarily.
For now, lets just stipulate that the "emails" are actually "email archive" files, generated by archiving software connected to the mail system.
The second troubling thing. The number of files. There are 1073. This is a tiny spec of email content over a period that spans more than 13 years. The first email is
file:/FOIA/mail/0826209667.txt dated March 7, 1996. The last email is f
ile://FOIA/FOIA/mail/1258053464.txt dated November 12, 2009. Also the numerical span of the file numbers indicates difference of 421,843,797 possible unique file number id's. Now it is possible that unique numbers are being assigned at other than "single digit" sequence, but given that there are almost 423 million unused unique sequential id's in this span - it is certain that the number of emails in the "archive" at least approach 100 million.
We've got 1073. That mean we have no more than 1 email per 100,000 that exist on this archive - and probably 1 per 400,000.
After reading through several dozen of these "email" files, no single common characteristic could be found. That is, no common sender, recipient, subject, date range, common key word, etc. This can lead us to only one logical conclusion:
The files in the "email" sub-directory are the result of a person or persons conducting several searches of a master mail archive.
Additionally what we can logically conclude based on an assumption of a "search range" that spans 13 years and at least 100 million email archive files, is that many of the email archive files have been culled from the search results. Consider this, a key word search of the "email files" posted at "climate-gate.org" shows that there are 70 files containing the word "trick." By comparison a search of my personal email files spanning about 5 years yield 59 emails containing the word "trick." The word "Fortran" only appears once. Consider that. The "documents" section of this "hack" contains dozens of Fortran programs written or edited by the main actors found in the "email" files - yet they only say "fortran" in 1 communication. The point is that whoever conducted the search of the main mail archive to yield these 1073 files almost certainly culled and discarded many of the results of key word searches which would lead to a much larger number of files. Further refining the above conclusion:
These "files" consist of files specifically culled from the results of several searches of a master mail archive.
That is, this file appears much more like a "presentation" than a "hack". Understand the enormity of this difference. We are being lead to believe that these files have been made public through an act of digital espionage which exposed highly protected communications. What we are in fact seeing is a compilation of a tiny amount of email archive files that were painstakingly prepared through a series of keyword searches and subsequent filtering or culling of the results. This compilation exists expressly for the purpose of being "presented" to some audience to convey some particular message or answer some particular question. It seems to be an effort to portray what are highly selected emails as a "random sample." (Not unlike representing highly selected climate data and contending it is the result of random sampling.)
In a metaphoric sense, we are being asked to believe we are seeing, "the man behind the curtain" when in fact we are looking at a carefully prepared presentation, that is "yet another curtain."
When one considers these "email" files to be a "presentation" rather than an hostile exposure, one must then ask what is the intended purpose and intended target of the "presentation."
-One theory is that the files were being prepared in response to FOIA requests and pre-maturely "leaked" by a disgruntled insider.
-A second theory is that a disgruntled insider compiled these emails on his own and leaked them to the public .ftp server.
-A third theory is that the "hack" was much more complete, and the "hackers" themselves searched and presented the most relevant email files, while excluding the balance.
Any of the above is possible. But lets consider each.
In the first theory, it would seem that any party receiving the these 1073 email files as a response to FOIA request that spanned 13 years of communication documents - would consider this file to be "non-responsive" or at least incomplete. It is difficult to believe that this tiny file would be considered a complete response to any FOIA request. Recall also, that in compiling items such as this, the progression of the work product is from large to small. That is, one does not start at zero and start reviewing the 100 million emails and build the answer one file at a time. The work starts with 100 million and then the results of searches are further refined and the results culled to yield a smaller and smaller file. Therefore, it must be assumed that files contained are very close to a completed process - hence the theory that these were leaked "prematurely" is weak.
In the second theory of the "disgruntled insider" acting alone (not in response to FOIA), it would seem odd that the email files would be culled so precisely. It's hard to imagine a "disgruntled" employee, carrying a grudge and going "digitally postal" on his superiors - but stopping to remove any and all potentially embarrassing personal communications. Remember, ALL of these files are pure business. No spam, no jokes, no wife, girlfriend, boyfriend, sports, etc - all of those "personal" emails kindly redacted by the supposed "grudge carrying disgruntled insider."
The third theory of the "thorough hack" shares much of the same difficulties as the second. When a "cracker" cracks and copies huge amounts of data, the tendency is to display as much as possible. If however this is true, then we can expect to see more documents and emails released in the coming days and weeks - because the current file is clearly incomplete. Consider, in one particular file, Phil Jones advises several parties to destroy communication evidence. Clearly, someone responded to that request either by agreeing or confirming, or by refusing to participate in a cover up. Yet, there is no specific response contained in the 1073 files. Now, Micheal Mann has publicly acknowledged receiving such a request - but adamantly denies he ever destroyed or deleted anything. I suspect that soon the others copied on that correspondence will soon make similar statements.
If the third theory of the "thorough hack" is true, we can expect to see specific responses to the Phil Jones request to delete emails very soon - if not, it's unlikely that there is anymore relevant data other than that which has already been released.
Considering the weak points of the above theories, let's consider another . . .
Theory 4: The files were compiled by Mann, Jones, et al and leaked into the public with the intent of making them appear to be the result of a hostile "hack."
Why?
The emails appear to be damaging when read by those of us that are already suspicious of "Global Warming" science. Remember, deception is much easier when the deceived "want to believe." These email files lead us down several paths that indicate the intent to defraud and destroy evidence - but seem to always stop just short of saying exactly that. Is it not possible that each of these emails was released because each has a readily available defense?
Take for example the now famous "trick" for "hiding the decline" quote. Phil Jones to buds : Mann, Bradly, and Hughes:
"I’ve just completed Mike’s Nature
trick of adding in the real temps to each series for the last 20 years
(ie from 1981 onwards) amd (sic) from 1961 for Keith’s to hide the
decline."
In defense of this phrase, Jones has described "trick" as something "clever" as in a trade secret or "a better way to do something." In fact, a search of "trick" in the email files yields 70 files - almost all of which use the word in the manner of Jones describes in his defense. As for the "hide the decline" phrase, the defense points to a published article in RealClimate titled "
Progress in reconstructing climate in recent millennia"
. This article discusses the methodology of how to handle the "so called divergence problem" which is what Jones contends he is referring to in his email. He is essentially saying that the "decline" in this case is the "divergence problem" widely discussed in published data - and not a temperature decline.
Guess what. Its a valid argument.
The balance of the now famed email reads:
"Mike's series got the annual land and marine values while the other two got April-Sept for NH land N of 20N. The latter two are real for 1999, while the estimate for 1999 for NH combined is +0.44C wrt 61-90. The Global estimate for 1999 with
data through Oct is +0.35C cf. 0.57 for 1998.
Thanks for the comments, Ray."
Which makes this email appear to be roughly a reply to an email from Tim Osborn
(#
939154709.txt) stating:
" Indeed, if the non-temperature signal that causes the decline in tree-ring density begins before 1960, then a short 1931-60 period might yield a more biased result than using a longer 1881-1960 period."
He then ends the email with a brief description of his attached data.
"Calibrated against observed Apr-Sep temperature over 1881-1960 averaged over all land grid boxes north of 20N
Year Reconstructed temperature anomaly (degrees C wrt 1961-90)"
The description of data matches that referred to in Jones first "hide the decline" email, and the "decline" to which he is referring is clearly proven to be that referenced in Osborn's earlier email about declining "tree ring density" not "declining temperature." All of which has been published and discussed extensively for 10 years and as recently as the previously noted RealClimate article published in September of 2008. Just for good measure, what appears to be one of the founding documents of this "reconstruction" theory just happens to be in the "documents" section of the "hacked" file titled "Mann uncertainty.doc", and a graphic titled "DeclineSeries.pdf" which clearly depicts tree ring density and width "decline".
See how this works?
First stage a "hack" of supposedly sensitive material. Make sure the "hack" contains some great eye popping phrases like "Hide the Decline." Make sure the "hack" also contains a benign explanation for the offending phrases. Make sure the "hack" gets out and feign outrage and embarrassment. Allow the "other side" time to gloat over their new found victory, and increasingly hail the "hacked data" as a definitive and final proof of wrong doing by the CRU.
Then, begin the "counter offensive," Point out articles, then supporting emails, then finally supporting documents and computer code that exist in the very "hacked" files that are being hailed as proof of wrong doing!
-Far fetched ? -Yes.
.
-Impossible? -No.
Understand, Jones, Mann, et al are very smart people. The small sample of work product that's been included in this "hacked" file clearly attest to that. They are all competent programmers, analysts, and writers. It is a mistake to underestimate their intelligence or resourcefulness. They have already succeeded in convincing huge numbers of people that the by product of human exhalation is a poison. They are on the brink of re-ordering the power and wealth structure of the world based on their work.
Consider:
Limbaugh has already produced a parody song, Hide the Decline . Its been made into a you tube video (at the link). Many others are on board. Hannity says "hide the decline" almost as much as he says "I'm not a Republican."
Anyone who's read this piece,to this point, and then plays the you tube link can see how the "skeptics" who highlighted this phrase, "hide the decline" are ill informed. Quite honestly. When one knows the background of the phrase - the song makes Limbaugh look stupid. In this scenario - if true - that is the goal. Get skeptics all on the same page, citing all the same reference points in a supposedly "hacked" and previously secret computer files - then "pull the rug out" by citing all of the exculpatory evidence that exists in the very files on which skeptics are basing their accusations. All the while maintaining "full snob" mode by gently pointing out how "non-scientific" people could make these juvenile errors.
In the mean time, most of these major players will agree to "full investigations" - which of course will be focused primarily on the contents of these leaked files - and they will be pronounced "clean as a whistle" because within the files of their indicting evidence also lies their exculpatory evidence.
Perhaps this is a wild flight of fancy. But it appears clear that we are reviewing a "Presentation" not a "Theft." In recognition of that fact, the files found should be considered a reason to conduct a broader investigation of the East Anglia CRU - not be held up as "proof" of their guilt. For skeptics to place undue weight on these highly refined "presentation files" of unknown origin may well prove to be a costly mistake.