## Guest Post – Bayesian Book Review

*My regular commenter Anton circulated this book review by email yesterday and it stimulated quite a lot of reaction. I haven’t read the book myself, but I thought it would be fun to post his review on here to see whether it provokes similar responses. You can find the book on Amazon here (UK) or here ( USA). If you’re not completely au fait with Bayesian probability and the controversy around it, you might try reading one of my earlier posts about it, e.g. this one. I hope I can persuade some of the email commenters to upload their contributions through the box below!*

-0-

**The Theory That Would Not Die: How Bayes’ Rule Cracked the Enigma Code, Hunted Down Russian Submarines, and Emerged Triumphant from Two Centuries of Controversy**

by Sharon Bertsch Mcgrayne

I found reading this book, which is a history of Bayes’ theorem written for the layman, to be deeply frustrating. The author does not really understand what probability IS – which is the key to all cogent writing on the subject. She never mentions the sum and product rules, or that Bayes’ theorem is an easy consequence of them. She notes, correctly, that Bayesian methods or something equivalent to them have been rediscovered advantageously again and again in an amazing variety of practical applications, and says that this is because they are pragmatically better than frequentist sampling theory – ie, she never asks the question: Why do they work better and what deeper rationale explains this? RT Cox is not mentioned. Ed Jaynes is mentioned only in passing as someone whose Bayesian fervour supposedly put people off.

The author is correct that computer applications have catalysed the Bayesian revolution, but in the pages on image processing and other general inverse problems (p218-21) she manages to miss the key work through the 1980s of Steve Gull and John Skilling, and you will not find “Maximum entropy” in the index. She does get the key role of Markov Chain Monte Carlo methods in computer implementation of Bayesian methods, however. But I can’t find Dave Mackay either, who deserves to be in the relevant section about modern applications.

On the other hand, as a historian of Bayesianism from Bayes himself to about 1960, she is full of superb anecdotes and information about

people who are to us merely names on the top of papers, or whose personalities are mentioned tantalisingly briefly in Jaynes’ writing.

For this material alone I recommend the book to Bayesians of our sort and am glad that I bought it.

May 30, 2011 at 2:04 pm

There is, IIRC, a paper by Coles and Garrett which might or might not mention Jaynes. Are you that Garrett? Is the paper available online anywhere?

I just finished reading Mlodinow’s

The Drunkard’s Walk, which is probably a better introduction to probability.May 30, 2011 at 2:14 pm

Yes that’s me Phillip. As to whether it is online – ask Peter.

May 30, 2011 at 2:24 pm

OK, maybe he can provide a link if there is one.

May 30, 2011 at 2:37 pm

The paper was published in Comments on Astrophysics, Vol. 17, p23, but that journal doesn’t have a website. Unfortunately I’ve lost the .tex file for the paper so I can’t post it to the arXiv. As far as I’m aware nobody else has put it up on the web…

May 30, 2011 at 2:43 pm

Are the contents available in any other paper? I won’t mention any names, but I have run across papers (especially conference proceedings) which are very similar to other papers. π (John Bahcall lists such papers as 548a, 548b, 548c etc in his list of publications.)

May 30, 2011 at 3:40 pm

It was on the anthropic principle but we also enjoyed ourselves by bashing non-Bayesian probability and the Copenhagen wimp-out interpretation (and some other views) of quantum mechanics. I published a paper on the anthropic principle in a proceedings of the Maximum Entropy conference series which certainly had some overlap. The basic idea is that the Bayesian view of probability is the razor needed to excise the ample nonsense written about the anthropic principle from the plentiful good sense that has also been written about it.

May 30, 2011 at 3:49 pm

Sounds worth reading. Maybe I can find a copy.

Yours isn’t one of the names I’m not mentioning. π

May 30, 2011 at 3:52 pm

I have a hard copy which I could scan, I guess, but I’m wary of copyright infringement if I put it on the web anywhere…

May 30, 2011 at 4:54 pm

Peter: While I know that the journal in question had a sensible editor, your comment shows EXACTLY what is wrong with science publishing: a senior scientist dare not put his own work up on the Web for fear of breaking copyright. That same Web will soon end this state of exploitation, I trust.

May 31, 2011 at 4:50 pm

What is the current status of copyright with respect to scientific papers? Since almost everything is on the ArXiv, presumably there is no conflict or the situation is tolerated, at least for newer papers. Also, many older papers are now scanned in and publicly available.

While I am a strong defender of copyright with respect to illegal downloading of music etc, the situation in science is clearly different (primarily since scientists don’t earn their living from selling copies of their papers). As John Baez said, in many cases scientists pay to have something published then have to buy the journal it gets published in. π¦

May 31, 2011 at 6:30 pm

Phillip: Isn’t the point that you submit to the arXiv *before* you sign away copyright to a publisher? I don’t know (do you Peter?) if publishers ever threatened not to publish stuff that had appeared on a scientist’s website, or on the arXiv in its early days; but if so then we won that one.

The original function of academic publishers was to disseminate information. Then a quality control element came in (via the refereeiing system). Now the internet has totally removed the original function. It should not be beyond the scientific community to work out some kind of quality control method for online publishing at a tiny fraction of the cost of today’s journals, and end the iniquity of signing away copyright in our own work.

May 31, 2011 at 7:06 pm

“Isnβt the point that you submit to the arXiv *before* you sign away copyright to a publisher?”That might be the rule, but there are folks who put stuff on the ArXiv only after it has been accepted. In any case, that is a weak defense, since usually the ArXiv preprints are processed with the LaTeX style file of the appropriate journal.

Some journals do enforce an embargo, i.e. any “pre”print cannot appear until after the article has appeared in the journal.

I don’t know if there is still a need for paper journals at all. Whether or not there is, I think that journals should be run as non-profit enterprises, perhaps by the appropriate professional organisations (IAU, RAS, IOP etc). IIRC, some professional societies do have open-access journals. (The New Journal of Physics is free to read, but costs to publish.)

May 31, 2011 at 2:57 am

Dear Anton

Thanks for pointing us to this book.

I will certainly give it a read especially since I teach a graduate

course on Bayesian Data Analysis here at the University at Albany and

new perspectives on the history will be useful.

It is disappointing that the Cox-Jaynes thread in the history is

missing since it is the one that I have found that has led to *deep

understanding*. I was at a NASA meeting a few years ago and a person

(a good guy who I will not name) from the ISBA crowd gave a tutorial

and at the end mentioned several introductory texts. Absent from the

tutorial was any mention of Cox, Jaynes, etc. I spoke up and

mentioned this important thread (which as you probably know led to

many of us at NASA Ames: Cheeseman, Stutz, Scargle, Wolpert, myself,

etc.). I also mentioned Devinder’s text as particularly accessible

and useful, as well as MacKay’s, Gregory’s and Bretthorst’s. I also

recall at the ISBA meeting in Chile in 2004 or so Arnold Zellner

scolding the crowd for neglecting 1/2 of the picture: entropy and

information. As revered as he was by that community, his outburst was

laughed off as the rantings of a crazy old man.

Sad really.

As John Skilling can attest, I myself have found Cox’s insights to be

profound. I have taken them ever so seriously and with surprising and

delightful results. The lesson I have taken away is that all one

needs to do is quantify the order in what you are looking at.

Consistent quantification may not be useful, but it can’t be wrong!

This makes it a great place to start. The result is that the

constraint equations enforcing consistent quantification will result

in the laws that govern the quantified system. This is precisely why

laws reflect an underlying order—laws derive from that order.

This idea of generalizing an algebra has led not only to a theory of

questions (which, for those of you who have been following, I now

understand how to handle properly), but also has led several of us

(Goyal, Skilling and myself) to a derivation of the Feynman rules of

quantum mechanics based on similar symmetries (inspired, in part, by

the earlier works of Tikochinsky, Gull and Caticha), as well as a

novel derivation of special relativity and the geometry and

dimensionality of space (which was quite unexpected and quite unlike

anything I have yet found):

Goyal, Knuth, Skilling, Phys Rev A, 2010 (

http://pra.aps.org/abstract/PRA/v81/i2/e022109 )

Goyal, Knuth, Symmetry 2011 ( http://www.mdpi.com/2073-8994/3/2/171/ )

Knuth, Bahreyni 2010 ( http://arxiv.org/abs/1005.4172 )

Skilling, Knuth 2010 ( http://arxiv.org/abs/1008.4831 )

Propelled and emboldened by these advances, my colleague Keith Earle

and myself (the most unlikely of suspects, as neither of us are

theorists by training) are working on a derivation of the Dirac equation and have already obtained new insights as well as a glimpse into how to handle both electromagnetism and gravity in concert with quantum mechanics. Keith is working from the middle up ( http://arxiv.org/abs/1102.1200 ) and I am working from the bottom to the middle. This is what I mainly am focused on now.

At this point in time, the ideas of Cox have inspired us to derive far more than probability theory!

Bayes was just the beginning.

Cox and Jaynes, taken seriously, have led to deep understanding and

their insights have now taken us light-years beyond statistics.

(I haven’t even begun to mention the advances in Maximum entropy)

Don’t despair, THE BOOK hasn’t been written yet.

I seriously feel that in the long run, this thread through the long

story of probability will turn out to be the most amazing one!

Cheers

Kevin Knuth

May 31, 2011 at 4:45 pm

Your name (which is not that common) prompts asking the question whether you are

hisson. π