Guest Post – Bayesian Book Review

My regular commenter Anton circulated this book review by email yesterday and it stimulated quite a lot of reaction. I haven’t read the book myself, but I thought it would be fun to post his review on here to see whether it provokes similar responses. You can find the book on Amazon here (UK) or here ( USA). If you’re not completely au fait with Bayesian probability and the controversy around it, you might try reading one of my earlier posts about it, e.g. this one. I hope I can persuade some of the email commenters to upload their contributions through the box below!

-0-

The Theory That Would Not Die: How Bayes’ Rule Cracked the Enigma Code, Hunted Down Russian Submarines, and Emerged Triumphant from Two Centuries of Controversy

by Sharon Bertsch Mcgrayne

I found reading this book, which is a history of Bayes’ theorem written for the layman, to be deeply frustrating. The author does not really understand what probability IS – which is the key to all cogent writing on the subject. She never mentions the sum and product rules, or that Bayes’ theorem is an easy consequence of them. She notes, correctly, that Bayesian methods or something equivalent to them have been rediscovered advantageously again and again in an amazing variety of practical applications, and says that this is because they are pragmatically better than frequentist sampling theory – ie, she never asks the question: Why do they work better and what deeper rationale explains this? RT Cox is not mentioned. Ed Jaynes is mentioned only in passing as someone whose Bayesian fervour supposedly put people off.

The author is correct that computer applications have catalysed the Bayesian revolution, but in the pages on image processing and other general inverse problems (p218-21) she manages to miss the key work through the 1980s of Steve Gull and John Skilling, and you will not find “Maximum entropy” in the index. She does get the key role of Markov Chain Monte Carlo methods in computer implementation of Bayesian methods, however. But I can’t find Dave Mackay either, who deserves to be in the relevant section about modern applications.

On the other hand, as a historian of Bayesianism from Bayes himself to about 1960, she is full of superb anecdotes and information about
people who are to us merely names on the top of papers, or whose personalities are mentioned tantalisingly briefly in Jaynes’ writing.
For this material alone I recommend the book to Bayesians of our sort and am glad that I bought it.

Advertisements

14 Responses to “Guest Post – Bayesian Book Review”

  1. There is, IIRC, a paper by Coles and Garrett which might or might not mention Jaynes. Are you that Garrett? Is the paper available online anywhere?

    I just finished reading Mlodinow’s The Drunkard’s Walk, which is probably a better introduction to probability.

    • Anton Garrett Says:

      Yes that’s me Phillip. As to whether it is online – ask Peter.

    • OK, maybe he can provide a link if there is one.

    • telescoper Says:

      The paper was published in Comments on Astrophysics, Vol. 17, p23, but that journal doesn’t have a website. Unfortunately I’ve lost the .tex file for the paper so I can’t post it to the arXiv. As far as I’m aware nobody else has put it up on the web…

    • Are the contents available in any other paper? I won’t mention any names, but I have run across papers (especially conference proceedings) which are very similar to other papers. πŸ™‚ (John Bahcall lists such papers as 548a, 548b, 548c etc in his list of publications.)

    • Anton Garrett Says:

      It was on the anthropic principle but we also enjoyed ourselves by bashing non-Bayesian probability and the Copenhagen wimp-out interpretation (and some other views) of quantum mechanics. I published a paper on the anthropic principle in a proceedings of the Maximum Entropy conference series which certainly had some overlap. The basic idea is that the Bayesian view of probability is the razor needed to excise the ample nonsense written about the anthropic principle from the plentiful good sense that has also been written about it.

    • Sounds worth reading. Maybe I can find a copy.

      Yours isn’t one of the names I’m not mentioning. πŸ™‚

    • telescoper Says:

      I have a hard copy which I could scan, I guess, but I’m wary of copyright infringement if I put it on the web anywhere…

    • Anton Garrett Says:

      Peter: While I know that the journal in question had a sensible editor, your comment shows EXACTLY what is wrong with science publishing: a senior scientist dare not put his own work up on the Web for fear of breaking copyright. That same Web will soon end this state of exploitation, I trust.

    • What is the current status of copyright with respect to scientific papers? Since almost everything is on the ArXiv, presumably there is no conflict or the situation is tolerated, at least for newer papers. Also, many older papers are now scanned in and publicly available.

      While I am a strong defender of copyright with respect to illegal downloading of music etc, the situation in science is clearly different (primarily since scientists don’t earn their living from selling copies of their papers). As John Baez said, in many cases scientists pay to have something published then have to buy the journal it gets published in. 😦

    • Anton Garrett Says:

      Phillip: Isn’t the point that you submit to the arXiv *before* you sign away copyright to a publisher? I don’t know (do you Peter?) if publishers ever threatened not to publish stuff that had appeared on a scientist’s website, or on the arXiv in its early days; but if so then we won that one.

      The original function of academic publishers was to disseminate information. Then a quality control element came in (via the refereeiing system). Now the internet has totally removed the original function. It should not be beyond the scientific community to work out some kind of quality control method for online publishing at a tiny fraction of the cost of today’s journals, and end the iniquity of signing away copyright in our own work.

    • “Isn’t the point that you submit to the arXiv *before* you sign away copyright to a publisher?”

      That might be the rule, but there are folks who put stuff on the ArXiv only after it has been accepted. In any case, that is a weak defense, since usually the ArXiv preprints are processed with the LaTeX style file of the appropriate journal.

      Some journals do enforce an embargo, i.e. any “pre”print cannot appear until after the article has appeared in the journal.

      I don’t know if there is still a need for paper journals at all. Whether or not there is, I think that journals should be run as non-profit enterprises, perhaps by the appropriate professional organisations (IAU, RAS, IOP etc). IIRC, some professional societies do have open-access journals. (The New Journal of Physics is free to read, but costs to publish.)

  2. Dear Anton

    Thanks for pointing us to this book.
    I will certainly give it a read especially since I teach a graduate
    course on Bayesian Data Analysis here at the University at Albany and
    new perspectives on the history will be useful.

    It is disappointing that the Cox-Jaynes thread in the history is
    missing since it is the one that I have found that has led to *deep
    understanding*. I was at a NASA meeting a few years ago and a person
    (a good guy who I will not name) from the ISBA crowd gave a tutorial
    and at the end mentioned several introductory texts. Absent from the
    tutorial was any mention of Cox, Jaynes, etc. I spoke up and
    mentioned this important thread (which as you probably know led to
    many of us at NASA Ames: Cheeseman, Stutz, Scargle, Wolpert, myself,
    etc.). I also mentioned Devinder’s text as particularly accessible
    and useful, as well as MacKay’s, Gregory’s and Bretthorst’s. I also
    recall at the ISBA meeting in Chile in 2004 or so Arnold Zellner
    scolding the crowd for neglecting 1/2 of the picture: entropy and
    information. As revered as he was by that community, his outburst was
    laughed off as the rantings of a crazy old man.
    Sad really.

    As John Skilling can attest, I myself have found Cox’s insights to be
    profound. I have taken them ever so seriously and with surprising and
    delightful results. The lesson I have taken away is that all one
    needs to do is quantify the order in what you are looking at.
    Consistent quantification may not be useful, but it can’t be wrong!
    This makes it a great place to start. The result is that the
    constraint equations enforcing consistent quantification will result
    in the laws that govern the quantified system. This is precisely why
    laws reflect an underlying order—laws derive from that order.

    This idea of generalizing an algebra has led not only to a theory of
    questions (which, for those of you who have been following, I now
    understand how to handle properly), but also has led several of us
    (Goyal, Skilling and myself) to a derivation of the Feynman rules of
    quantum mechanics based on similar symmetries (inspired, in part, by
    the earlier works of Tikochinsky, Gull and Caticha), as well as a
    novel derivation of special relativity and the geometry and
    dimensionality of space (which was quite unexpected and quite unlike
    anything I have yet found):

    Goyal, Knuth, Skilling, Phys Rev A, 2010 (
    http://pra.aps.org/abstract/PRA/v81/i2/e022109 )
    Goyal, Knuth, Symmetry 2011 ( http://www.mdpi.com/2073-8994/3/2/171/ )
    Knuth, Bahreyni 2010 ( http://arxiv.org/abs/1005.4172 )
    Skilling, Knuth 2010 ( http://arxiv.org/abs/1008.4831 )

    Propelled and emboldened by these advances, my colleague Keith Earle
    and myself (the most unlikely of suspects, as neither of us are
    theorists by training) are working on a derivation of the Dirac equation and have already obtained new insights as well as a glimpse into how to handle both electromagnetism and gravity in concert with quantum mechanics. Keith is working from the middle up ( http://arxiv.org/abs/1102.1200 ) and I am working from the bottom to the middle. This is what I mainly am focused on now.

    At this point in time, the ideas of Cox have inspired us to derive far more than probability theory!

    Bayes was just the beginning.
    Cox and Jaynes, taken seriously, have led to deep understanding and
    their insights have now taken us light-years beyond statistics.
    (I haven’t even begun to mention the advances in Maximum entropy)

    Don’t despair, THE BOOK hasn’t been written yet.
    I seriously feel that in the long run, this thread through the long
    story of probability will turn out to be the most amazing one!

    Cheers
    Kevin Knuth

Leave a Reply to Phillip Helbig Cancel reply

Fill in your details below or click an icon to log in:

WordPress.com Logo

You are commenting using your WordPress.com account. Log Out /  Change )

Google photo

You are commenting using your Google account. Log Out /  Change )

Twitter picture

You are commenting using your Twitter account. Log Out /  Change )

Facebook photo

You are commenting using your Facebook account. Log Out /  Change )

Connecting to %s

%d bloggers like this: