## On the Edgeworth Series…

Posted in Bad Statistics, The Universe and Stuff with tags , , on September 12, 2017 by telescoper

There’s a nice paper on the arXiv today by Elena Sellentin, Andrew Jaffe and Alan Heavens about the use of the Edgeworth series in statistical cosmology; it is evidently the first in a series about the Edgeworth series.

Here is the abstract:

Non-linear gravitational collapse introduces non-Gaussian statistics into the matter fields of the late Universe. As the large-scale structure is the target of current and future observational campaigns, one would ideally like to have the full probability density function of these non-Gaussian fields. The only viable way we see to achieve this analytically, at least approximately and in the near future, is via the Edgeworth expansion. We hence rederive this expansion for Fourier modes of non-Gaussian fields and then continue by putting it into a wider statistical context than previously done. We show that in its original form, the Edgeworth expansion only works if the non-Gaussian signal is averaged away. This is counterproductive, since we target the parameter-dependent non-Gaussianities as a signal of interest. We hence alter the analysis at the decisive step and now provide a roadmap towards a controlled and unadulterated analysis of non-Gaussianities in structure formation (with the Edgeworth expansion). Our central result is that, although the Edgeworth expansion has pathological properties, these can be predicted and avoided in a careful manner. We also show that, despite the non-Gaussianity coupling all modes, the Edgeworth series may be applied to any desired subset of modes, since this is equivalent (to the level of the approximation) to marginalising over the exlcuded modes. In this first paper of a series, we restrict ourselves to the sampling properties of the Edgeworth expansion, i.e.~how faithfully it reproduces the distribution of non-Gaussian data. A follow-up paper will detail its Bayesian use, when parameters are to be inferred.

The Edgeworth series – a method of approximating a probability distribution in terms of a series determined by its cumulants – has found a number of cosmological applications over the years, but it does suffer from a number of issues, one of the most important being that it is not guaranteed to be a proper probability distribution, in that the resulting probabilities can be negative…

I’ve been thinking about how to avoid this issue myself, and mentioned a possibility in the talk I gave at South Kensington Technical Imperial College earlier this summer. The idea is to represent the cosmological density field (usually denoted δ) in terms of the square of the modulus of a (complex) wave function ψ i.e. |ψψ*|. It then turns out that the evolution equations for cosmic fluid can be rewritten as a kind of Schrodinger equation. One powerful advantage of this approach is that whatever you do in terms of approximating ψ, the resulting density ψψ* is bound to be positive. This finesses the problem of negative probabilities but at the price of introducing more complexity (geddit?) into the fluid equations. On the other hand, it does mean that even first-order perturbative evolution of ψ guarantees a sensible probability distribution whereas first-order evolution of δ does not and has

## Cosmological Results from the Dark Energy Survey

Posted in The Universe and Stuff with tags , , , , , on August 4, 2017 by telescoper

At last the Dark Energy Survey has produced its first cosmological results. The actual papers have not yet hit the arXiv but they have been announced at a meeting in the USA and are linked to from this page.

I’ll jump straight to this one, which shows the joint constraints on S8 which is related to σ8 (a measure of the level of fluctuations in the cosmological mass distribution) via S8= σ8m/0.3)0.5 against the cosmological density parameter, Ωm.

These constraints, derived using DES Y1 measurements of galaxy clustering, galaxy-galaxy lensing, and weak lensing cosmic shear are compared with those obtained from the cosmic microwave background using Planck data, and also combined with them to produce a joint constraint. Following usual practice, the contours are 68% and 95%  posterior probability regions.

The central values of DES and Planck values are different, but the discrepancy is only marginal. Compare this with a an equivalent diagram from a paper I discussed last year.

The KIDS analysis used to produce this plot uses only weak lensing tomography, so you can see that using additional measures reduces the viable region in this parameter space.

It’s great to see new data coming in, but at first sight it seems it is tending to confirm the predictions of the standard cosmological model, rather than providing evidence of departures from it.

Incidentally, this little video shows the extent to which the Dark Energy Survey is a global project, including some of my former colleagues at the University of Sussex!

## What the Power Spectrum misses

Posted in The Universe and Stuff with tags , , , , , , , on August 2, 2017 by telescoper

Just taking a short break from work I chatted over coffee to one of the students here at the Niels Bohr Institute about various things to do with the analysis of signals in the Fourier domain (as you do). That discussion reminded me of this rather old post (from 2009) which I thought might be worth a second airing (after a bit of editing). The discussion is all based on past cosmological data (from WMAP) rather than the most recent (from Planck), but that doesn’t change anything qualitatively. So here you are.

The picture above shows the all-sky map of fluctuations in the temperature of the cosmic microwave background across the sky as revealed by the Wilkinson Microwave Anisotropy Probe, known to its friends as WMAP.

I spent many long hours fiddling with the data coming from the WMAP experiment, partly because I’ve never quite got over the fact that such wonderful data actually exists. When I started my doctorate in 1985 the whole field of CMB analysis was so much pie in the sky, as no experiments had yet been performed with the sensitivity to reveal the structures we now see. This is because they are very faint and easily buried in noise. The fluctuations in temperature from pixel to pixel across the sky are of order one part in a hundred thousand of the mean temperature (i.e. about 30 microKelvin on a background temperature of about 3 Kelvin). That’s smoother than the surface of a billiard ball. That’s why it took such a long time to make the map shown above, and why it is such a triumphant piece of science.

I blogged a while ago about the idea that the structure we see in this map was produced by sound waves reverberating around the early Universe. The techniques cosmologists use to analyse this sound are similar to those used in branches of acoustics except that we only see things in projection on the celestial sphere which requires a bit of special consideration.

One of the things that sticks in my brain from my undergraduate years is being told that `if you don’t know what you’re doing as a physicist you should start by making a Fourier transform of everything. This approach breaks down the phenomenon being studied into a set of  plane waves with different wavelengths corresponding to analysing the different tones present in a complicated sound.

It’s often very good advice to do such a decomposition for one-dimensional time series or fluctuation fields in three-dimensional Cartesian space, even you do know what you’re doing, but it doesn’t work with a sphere because plane waves don’t fit properly on a curved surface. Fortunately, however, there is a tried-and-tested alternative involving spherical harmonics rather than plane waves.

Spherical harmonics are quite complicated beasts mathematically but they have pretty similar properties to Fourier harmonics in many respects. In particular they are represented as complex numbers having real and imaginary parts or, equivalently, an amplitude and a phase (usually called the argument by mathematicians),

$Z=X+iY = R \exp(i\phi)$

This latter representation is the most useful one for CMB fluctuations because the simplest versions of inflationary theory predict that the phases φ of each of the spherical harmonic modes should be randomly distributed. What this really means is that there is no information content in their distribution so that the harmonic modes are in a state of maximum statistical disorder or entropy. This property also guarantees that the distribution of fluctuations over the sky should have a Gaussian distribution.

If you accept that the fluctuations are Gaussian then only the amplitudes of the spherical harmonic coefficients are useful. Indeed, their statistical properties can be specified entirely by the variance of these amplitudes as a function of mode frequency. This pre-eminently important function is called the power-spectrum of the fluctuations, and it is shown here for the WMAP data:

Although the units on the axes are a bit strange it doesn”t require too much imagination to interpret this in terms of a sound spectrum. There is a characteristic tone (at the position of the big peak) plus a couple of overtones (the bumps at higher frequencies). However these features are not sharp so the overall sound is not at all musical.

If the Gaussian assumption is correct then the power-spectrum contains all the useful statistical information to be gleaned from the CMB sky, which is why so much emphasis has been placed on extracting it accurately from the data.

Conversely, though, the power spectrum is completely insensitive to any information in the distribution of spherical harmonic phases. If something beyond the standard model made the Universe non-Gaussian it would affect the phases of the harmonic modes in a way that would make them non-random.

However,I will now show you how important phase information could actually be, if only we could find a good way of exploiting it. Let’s start with a map of the Earth, with the colour representing height of the surface above mean sea level:

You can see the major mountain ranges (Andes, Himalayas) quite clearly as red in this picture and note how high Antarctica is…that’s one of the reasons so much astronomy is done there.

Now, using the same colour scale we have the WMAP data again (in Galactic coordinates).

The virture of this representation of the map is that it shows how smooth the microwave sky is compared to the surface of the Earth. Note also that you can see a bit of crud in the plane of the Milky Way that serves as a reminder of the difficulty of cleaning the foregrounds out.

Clearly these two maps have completely different power spectra. The Earth is dominated by large features made from long-wavelength modes whereas the CMB sky has relatively more small-scale fuzz.

Now I’m going to play with these maps in the following rather peculiar way. First, I make a spherical harmonic transform of each of them. This gives me two sets of complex numbers, one for the Earth and one for WMAP. Following the usual fashion, I think of these as two sets of amplitudes and two sets of phases. Note that the spherical harmonic transformation preserves all the information in the sky maps, it’s just a different representation.

Now what I do is swap the amplitudes and phases for the two maps. First, I take the amplitudes of WMAP and put them with the phases for the Earth. That gives me the spherical harmonic representation of a new data set which I can reveal by doing an inverse spherical transform:

This map has exactly the same amplitudes for each mode as the WMAP data and therefore possesses an identical power spectrum to that shown above. Clearly, though, this particular CMB sky is not compatible with the standard cosmological model! Notice that all the strongly localised features such as coastlines appear by virtue of information contained in the phases but absent from the power-spectrum.

To understand this think how sharp features appear in a Fourier transform. A sharp spike at a specific location actually produces a broad spectrum of Fourier modes with different frequencies. These modes have to add in coherently at the location of the spike and cancel out everywhere else, so their phases are strongly correlated. A sea of white noise also has a flat power spectrum but has random phases. The key difference between these two configurations is not revealed by their spectra but by their phases.

Fortunately there is nothing quite as wacky as a picture of the Earth in the real data, but it makes the point that there are more things in Heaven and Earth than can be described in terms of the power spectrum!

Finally, perhaps in your mind’s eye you might consider what it might look lie to do the reverse experiment: recombine the phases of WMAP with the amplitudes of the Earth.

If the WMAP data are actually Gaussian, then this map is a sort of random-phase realisation of the Earth’s power spectrum. Alternatively you can see that it is the result of running a kind of weird low-pass filter over the WMAP fluctuations. The only striking things it reveals are (i) a big blue hole associated with foreground contamination, (ii) a suspicious excess of red in the galactic plane owing to the same problem, and (iiI) a strong North-South asymmetry arising from the presence of Antarctica.

There’s no great scientific result here, just a proof that spherical harmonic phases are potentially interesting because of the information they contain about strongly localised features

PS. These pictures were made by a former PhD student of mine, Patrick Dineen, who has since quit astrophysics  to work in the financial sector for Winton Capital, which has over the years recruited a number of astronomy and cosmology graduates and also sponsors a Royal Astronomical Society prize. That shows that the skills and knowledge obtained in the seemingly obscure field of cosmological data analysis have applications elsewhere!

## CMB Spectral Distortions Revisited

Posted in Biographical, The Universe and Stuff with tags , , , , , , , , , on July 27, 2017 by telescoper

While uploading some bibliographic information for bureaucratic purposes yesterday I noticed that an old paper of mine had recently attracted a number of citations. The paper was written while I was a postdoctoral research fellow in the Astronomy Centre at the University of Sussex in 1990, but not published until 1991 by which time I had moved to Queen Mary College (as it was then called). The citation history of this article is actually quite interesting:

You can see that it was cited a bit immediately after publication, then endured a long spell from 1997 to 2012 in which nobody seemed interested in it, then experienced something of a revival. It currently has a total of about 49 citations, which doesn’t exactly make it a classic in a field which is extremely active, but it’s nice to see it hasn’t been forgotten entirely.

Here is the abstract of the paper:

As the abstract makes clear we wrote this paper in response to a measurement of the spectrum of the cosmic microwave background radiation by the FIRAS instrument on the satellite COBE that had demonstrated that it was extremely well fitted by a Planck spectrum, with little room for any deviation away from a perfect black-body shape. Here’s the measured curve from COBE and some other experiments at the time:

The accuracy of the fit allows one to place limits on any process happening in the early Universe that might produce a distortion of the spectrum. There are a number of things that could do this. Any energy released in the early Universe takes time to thermalise, i.e. for the radiation field and the matter to come into thermal equilibrium via Compton scattering, double Compton scattering and Bremsstrahlung. Imperfect thermalisation produces a spectrum which doesn’t quite match the Planck curve.

Two types of distortion are possible, both introduced in classic papers from 1969 and 1970 by Rashid Sunyaev and Ya. B. Zel’dovich. One type is called a y-distortion (which corresponds to photons being shifted from low frequency and the other is called a μ-distortion, which is described by inserting a chemical potential term to the usual Planck formula for the black-body spectrum. Observational limits on both forms of distortion are very tight : |y|<1.5 ×10-5; |μ|<1.5 ×10-5, which places stringent limits on any energy release, including that which would arise from the dissipation of primordial acoustic waves (which is what John and I concentrated on in the paper).

So why did interest in this get revived a few years ago? The answer to that is that advances in relevant technology have now made it possible to think about an experiment that can measure much smaller spectral distortions than has hitherto been possible. A proposal for an experiment, called PIXIE, which includes such a measurement, is described here. Although spectral distortions are only a secondary science goal for PIXIE, it could push down the upper limits quoted above by a factor of 1000 or so, at which level we should expect to see departures from the Planck curve within the standard model, which would be a very important test of basic cosmological theory.

That all depends on whether PIXIE – or something like it – goes ahead.

## The Early Stages of Cosmic Inflation

Posted in The Universe and Stuff with tags , , , , on July 13, 2017 by telescoper

When asked to cite the article that first presented the theory of the inflationary Universe most cosmologists would probably offer the famous paper published by Alan Guth in 1981.

However, I recently stumbled across a paper by Demosthenes Kazanas that was published (in the Astrophysical Journal Letters in 1980. I hadn’t seen this paper before a few days ago, and I don’t think it is very well known. Here is part of the front page:

You can get the full paper here.

I know that there were other papers floating around in 1980 that got part of the way to the theory of inflation, but this one seems very close to the theory, e.g. talking about exponential expansion in the context of the cosmological horizon problem.

Interestingly, while Guth (1981) has garnered many thousands of citations, Kazanas (1980) has been cited fewer than 300 times.

Does anyone know the story of this paper, and why it has largely  been overlooked by the exponentially-expanding literature on cosmic inflation? And,while I’m on the topic, can anyone suggest other early contributions to the theory that have been similarly neglected? Please let me know through the comments box below.

## Film Noir, Physics, and the Futility of Existence

Posted in Biographical, Literature, The Universe and Stuff with tags , , , , , on July 5, 2017 by telescoper

Last night I decided to treat myself to the umpteenth viewing of a DVD of the  classical film  Double Indemnity. It’s a great movie that repays repeated viewing and is historically important for many reasons, not least because when it was released in 1944 it immediately established much of the language and iconography of the genre that has come to be known as Film Noir , which I’ve written about on a number of occasions on this blog; see here, for example.

After watching the film I had a look on Twitter to see if I had any messages and saw a thread about how modern physics inspires, in some people, an all-pervading sense of existential angst.  In the light of that discussion I decided to use my morning off to rehash that old post and add a few embellishments.

It’s difficult to define exactly what turns a film noir, but there are some common characteristics. First the male lead protagonist is far from the dashing romantic character portrayed in mainstream Hollywood fare. Often a troubled and dysfunctional character, cynical and hard-bitten, distrustful and alienated, the classic noir anti-hero is often a private investigator or in any case a loner who lives in a kind of moral vacuum. To counterpoint this, the female lead is usually a femme fatale, glamorous but duplicitous, sexy and dangerous, manipulative and assertive. There are definitely shades of Macbeth in that the female lead is usually a more compelling and impressive personality than the supposed hero. The inversion of stereotypical roles also serves to hold a “dark mirror” up to society, an effect which other elements of these films also strive to achieve.

The plots usually deal with the seedy side of human life: crime, betrayal, jealousy and revenge, much of it sexually motivated. Narrative strategies involve repeated use of flashbacks, first-person voiceovers, dream-like sequences, and unresolved episodes that emphasize the overall lack of moral direction. The photography is dominated by high contrast lights surrounding the protagonists with dark, threatening shadows while odd angles and unbalanced framing produce unstable, disorienting images. The chiaroscuro lighting makes even mundane encounters seem charged with danger or erotic suspense.

This is a still from Double Indemnity which shows a number of trademark features. The shadows cast by Venetian blinds on the wall, the cigarette being smoked by Barbara Stanwyck and the curious construction of the mise en scene are all very characteristic of the style. What is even more wonderful about this particular shot however is the way the shadow of Fred McMurray’s character enters the scene before he does. The Barbara Stanwyck character is just about to shoot him with a pearl-handled revolver; this image seems to be hinting that he is already on his way to the underworld even before he arrives in the room.

Noir settings are almost exclusively urban: the resulting iconography consists of images of dark night-time cities with rain-soaked streets reflecting dazzling neon lights that intrude into the picture and fracture the composition. Interiors are almost always cramped and claustrophobic: dingy hotel rooms, night clubs or even the backs of taxi cabs. The dark outside world presses in on the characters and is full of danger. Soundtracks often include jazz in the bebop style from the late 1940s or early 1950s, with its jagged melodic lines and stuttering rythms, emphasizing the psychological instability displayed by the characters and settings.

The protagonists are trapped, perhaps just by mischance, in an alienating lonely world, usually a night-time city, where they are constantly in danger for their lives. The chaotic, random violence of this world gives rise to feelings of persecution and paranoia and a sense that life is absurd, meaningless, without order or purpose, and governed by contingency rather than design.

Much has been written about the origins of Film Noir, but it does seem clear to me that, although it is essentially an American style, it owes many of its roots to European existentialism, a point further reinforced by the fact that many great movie directors of the noir period (including the great Billy Wilder, who directed Double Indemnity) were in fact European emigres.

Anyway, I digress. What I wanted to say really was that during the course of watching all these wonderful films from a bygone age it struck me how much the language and iconography of modern cosmology shares this existentialist heritage. Our new standard cosmological model is full of references to the “dark” sector (dark matter and dark energy) which dominates the energy budget of the Universe, but which not just invisible but also unfathomable. The cosmos is lit by garish starlight from small islands of luminosity embedded in this sea of darkness. Long chains of bright galaxies stretch across space like rows of streetlights whose glare fractures and disturbs the celestial dark. We cling to a precarious existence on a tiny rock that is surrounded by danger. Even the stuff from which our atoms are made is completely overshadowed by alien matter. The universe is oblivious to us and we are irrelevant to it.

But it’s not only the surface imagery of cosmology that resembles that of a noir movie. The existentialist trend runs deep. Cosmology seems to be abandoning the idea that there is a design behind it all. The idea that there is a single explanatory principle “a theory of everything” that accounts for why our Universe is the way it is and why life is possible within it, is losing ground to the idea that there is a multiverse in which all possible laws of nature are realised; we just live in a place where life happens to be possible. I’m not at all convinced that it is a good route for science to follow, but many cosmologists seem to be accepting this kind of thing as the best we will ever do to explain the Universe.

The physicist Steven Weinberg summed up the way this view of the Universe challenges us:

It is almost irresistible for humans to believe that we have some special relation to the universe, that human life is not just a more-or-less farcical outcome of a chain of accidents reaching back to the first three minutes, but that we were somehow built in from the beginning. … It is very hard to realise that this is all just a tiny part of an overwhelmingly hostile universe. It is even harder to realise that this present universe has evolved from an unspeakably unfamiliar early condition, and faces a future extinction of endless cold or intolerable heat. The more the universe seems comprehensible, the more it also seems pointless.

In an interview he put it thus:

I believe that there is no point in the universe that can be discovered by the methods of science. I believe that what we have found so far, an impersonal universe in which it is not particularly directed toward human beings is what we are going to continue to find. And that when we find the ultimate laws of nature they will have a chilling, cold impersonal quality about them.

The influential American horror writer Howard Phillips Lovecraft (known to his friends as “H.P.”), wrote a letter in 1919 that argued much the same:

As you are aware, I have never been able to soothe myself with the sugary delusions of religion; for these things stand convicted of the utmost absurdity in light of modern scientific knowledge. With Nietzsche, I have been forced to confess that mankind as a whole has no goal or purpose whatsoever, but is a mere superfluous speck in the unfathomable vortices of infinity and eternity. Accordingly, I have hardly been able to experience anything which one could call real happiness; or to take as vital an interest in human affairs as can one who still retains the hallucination of a “great purpose” in the general plan of terrestrial life. … However, I have never permitted these circumstances to react upon my daily life; for it is obvious that although I have “nothing to live for”, I certainly have just as much as any other of the insignificant bacteria called human beings. I have thus been content to observe the phenomena about me with something like objective interest, and to feel a certain tranquillity which comes from perfect acceptance of my place as an inconsequential atom. In ceasing to care about most things, I have likewise ceased to suffer in many ways. There is a real restfulness in the scientific conviction that nothing matters very much; that the only legitimate aim of humanity is to minimise acute suffering for the majority, and to derive whatever satisfaction is derivable from the exercise of the mind in the pursuit of truth (from Letter to Reinhardt Kleiner  (14 September 1919), in Selected Letters I, 1911-1924 edited by August Derleth and Donald Wandrei, pp. 86-87).

I’ve thought about this quite a lot over the last few years and am gradually finding myself more and more in agreement with Lovecraft. I would say further that that one of the few things that make life bearable is the futility of existence. Futility is very reassuring. If all the shit that happens in this world were designed to serve some higher purpose then that really would be terrifying. And even more reassuring than its futility is the knowledge that we will soon return to dust and be quickly forgotten.

Here’s  Weinberg again, from the same interview quoted above:

…if there is no point in the universe that we discover by the methods of science, there is a point that we can give the universe by the way we live, by loving each other, by discovering things about nature, by creating works of art. And that — in a way, although we are not the stars in a cosmic drama, if the only drama we’re starring in is one that we are making up as we go along, it is not entirely ignoble that faced with this unloving, impersonal universe we make a little island of warmth and love and science and art for ourselves. That’s not an entirely despicable role for us to play.

Inspired by this, I’m going to make a point of existence not by doing science nor creating a work of art, but by making a nice cup of tea.

## Cosmology Caption Competition

Posted in The Universe and Stuff, Biographical with tags , , on July 3, 2017 by telescoper

I’ve been a bit busy today so in lieu of a proper post I thought I’d try for a bit of audience participation. In one of the discussion sessions at last week’s workshop on Isotropic Random Fields in Astrophysics I was called upon to deliver a very short impromptu talk on cosmological applications of Burger’s Burgers Equation, during the course of which the following photograph was taken. Appropriate (or inappropriate) suggestions for captions, please, through the comments box!