What the Power Spectrum misses

Just taking a short break from work I chatted over coffee to one of the students here at the Niels Bohr Institute about various things to do with the analysis of signals in the Fourier domain (as you do). That discussion reminded me of this rather old post (from 2009) which I thought might be worth a second airing (after a bit of editing). The discussion is all based on past cosmological data (from WMAP) rather than the most recent (from Planck), but that doesn’t change anything qualitatively. So here you are.

The picture above shows the all-sky map of fluctuations in the temperature of the cosmic microwave background across the sky as revealed by the Wilkinson Microwave Anisotropy Probe, known to its friends as WMAP.

I spent many long hours fiddling with the data coming from the WMAP experiment, partly because I’ve never quite got over the fact that such wonderful data actually exists. When I started my doctorate in 1985 the whole field of CMB analysis was so much pie in the sky, as no experiments had yet been performed with the sensitivity to reveal the structures we now see. This is because they are very faint and easily buried in noise. The fluctuations in temperature from pixel to pixel across the sky are of order one part in a hundred thousand of the mean temperature (i.e. about 30 microKelvin on a background temperature of about 3 Kelvin). That’s smoother than the surface of a billiard ball. That’s why it took such a long time to make the map shown above, and why it is such a triumphant piece of science.

I blogged a while ago about the idea that the structure we see in this map was produced by sound waves reverberating around the early Universe. The techniques cosmologists use to analyse this sound are similar to those used in branches of acoustics except that we only see things in projection on the celestial sphere which requires a bit of special consideration.

One of the things that sticks in my brain from my undergraduate years is being told that if you don’t know what you’re doing as a physicist you should start by making a Fourier transform of everything. This approach breaks down the phenomenon being studied into a set of  plane waves with different wavelengths corresponding to analysing the different tones present in a complicated sound.

It’s often very good advice to do such a decomposition for one-dimensional time series or fluctuation fields in three-dimensional Cartesian space, even you do know what you’re doing, but it doesn’t work with a sphere because plane waves don’t fit properly on a curved surface. Fortunately, however, there is a tried-and-tested alternative involving spherical harmonics rather than plane waves.

Spherical harmonics are quite complicated beasts mathematically but they have pretty similar properties to Fourier harmonics in many respects. In particular they are represented as complex numbers having real and imaginary parts or, equivalently, an amplitude and a phase (usually called the argument by mathematicians),

$Z=X+iY = R \exp(i\phi)$

This latter representation is the most useful one for CMB fluctuations because the simplest versions of inflationary theory predict that the phases φ of each of the spherical harmonic modes should be randomly distributed. What this really means is that there is no information content in their distribution so that the harmonic modes are in a state of maximum statistical disorder or entropy. This property also guarantees that the distribution of fluctuations over the sky should have a Gaussian distribution.

If you accept that the fluctuations are Gaussian then only the amplitudes of the spherical harmonic coefficients are useful. Indeed, their statistical properties can be specified entirely by the variance of these amplitudes as a function of mode frequency. This pre-eminently important function is called the power-spectrum of the fluctuations, and it is shown here for the WMAP data:

Although the units on the axes are a bit strange it doesn”t require too much imagination to interpret this in terms of a sound spectrum. There is a characteristic tone (at the position of the big peak) plus a couple of overtones (the bumps at higher frequencies). However these features are not sharp so the overall sound is not at all musical.

If the Gaussian assumption is correct then the power-spectrum contains all the useful statistical information to be gleaned from the CMB sky, which is why so much emphasis has been placed on extracting it accurately from the data.

Conversely, though, the power spectrum is completely insensitive to any information in the distribution of spherical harmonic phases. If something beyond the standard model made the Universe non-Gaussian it would affect the phases of the harmonic modes in a way that would make them non-random.

However,I will now show you how important phase information could actually be, if only we could find a good way of exploiting it. Let’s start with a map of the Earth, with the colour representing height of the surface above mean sea level:

You can see the major mountain ranges (Andes, Himalayas) quite clearly as red in this picture and note how high Antarctica is…that’s one of the reasons so much astronomy is done there.

Now, using the same colour scale we have the WMAP data again (in Galactic coordinates).

The virture of this representation of the map is that it shows how smooth the microwave sky is compared to the surface of the Earth. Note also that you can see a bit of crud in the plane of the Milky Way that serves as a reminder of the difficulty of cleaning the foregrounds out.

Clearly these two maps have completely different power spectra. The Earth is dominated by large features made from long-wavelength modes whereas the CMB sky has relatively more small-scale fuzz.

Now I’m going to play with these maps in the following rather peculiar way. First, I make a spherical harmonic transform of each of them. This gives me two sets of complex numbers, one for the Earth and one for WMAP. Following the usual fashion, I think of these as two sets of amplitudes and two sets of phases. Note that the spherical harmonic transformation preserves all the information in the sky maps, it’s just a different representation.

Now what I do is swap the amplitudes and phases for the two maps. First, I take the amplitudes of WMAP and put them with the phases for the Earth. That gives me the spherical harmonic representation of a new data set which I can reveal by doing an inverse spherical transform:

This map has exactly the same amplitudes for each mode as the WMAP data and therefore possesses an identical power spectrum to that shown above. Clearly, though, this particular CMB sky is not compatible with the standard cosmological model! Notice that all the strongly localised features such as coastlines appear by virtue of information contained in the phases but absent from the power-spectrum.

To understand this think how sharp features appear in a Fourier transform. A sharp spike at a specific location actually produces a broad spectrum of Fourier modes with different frequencies. These modes have to add in coherently at the location of the spike and cancel out everywhere else, so their phases are strongly correlated. A sea of white noise also has a flat power spectrum but has random phases. The key difference between these two configurations is not revealed by their spectra but by their phases.

Fortunately there is nothing quite as wacky as a picture of the Earth in the real data, but it makes the point that there are more things in Heaven and Earth than can be described in terms of the power spectrum!

Finally, perhaps in your mind’s eye you might consider what it might look lie to do the reverse experiment: recombine the phases of WMAP with the amplitudes of the Earth.

If the WMAP data are actually Gaussian, then this map is a sort of random-phase realisation of the Earth’s power spectrum. Alternatively you can see that it is the result of running a kind of weird low-pass filter over the WMAP fluctuations. The only striking things it reveals are (i) a big blue hole associated with foreground contamination, (ii) a suspicious excess of red in the galactic plane owing to the same problem, and (iiI) a strong North-South asymmetry arising from the presence of Antarctica.

There’s no great scientific result here, just a proof that spherical harmonic phases are potentially interesting because of the information they contain about strongly localised features

PS. These pictures were made by a former PhD student of mine, Patrick Dineen, who has since quit astrophysics  to work in the financial sector for Winton Capital, which has over the years recruited a number of astronomy and cosmology graduates and also sponsors a Royal Astronomical Society prize. That shows that the skills and knowledge obtained in the seemingly obscure field of cosmological data analysis have applications elsewhere!

23 Responses to “What the Power Spectrum misses”

1. This is a very nice piece. Years ago, back in the COBE days, I used to show an elevation map of the Earth just like the one you’re using, and the result of randomizing the phases to get something like the last picture you show. I never thought of doing the opposite — keeping the Earth-map’s phases but randomizing the amplitude. That’s very nice.

There is a technical errorsin this piece, although it doesn’t affect the important points you’re making. It’s not true that randomness of phases “guarantees that the distribution of fluctuations over the sky should have a Gaussian distribution.” A statistically isotropic Gaussian random process is guaranteed to have random phases, but the converse is not true.

For Gaussianity, you need all of the spherical harmonic coefficients to be drawn from a multivariate normal distribution. So, for instance, if you chose all of the coefficients to have the same absolute value, but to have random phases, that would not be a Gaussian random process.

• telescoper Says:

By the central limit theorem (assuming it applies) if you add a lot of harmonic modes with random phases the resulting 1-pt PDF will be Gaussian. I usually call this a weakly Gaussian process.

A strictly Gaussian process is one defined as you say.

• That’s true, so with the extra assumption that each spherical harmonic coefficient is produced by adding together “a lot” of independent terms, you do get (approximately) Gaussian statistics. But that extra assumption — that many independent things are being added together — is quite different from the random-phase assumption, which is why I object to the claim that random phases imply Gaussianity.

In fact, the central limit theorem applies even if the phases aren’t random. If there are many independent processes that get added together to make our CMB sky, and if those processes have nonrandom phases (say, the mean of each contribution to each a_{lm} is a positive real number, so phase = 0 is preferred), the central limit theorem will still say that the final result is Gaussian. It just won’t be statistically isotropic.

The statement that random phases imply Gaussianity comes up quite a lot when cosmologists talk about Gaussian processes. In fact, random phases and Gaussianity are pretty much logically independent of each other. It’s closer to the truth to say that random phases imply statistical isotropy, which is quite a different thing.

• telescoper Says:

If a spectral representation exists, which it has to for the phases to be defined, then the field at a given location *is* the sum over all the modes which will be independent if the phases are random.

• Randomness of phases does not imply independence. To see this, consider the following admittedly silly random process:

1. Generate a Gaussian random field in the usual way.
2. Flip a coin, and multiply all the values in this map by 1000 if it comes up heads.

This process produces a map with random phases (after all, the phases after step 2 are the same as before), but the coefficients of the various harmonic modes are not independent, and the random process is not even approximately Gaussian.

As I’ve admitted, this example is silly, but it does show that there is no necessary logical connection between random phases and Gaussianity.

The reason this example yields non-Gaussianity is also the reason it’s silly: it involves strong (and implausible) dependence among the various modes. If the various modes are independent, then I agree that you get (approximate) Gaussianity.

But here’s the point: That last statement is true regardless of whether the phases are random. If you build your map out of many independent random variables, in such a way that many terms contribute significantly to each point in the map, then the central limit theorem will give you approximate Gaussianity, whether or not the phases are random.

So there is, as far as I am aware, precisely no logical connection between randomness of phases and Gaussianity.

• telescoper Says:

I don’t understand your example: if you introduce a set of spikes’ each of these produces a harmonic-space signature each of which has highly correlated phases. It’s not true that in your prescription the phases are unchanged, unless I’ve misunderstood something..

2. Anton Garrett Says:

That shows that the skills and knowledge obtained in the seemingly obscure field of cosmological data analysis have applications elsewhere!

I’d criticise nobody for seeking to provide better for their family, and good for Winton in sponsoring science, but that not necessarily so. Some of us think that quantitative financial prediction is based on a wrong understanding of money and a false analogy with physics. In physics, quantitative variables form a closed system. In finance, quantitative variables are influenced by, and influence, qualitative variables. This is why you can’t ‘time the markets’, ie predict when a peak or trough in the market sector or share in question has been reached. Without that, you might do well in the short term but you have no way of knowing when the short term transitions into the long term, and it’s a matter of sheer luck when to get in and out. In 18th century categories, study of the world of money is a ‘moral science’ rather than a ‘natural science’.

• I doubt that so many physicists work in the finance industry because of a false analogy with physics. Rather, the industry often looks for people with hands-on programming skills.

• Anton Garrett Says:

Are they employed exclusively as programmers or (also?) as people who come up with the ideas to be programmed? “Quantitative finance” is now regarded as an academic area. Absurdly.

• It’s certainly the case that one needs some programming knowledge, especially in the area of algorithmic trading. I’m sure you’ll find physicists at all levels in the finance industry (and elsewhere—Oskar Lafontaine and Angela Merkel are physicists, for example).

Sure, no-one can predict the market exactly. (When I see market experts on television, I always think that if they were really so good, they wouldn’t have to be paid for being on television at 5 in the morning or whatever.) On the other hand, while monkeys might make better choices than some portfolio managers (I think that there was actually an experiment to test this), some things are quantitative and can be attacked (in practice only) via a programming solution, for example rapidly buying and selling at several exchanges, making a profit from the small price differences. Also, the speed of light is a limiting factor; in some cases, microseconds can provide a competitive edge. There is a lot of work in technical optimization.

There are also technical-support jobs; being a sysadmin in the business world differs from being a sysadmin in academia primarily in better pay in the former case. Most people who work in the finance industry are dependent on technology but don’t have a technical background themselves. Then there is the whole area of big data, especially in relation to regulations and so on (archive data for 10 years, in case someone needs it for an investigation, etc). With almost all trading electronic these days, a huge amount of data is produced.

Of course, whether you think that this is meaningful is another question, but who pays the piper calls the tune. 😐 I don’t know why Patrick left academia, but not everyone who leaves does so for lack of talent; some need, or at least want, more job security. (They usually earn much more as well, but this is less important than job security, at least to people who have spent some time in academia (as opposed to those who want to get rich as fast as possible).)

• Anton Garrett Says:

Smart pattern-recognition algorithms deployed in markets can harvest small amounts of money in small amounts of time, but they can’t know when a big crash is coming for which reason I’d expect their longterm time average to show that they are useless.

Investment houses make a lot of money by taking a percentage of the client’s cash. It’s a contract freely entered into and I expect most investment houses genuinely believe in their methods, ie they are in good faith, but that doesn’t mean they are any good. Most don’t outperform the market average and those that do over one business cycle often don’t over the next, ie they were just lucky.

• Dave Glyer Says:

This comment looks a bit like a situation where a little bit of knowledge is dangerous, or at least makes one think that they know more than they really do; although it also makes good points. Quantitative skills and reasoning are very useful in Finance and it does not drive primarily from trying to use physics-type of models as the basis of analysis. Quant analysis is not primarily about ‘getting in and out’.

In general, I doubt the PhD-type physics people are primarily programmers, they are more on the ‘quant’ side doing modeling and extensive data analysis. Analyzing risk and the interaction of risk components across assets and asset classes on various time domains is one key area for highly quantitative people in the field. One of the short comings of the quants is that they are not as strong on intuition as they are with their other tools and thus they tend not to build the ability to apply intuition to the problems that they face.

I think that this was a problem for the guys at Long-Term Capital Management (Merton and Scholes, Nobels in 1997) especially because great early returns pushed them more and more into risky assets and then the hedge fund imploded in four months with the Asian and Russian crises, going bankrupt. Hard to model that the modest and even negative correlations under normal conditions get more and more positively correlated under extreme conditions. Most straight models can be very misleading for situations where there is not much data, and quants can get badly burned. [note: my background in economics; I am a neophyte in physics but I find it more interesting in retirement.]

3. V nice article Peter, most helpful. If I may ask a dumb question: are there other reasons (besides inflation) that we might expect a Gaussian distribution? Given the primordial nature of the radiation, isn’t Gaussianity to be expected in simple models, with the same reasoning that one expects the CMB spectrum to be close to that of a black body?

• telescoper Says:

Cormac,

Good question. People were talking about Gaussian primordial fluctuations long before anyone ever thought of inflation, as that is the generic result of linear physical processes. Inflation yields a specific mechanism for generating Gaussian perturbations (from quantum-mechanical processes) but that’s not to say such perturbations couldn’t have arisen in another way.

• Although inflation is more a paradigm than a theory, and hence specific predictions are a bit tricky, one relatively robust one was a spectral index of approximately 1, but slightly less. This was a pretty generic prediction, long before data were anywhere good enough to measure it. The measured value of slightly less than one significantly increased my confidence that inflation or something very similar probably happened.

A scale-invariant spectrum is pretty generic; this goes back to Harrison (1970) (the same Harrison of “who has written a very good cosmology textbook?” fame) and Zeldovich (1972). That inflation produces a nearly scale-invariant spectrum is thus no big deal; the interesting thing is the slight tilt.

• telescoper Says:

Yes, the early papers by Harrison and Zel’dovich were well before inflation and the arguments for n=1 are made on general grounds. What is interesting about the inflation models is the connection between departures from n=1 and gravitational wave perturbation amplitude (at least within slow-roll inflation).

• Shantanu Says:

Philip, I believe Starobinsky has a model of inflation with ns exactly equal to 1. So again ns different from 1 is also not a smoking gun signature of inflation

• There are also inflation models with significant curvature at our epoch. I think the consensus, though, is that they are contrived. I’m no expert on this, but my impression is that n slightly less than 1 is a prediction of many models.

• telescoper Says:

That’s what you get in single-field slow-roll inflation…

4. Many thanks for that reply, Peter. it does clarify things.
My understanding is that the assumption of Gaussianity in the CMB is reasonable, in the sense that it would be anyone’s first guess (with or without inflation) and no counter evidence has emerged from lensing etc. Thus, arguing from naturalness, it’s likely that the amplitudes of the peaks should indeed be interpreted the way they are, is that right?

5. Wow, that paper and the references in it are quite a treat for the historian!

6. Dr.William Joseph Bray Says:

How did you derive the power, amplitude, and phase for the earth map? I’m curious about the Chaotic (but not truly random) attractor in this overlay of the power and phase of Earth and Sky.
I’m guessing you used altitude for amplitude, but for power (angular) and phase, I’m at a loss (for the Earth map).