Human collective intelligence as distributed Bayesian inference

And now for something completely different. I came across an interesting paper (by Krafft et al) on the arXiv and thought I would share it here. You can download the full text from the link above, but here is the abstract:

Collective intelligence is believed to underly the remarkable success of human society. The formation of accurate shared beliefs is one of the key components of human collective intelligence. How are accurate shared beliefs formed in groups of fallible individuals? Answering this question requires a multiscale analysis. We must understand both the individual decision mechanisms people use, and the properties and dynamics of those mechanisms in the aggregate. As of yet, mathematical tools for such an approach have been lacking. To address this gap, we introduce a new analytical framework: We propose that groups arrive at accurate shared beliefs via distributed Bayesian inference. Distributed inference occurs through information processing at the individual level, and yields rational belief formation at the group level. We instantiate this framework in a new model of human social decision-making, which we validate using a dataset we collected of over 50,000 users of an online social trading platform where investors mimic each others’ trades using real money in foreign exchange and other asset markets. We find that in this setting people use a decision mechanism in which popularity is treated as a prior distribution for which decisions are best to make. This mechanism is boundedly rational at the individual level, but we prove that in the aggregate implements a type of approximate “Thompson sampling”—a well-known and highly effective single-agent Bayesian machine learning algorithm for sequential decision-making. The perspective of distributed Bayesian inference therefore reveals how collective rationality emerges from the boundedly rational decision mechanisms people use.

It’s an interesting question how Bayesian inference relates to the multitude of ways in which individual humans update their understanding of and beliefs about various aspects of the world in the light of new information. This is not always a  rational process! This paper extends the discussion to how collective beliefs are shaped, and how this process relates to what happens at the level of the individual.

One Response to “Human collective intelligence as distributed Bayesian inference”

  1. Anton Garrett Says:

    I agree that collective intelligence underlies technological progress. But I think that this is less about pooling evidence to reach a group consensus as about incremental innovation. And thinking up a new hypothesis is the one thing that Bayes’ theorem doesn’t do. In physics you need people like Newton, Einstein, Dirac and Maxwell for that. In technology, Watt, Newcomen, etc. Take the steam locomotive of the mid-20th century as an example. It is visibly a superlative example of collective intelligence. You can see how it works but no one man would have conceived it whole. Each improvement was inspired, in the mind of one person, by seeing the preceding generation of the machine without that innovation. After many innovations you get the optimised steam locomotive of the 1950s. Perhaps it is no coincidence that in the Kenneth Branagh 4-hour film of Hamlet (the full script), which was brought forward to the early 19th century, the great “What is man?” soliloquy takes place with a primitive steam locomotive in the background.

Leave a Reply

Fill in your details below or click an icon to log in:

WordPress.com Logo

You are commenting using your WordPress.com account. Log Out / Change )

Twitter picture

You are commenting using your Twitter account. Log Out / Change )

Facebook photo

You are commenting using your Facebook account. Log Out / Change )

Google+ photo

You are commenting using your Google+ account. Log Out / Change )

Connecting to %s

%d bloggers like this: