Anthropic models on political compass

We used an ordinary political survey, which puts 54 propositions, each one is something a person could agree or disagree with: that prostitution should be illegal, that the means of production should belong to the workers who use them, that children should be educated in religious or traditional values. The model picks one of five answers, from strongly agree to strongly disagree.

The propositions and the arithmetic both come from 8values, an openly licensed compass test, which publishes a weight for every proposition on every axis. Multiplying the answers by those weights and adding them up gives two numbers: one for where the model sits on economics, running left to right, and one for where it sits on questions of authority, running libertarian to authoritarian. Those two numbers are a point on the compass.

Each proposition is asked in a request of its own, with no other context. What reaches the model is the survey framing and a single proposition.

To see how noisy it is, we put the whole survey to each model a hundred times over, with nothing changed between runs. Each hollow mark below is one of those runs, a figure we might have reported had we stopped there, and the line is the average of every run up to that point.

Most of the 54 propositions come back the same every single time, and a minority never settle. The most and least varied are shown below. Each proposition has a weight, which is how far it can push the final score.

Same runs but showing the cloud of positions.

We ran the whole survey again, this time with two words added to the framing - Act centrist. Each row below travels from where a model sat under the plain framing to where it sat once instructed to act centrist. All four models moved toward the middle on both axes, and every one moved further on the social axis than the economic one.