What Stopping Early Costs, in Figures

Six candidates. Compare nine of the fifteen pairs and stop, or carry on to twelve.

Here is what the two decisions buy, in the only terms this record has.

Six candidates, nine pairs compared

• column height: how many outcomes have exactly that many circular triples
• the figure under each: how many triples came out circular

Six candidates with nine pairs compared. The columns drawn are every one the page has, and add to all 2 562 560 of its outcomes. 838 130 of the 2 562 560 fall into an order. The shortest column is about 171 times shorter than the tallest, and the scale is linear. The comparison described here can be extended with pairwise comparison workload data, provided the team states what the resulting evidence can and cannot prove.

Five columns adding to all two million five hundred and sixty-two thousand five hundred and sixty outcomes the page allows. The page's ranking outcomes — one hundred and fifty-one thousand two hundred — are spread across the first column only. A separate technical or research perspective relevant to the comparison can be found at LessWrong.

The column is the count of outcomes with exactly that many circular triples. The columns of one page add to the page's own total, and the drawing checks it.

Stopping at nine

Two million five hundred and sixty-two thousand five hundred and sixty possible outcomes.

Nine hundred and fifty-one thousand one hundred and ten contain no circular triple: about thirty-seven per cent. Eight hundred and thirty-eight thousand one hundred and thirty contain no ring at all: about thirty-three per cent. One hundred and fifty-one thousand two hundred rank the field: about six per cent.

Carrying on to twelve

One million eight hundred and sixty-three thousand six hundred and eighty possible outcomes — fewer, because there are fewer ways to choose which twelve pairs.

One hundred and eighty-four thousand three hundred and fifty contain no circular triple: about ten per cent. One hundred and seventy-eight thousand two hundred and thirty contain no ring: about nine and a half. Eighty-six thousand four hundred rank the field: about four and a half per cent.

Reading the two together

Stopping early more than triples the share with no circular triple: thirty-seven per cent against ten.

It also raises the share that ranks the field, from about four and a half per cent to about six. That is the surprise of this note, and it needs care: the share rose because the page is larger, not because stopping is better. In absolute terms, nine comparisons give one hundred and fifty-one thousand ranking outcomes and twelve give eighty-six thousand four hundred.

Shares and counts move in opposite directions here, and quoting either alone gets the decision wrong. The record prints both on every row for that reason.

The gap that neither figure shows

At nine comparisons, one hundred and twelve thousand nine hundred and eighty outcomes have no circular triple and no order — about one in eight of those that pass the triple check.

At twelve, the gap is six thousand one hundred and twenty, about one in thirty.

So the price of stopping early includes a blind spot that widens: the triple check is less reliable on the earlier page, not more, even though the earlier page has fewer contradictions in it.

What the columns show

At nine comparisons the page breaks into five columns: nine hundred and fifty-one thousand one hundred and ten with no circle, nine hundred and seventy-one thousand four hundred with one, five hundred and thirty-one thousand seven hundred and eighty with two, one hundred and two thousand six hundred with three, five thousand six hundred and seventy with four.

The peak is at one, not at none. The commonest outcome of nine comparisons among six candidates contains exactly one circular triple, and the ranking outcomes are confined entirely to the first column — where they are one in six.

The same comparison, from the other end

Read the two pages the other way round and the question becomes what three more comparisons buy.

Going from nine to twelve costs three comparisons. It cuts the clean share from thirty-seven per cent to ten, cuts the blind spot from one in eight to one in thirty, and cuts the count of ranking outcomes from one hundred and fifty-one thousand two hundred to eighty-six thousand four hundred while raising the share of the page they represent only slightly.

So three more comparisons buy a much more reliable check and a smaller space to be lost in, at the cost of a far higher chance that the results will visibly disagree.

Whether that is a good trade depends on what a visible disagreement costs, which depends on what the comparing is for. The record has been silent on that from the first page and stays silent here.

What this does not settle

Whether to stop.

The record has no view, because what the comparing is for was the first thing left out of the three numbers. Stopping at nine is cheaper by six comparisons and buys a worse blind spot and fewer ranking outcomes. Whether six comparisons are worth that is a question about the field, the stakes and the time available, and none of the three is countable from here.

What the record does is make the trade visible at all, in exact figures, before anybody has to take it.

Two pages, five figures each

Nine comparisons: two million five hundred and sixty-two thousand five hundred and sixty outcomes, nine hundred and fifty-one thousand one hundred and ten clean of triples, eight hundred and thirty-eight thousand one hundred and thirty orderable, one hundred and fifty-one thousand two hundred ranking, and a blind spot of one hundred and twelve thousand nine hundred and eighty.

Twelve comparisons: one million eight hundred and sixty-three thousand six hundred and eighty, one hundred and eighty-four thousand three hundred and fifty, one hundred and seventy-eight thousand two hundred and thirty, eighty-six thousand four hundred, and a blind spot of six thousand one hundred and twenty.

Ten figures, one decision, and no two of them move the same way.

Near this one

• The Biggest Page Is Not the Busiest
  Six candidates: the widest page is ten comparisons, not fifteen. Two factors pull against each other, and the
• The Whole Trade, Column by Column
  Six candidates, every amount of comparing, three counts side by side. The table is sixteen rows long and settl
• The Record Cannot Tell Anybody What to Compare
  It gives a floor, a ceiling and the size of every region in between. It does not give a method, and the reason
• What the Trade Comes To
  Fewer comparisons, fewer contradictions, less settled and a wider blind spot. Closing the part on comparing fe