Probability and the law

What constitutes a reasonable doubt? The Mail on Sunday and its polemical columnist Peter Hitchens have raised the question again, deploring the recent guidance that encourages judges to frame the issue for juries as ‘are you sure?’ As various commentators have pointed out, this interpretation appeared in the 2010 Crown Court Compendium but has a longer history. In 1949, Lord Chief Justice Goddard cited ‘the direction which I myself constantly give to juries’. He explained ‘They must be fully satisfied of the guilt of the accused person and should not find a verdict against him unless they feel sure’.

Like most people whose training is in economics and statistics, when we encountered the legal standards of proof of ‘on the balance of probabilities’ and ‘beyond reasonable doubt’ we were confident that we knew what they meant. ‘Balance of probabilities’ refers to a probability in excess of 0.5; beyond reasonable doubt is much higher – .95 or .99. And then we talked to lawyers and understood that this was not what those terms meant. But we found that these lawyers were much better at stating what the terms did not mean than explaining what they did mean.

In our recent book Radical Uncertainty we criticise the inappropriate extension of probabilistic language and reasoning. It is common today to use the words likelihood, confidence and probability as if they were interchangeable. But they are not interchangeable. It is likely that Philadelphia is the capital of Pennsylvania (although it is not). When I make such a statement, I am applying the general rule that the capital of a state is often its largest city. I am confident that Harrisburg is the capital of Pennsylvania – I looked it up in an encyclopaedia. And the statement; ‘the probability that Philadelphia is the capital of Pennsylvania is 0.7’ is absurd – either it is or it isn’t. Likelihood is the product of our general knowledge of the world. The confidence with which someone holds a belief is a poor guide to its likely truth. And probabilities are numbers derived from some known or potentially knowable statistical distribution. Because probabilities are quantitative, and probabilistic reasoning is a well-developed mathematical field, people think they are being ‘scientific’ when they talk in probabilistic language.

Some lawyers talk of ‘Blackstone’s ratio’; the great jurist wrote ‘better that ten guilty persons escape than that one innocent person suffer’. And we would all agree that in judging guilt it is more important to minimise the statistician’s type II error – accepting the hypothesis when it is false – at the expense of the type I error – rejecting the hypothesis when it is true. But why should the ratio be ten to one?  Rather than five to one as proposed by Sir Matthew Hale, or one hundred to one, as favoured by Benjamin Franklin? The French scholar Condorcet concluded that the probability of an innocent man being convicted should be exactly one in 144,768 and that this could be achieved if guilt were determined by a panel of thirty judges of whom twenty-three must vote in favour. (Condorcet himself committed suicide to escape the revolutionary guillotine.)

These calculations are absurd. As the Oxford philosopher Jonathan Cohen explained ‘The advancement of truth in the long run is not necessarily the same thing as the dispensation of justice in each individual case. It bears hard on an individual…if he has to lose his own particular suit in order to maintain a stochastic probability for the system as a whole.’ US legal scholar Laurence Tribe makes a similar point. ‘Tolerating a system in which perhaps one innocent man in a hundred is erroneously convicted despite each jury’s attempt to make as few mistakes as possible is vastly different from instructing a jury to aim at a 1% rate of mistaken convictions.’

Following the misuse of statistical evidence in some high-profile cases, the Royal Statistical Society, in consultation with eminent lawyers, prepared a report which advocates a technique known as Bayesian nets. But this analysis has – rightly – been largely ignored. Justice is necessarily individual, not statistical.

In fact we think that statisticians have much to learn from the law, whose processes are among the oldest techniques we have for making decisions in the face of unavoidable uncertainty. More than a century ago, the American pragmatist philosopher Charles Sanders Pierce distinguished three styles of scientific reasoning. Deductive reasoning is the method of mathematics and theoretical physics – reaching conclusions derived logically from clearly stated premises. Inductive reasoning is more appropriate for subjects such as medicine and chemistry – formulating theories based on repeated observation and testing them through experiment and further observation.

But law, like history, uses our human penchant for narrative – we told stories for millennia before anyone developed the mathematics of probability, and made use of these stories to make decisions and illuminate our experience. These subjects rely more on abductive reasoning, or ‘inference to best explanation’: taking a disparate set of facts and arranging them into some comprehensible account. Answering the question posed by the German historiographer Leopold von Ranke – ‘wie es gewesen ist’, which we prefer to translate into the English of the modern world as ‘what is going on here?’ – a question which seems banal but is the essential preliminary to describing the Napoleonic Wars, managing the global financial crisis, or determining policy for Covid-19.

The burden of proof imposes on the claimant or prosecutor the obligation to present an account relevant to the case which is coherent and consistent – logically structured and supported by the evidence – and which is likely, i.e., consistent with our general knowledge of the world. (Courts emphasise that judgments must be based on the evidence alone, but do not really mean this. Justice deplores the employment of what statisticians call ‘priors’ – ‘he must be guilty or he wouldn’t be here’, or ‘I never trust someone with eyes so far apart’. But juries are valuable for their application of common sense – they know that people do not usually impale themselves voluntarily on someone else’s pointed knife.)

Balance of probabilities (or a preponderance of the evidence, which lawyers generally think means much the same thing, although most statisticians would interpret them differently) implies not only that the account offered is a good one but that it is better than any alternative account. Beyond reasonable doubt means that the prosecution account is sufficiently compelling that there seems no other possible account which meets the criteria of coherence and consistency.

This does not mean that probability has no place in the courts, although we are shocked by its misuse, particularly in the tragic case of Sally Clark, who suffered severe psychiatric problems and died from alcohol poisoning after being wrongfully convicted following a trial in which grossly misleading statistical evidence was presented. Expert witness Sir Roy Meadow made an error so common that it is sometimes called the ‘prosecutor’s fallacy’ – the failure to recognise that the events which give rise to a criminal charge are, fortunately, inherently out of the ordinary. The relevant question in that infamous case was not whether the occurrence of two successive cot deaths in a single household was unlikely – of course it was – but whether, given that Mrs Clark’s two children had both died in unexplained circumstances, such an explanation was sufficiently probable to constitute a reasonable doubt.

Probabilistic reasoning was also misused in the notorious OJ Simpson trial. The flamboyant defence pointed out that since six people in the Los Angeles area alone had DNA forensically indistinguishable from Simpson’s, the presence of that DNA at the crime scene was not significant. But here the relevant probabilistic question was ‘what is the probability that Nicole was murdered by someone who, although not her husband, had the same DNA as her husband?’ – a contingency so improbable as not to constitute a reasonable doubt. That does not form a basis for conviction, but is sufficient to exclude one possible source of reasonable doubt.

We share Hitchens’ concern. Sureness is a function of personality as much as evidence – as in the old aphorism, attributed to Lord Melbourne describing Thomas Macaulay, ‘I wish I was as sure of anything as that man is of everything.’  The reflective juror will always entertain some doubt. But not, if he or she is to convict, a reasonable doubt.

By John Kay and Mervyn King

John Kay, CBE, FRSE, FBA, FAcSS was the first dean of Oxford’s Said Business School and has held chairs at London Business School, the University of Oxford, and the London School of Economics. He has been a fellow of St John’s College, Oxford, since 1970

Mervyn King, former Governor of the Bank of England

Radical Uncertainty by John Kay and Mervyn King was published by The Bridge Street Press on  the 5th March 2020

Share this post