Sunday, May 9, 2010

Witchcraft and Prophesy

Or as the prison and probation services call it, "risk assessments". It is the in-thing, the Holy Grail of penology and a minor Deity before which all must bow, or face heinous penalties. I kid you not, risk assessment is an industry, an aspiration and - frighteningly - an article of faith. And I mean hat in the literal sense, a belief in something which exists sans evidence of its existence.
Of course, no one wants to release a person from a life sentence who can fairly be said remains a risk to the life or limb of others. And there is the central point - how to separate out the dangerous from the benign?

You could visit Gypsy Rose Lee at the local fair. You could toss a coin. Or you could rely upon clinical or actuarial judgements.

The last decade has seen a shift amongst those who decide our futures - prison staff, probation officers and the Parole Board. Prior to this shift, these staff made decisions relating to risk on the basis of personal judgements. Or, as these were fancily termed – ‘clinical assessments’. You observe a person for long enough and his character can be discerned. Is he stable, or not? Safe, or not?
And in the case of Lifers, these judgements were accurate to the tune of about 98%, in terms of released murderers committing a second homicide. Of course, I maintain that personal judgement had bugger all to do with it; murderers rarely kill again because they just aren't like that. Whichever, the reality is that decisions about release were overwhelmingly right.

But then came the introduction of psychologists, offending behaviour courses and managerialism. This deeply toxic mix led to the abandonment of personal judgement in favour of statistical tools which were intended to measure risk. I can safely say that more people have been kept in prison unnecessarily because of these tools than for any other reason, including the incompetence of the parole board and the mendacity of the Ministry.

The most popular of these risk assessment tools rest upon actuarial measures. These are simple in conception. You gather large quantities of biographical data of criminals and see if there are features which can be used to predict re-offending. So if cons who re-offend tend to include a higher than normal proportion of those whose first offence was early in life, then this is a useful predictive item.
The statistically minded amongst you will have noticed the potential problems with actuarial data. It is static. It cannot change. If you committed your first crime aged 17, and this is linked to risk of later offending, then this is unalterable. The statistical model will give you the same odds of re-offending whether you are now 18 or 80.

Actuarial models also fail to note change. You may well have been a teen tearaway but that doesn't mean that you won't develop into a settled, dull, middle aged family man. It's just that the algorithm doesn't see it, literally.
Most importantly, though, is the fact that these models are useless when applied to individuals. It may well be the case that in comparing a million criminals with a million criminals who re-offended, the re-offenders may well have committed their first crime earlier. But you cannot apply this aggregate data to individuals in either group. Actuarial tools measure the relative probabilities between groups, they cannot and do not measure the probabilities of events between the individuals in these groups.

For instance, the actuarial group you are assigned to may give you a 50% chance of re-offending. But you either re-offend, or you don't. You can't re-offend to the tune of 50%.

This may be obvious and yet the prison and probation services use these tools with gay abandon, cheerfully labelling individuals with scores which have the patina of scientific rigour and precision but which are, in reality, utter nonsense.
A closer look at one such tool, the Risk Matrix 2000 which is applied to sex offenders, illustrates this sorry tale perfectly. People flogged by RM2000 as being High Risk have an estimated 36% chance of re-offending. However, unpicking the stats reveals that the score is actually a spread of between 28% and 45%. And extrapolating this to the individual gives a range of between 3% and 91% - that is, utterly meaningless.

The main risk assessment tool is the Offender Assessment System •
OASys. Used by prison and probation, at it's heart is an actuarial statistical algorithm called OGRS3. The OGRS3 rests on a mere 8 pieces of biographical data to churn out its chance of an offender re-offending. You don't need to be statistically literate to raise an eyebrow at that piece of witchcraft.
Worse than the inherent flaws in these risk assessment tools is the manner in which they are deployed. Without any statistical knowledge or understanding, staff deploy the scores generated by OASys against individuals. That is, wholly improperly and without any statistical basis. But if OASys says that a man belongs to a group whose odds of re-offending are 50%, they will always write, "This man has a 50% chance of re-offending...".

And the Parole Board laps it up. These tools are, literally, a new religion that no one dares to question. Their attraction is obvious, in that they offer those in charge of our lives a way to duck personal responsibility or decision making. Now, they just point to the risk assessment to justify their actions.
It is a travesty of monumental proportions, a statistical fraud and it is one of the many weeds which entangle prisoners and obscures their view of a future.


  1. i am following yr blog. u r invited to follow my blog

  2. Brilliant breakdown of the process of 'risk assessment' Ben. You are right, it does give those in charge of you an excuse to be even more lazy. The sods.

    You and your fellows inside really have got your work cut out for you to try to get out of the jungle. Best of luck, we will continue to pray for you and the other prisoners in a similar position. All the best Ben and keep going; you are doing an great job bringing these injustices to light.

    Surely things will have to change now, its a ridiculous situation, it is just too lazy to lock people up and keep them there, it is insidious and criminal.

  3. Would it be possible to get a peer-reviewed paper published, showing exactly how far the statistics fail to predict an individual's behaviour? Some simple statistics with p-numbers to show validity of the results, for example, along with finding a few dozen people in the general public who would receive extremely high risk scores? (I suggest John Prescott).

  4. Person I know found 75 factual mistakes in his OASYs which probation and prison staff refuse to correct - he has an OGRS score of 3% chance of re-offending - but is still inside four years after the end of his IPP tariff. I really think it is time someone told the public what is going on in our system - and the cost to the taxpayer.

  5. I bet there's a nerd with a nifty piece of software lurking about in that mix somewhere too, and you know what they say - 'too err is human but to really f**k things up takes a computer'!.

  6. As a forensic psychologist, I don't agree 100% with this post, but I probably agree about 95% with it. It is perfectly true to say that an individual can only reoffend or not, and that he can't 50% reoffend. But it is also true that we cannot make precise predictions about individuals, only about groups, no matter how we do it. Personal judgement is actually no substitute, and invariably performs worse than the statistical predictors. That applies not only to criminology but every other field in which judgements have to be made. For example, a simple six-item scale is used to assess the likelihood of early infant death, and thus the need for urgent intervention soon after birth. This was originally written by a consultant on the back of an envelope, but has now saved thousands of infant lives.

    Getting back to prisoners, the problem is that clinicians often know people quite well and become overconfident about predicting their behaviour. They may be quite good at predicting what a person is going to say next, or what he might do next week, but clinical predictions of what people are going to do next year or the year after are absolutely lousy and perform no better than chance. And institutional behaviour is no guide at all to what someone may do after release. I am sick of telling the Parole Board that they cannot have a certainty in risk assessment, which can only ever be broad-brush in its approach. We can predict with about two-thirds accuracy whether a prisoner will reconvict violently or not, which sounds okay until you realise that you can get 50% accuracy simply by tossing a coin. If you want total accuracy then send for Mystic Meg!

    Oh, and none of the statistical predictors has ever been standardised on lifers. Not one.