Skip to Content

Coffee House

Do ‘tutor-proof’ grammar school tests exist?

8 September 2016

5:57 PM

8 September 2016

5:57 PM

As part of its plan to expand the number of grammar schools, the Government has proposed making ‘tutor-proof’ tests. Unfortunately, this is more difficult to accomplish than Theresa May imagines and even if it were realistic, it might not solve the problem of under-representation of poor children in grammar schools.

It’s possible to make this argument thanks to the large amount of scientific literature on the effects of practice and coaching on cognitive test performance. This is based on the popularity of such tests in job and educational selection over the decades and the results are both clear and consistent. Practice and coaching do have positive effects. What’s more, coaching has an effect over and above that of practice alone. These effects are modest in magnitude – equivalent to a gain of three to four IQ points. But they are certainly big enough to cause distorting effects amongst that group of marginal candidates close to the selection cut-off.

These coaching effects are generally greater for more abstract cognitive tasks, such as non-verbal reasoning tests like Raven’s Progressive Matrices, a kind of test where you have to pick a certain shape that completes a pattern. These tests operate according to hidden rules that can be taught. Furthermore, schoolchildren do not typically spend much time in lessons doing matrix reasoning tests, and consequently the pupil who is exposed to this type of test by a tutor has an immediate advantage over a child who may well have never encountered them before.

Potential solutions to this problem are never going to be perfect, but the literature does also provide some suggestions. Testing children on their proficiency on material that all will have covered in school – such as basic arithmetic – will yield a test with less marked coaching effects. A well-constructed test of vocabulary, comprehension, or general knowledge should also be harder to prepare for, provided the test papers aren’t leaked in advance (a surprisingly common problem). However, even the best-designed tests of this kind are culturally loaded and may well unfairly discriminate against immigrant children who have not spent long in the UK.

Perhaps it would be best to move away from paper-and-pencil testing altogether. Modern computerised testing, based on item response theory, gives us the ability to construct highly diverse tests, automatically matched to the ability of the test-taker, from a huge pool of items.

However, even if it was possible to design a perfectly tutor-proof test, this would still not in itself justify grammar schools and selection at 11. The negative effects on those not selected could well outweigh the benefits for those selected, and the bulk of the empirical evidence tends to suggest that this is in fact the case. Politics, though, is rarely just about evidence. My view is that if grammar selection must take place, it would be far better at 14. From behaviour genetic research we know that the effect of family environment on variation in cognitive ability tends to fade dramatically throughout adolescence, and the influence of genes comes increasingly to the foreground. Selection at 14 would do a better job of capturing natural ability than selection at 11.

Lastly, it should be stressed that no system or test can ever equalise disparities between the rich and poor in education. Ability is heritable and smart parents tend to earn more money than the less gifted in the labour market. They don’t just pass their money down to the children, but the genes that allowed them to earn that money in the first place. We cannot ever hope to eliminate the achievement gap, and if inequality is thought undesirable, the best we can hope for is a system that minimises it.

Andrew Sabisky is a freelance writer and independent research worker with a background in educational psychology.

Show comments