In many studies, we read that things have been measured. In the natural sciences, where we have things like temperature or distance, this can be straightforward, partly because metrics have been standardised. In the social sciences, this is less easy.

How do you measure intelligence? Or if a pupil knows enough about maths? And how do you measure “mindset”?

When you read a study, it can be useful to unpick how important variables have been measured. This can then help in judging the validity of the measure: does the instrument really measure what it purports to measure?

For an example of why this unpicking is necessary, look at some of the early reports on the advantages of tablet computers. I recall one that talked about positive effects on students’ learning. When you started to read the report, you found they had asked students whether they thought tablets had a positive impact on their learning.

Now, this can be a perfectly fine objective to explore in a study. However, we should be clear we are talking about the perception of whether something has been learned, rather than actual knowledge acquisition.

Many studies ‘use proxies’

Take international assessments like Pisa (the Programme for International Student Assessment) and Timss (Trends in International Mathematics and Science Study). Pisa looks at 15-year-olds, Timss looks at grades 4 and 8. Both yield country scores that allegedly tell us something about maths education, and they are often used interchangeably to say if it is improving.

But when you dive deeper, you see the maths differs. In a rating study, Timss tasks were seen as more curriculum-related and requiring more school knowledge than Pisa tasks. For solving Pisa’s more contextual tasks, thinking/reasoning ability and general intelligence were rated more important.

In reality, measurement is just very difficult, which means many studies use “proxies”. A proxy is an indirect measure of the desired outcome, itself strongly correlated to that outcome. They are commonly used when direct measures of the outcome are unobservable and/or unavailable.

We could say the report discussed above about “learning with tablets” used perception of learning as a proxy for real learning. Other examples include using the number of books someone has at home as a proxy for socio-economic status or perceived mental effort as a proxy for the load on working memory. Using proxies is fine, but it is vital to be aware that they will most likely tell only a limited story.

It is not easy to measure certain educational outcomes. Assessment, psychometrics and measurement are complex fields. However, an easy, critical way to approach research studies is to ask yourself “How was it measured?”

Christian Bokhove is an associate professor in mathematics education at the University of Southampton and a specialist in research methodologies