Unpick the data to get the true measure of a study

5th October 2018 at 00:00
When weighing up the value of research, it is vital to understand the metrics used, argues Christian Bokhove

In many studies, we read that things have been measured. In the natural sciences, where we have things like temperature or distance, this can be straightforward, partly because metrics have been standardised. In the social sciences, this is less easy.

How do you measure intelligence? Or if a pupil knows enough about maths? And how do you measure “mindset”?

When you read a study, it can be useful to unpick how important variables have been measured. This can then help in judging the validity of the measure: does the instrument really measure what it purports to measure?

For an example of why this unpicking is necessary, look at some of the early reports on the advantages of tablet computers. I recall one that talked about positive effects on students’ learning. When you started to read the report, you found they had asked students whether they thought tablets had a positive impact on their learning.

Now, this can be a perfectly fine objective to explore in a study. However, we should be clear we are talking about the perception of whether something has been learned, rather than actual knowledge acquisition.

Many studies 'use proxies'

Take international assessments like Pisa (the Programme for International Student Assessment) and Timss (Trends in International Mathematics and Science Study). Pisa looks at 15-year-olds, Timss looks at grades 4 and 8. Both yield country scores that allegedly tell us something about maths education, and they are often used interchangeably to say if it is improving.

But when you dive deeper, you see the maths differs. In a rating study, Timss tasks were seen as more curriculum-related and requiring more school knowledge than Pisa tasks. For solving Pisa’s more contextual tasks, thinking/reasoning ability and general intelligence were rated more important.

In reality, measurement is just very difficult, which means many studies use “proxies”. A proxy is an indirect measure of the desired outcome, itself strongly correlated to that outcome. They are commonly used when direct measures of the outcome are unobservable and/or unavailable.

We could say the report discussed above about “learning with tablets” used perception of learning as a proxy for real learning. Other examples include using the number of books someone has at home as a proxy for socio-economic status or perceived mental effort as a proxy for the load on working memory. Using proxies is fine, but it is vital to be aware that they will most likely tell only a limited story.

It is not easy to measure certain educational outcomes. Assessment, psychometrics and measurement are complex fields. However, an easy, critical way to approach research studies is to ask yourself “How was it measured?”

Christian Bokhove is an associate professor in mathematics education at the University of Southampton and a specialist in research methodologies

Subscribe to get access to the content on this page.

If you are already a Tes/ Tes Scotland subscriber please log in with your username or email address to get full access to our back issues, CPD library and membership plus page.

Not a subscriber? Find out more about our subscription offers.
Subscribe now
Existing subscriber?
Enter subscription number

The guide by your side – ensuring you are always up to date with the latest in education.

Get Tes magazine online and delivered to your door. Stay up to date with the latest research, teacher innovation and insight, plus classroom tips and techniques with a Tes magazine subscription.
With a Tes magazine subscription you get exclusive access to our CPD library. Including our New Teachers’ special for NQTS, Ed Tech, How to Get a Job, Trip Planner, Ed Biz Special and all Tes back issues.

Subscribe now