The tests showed that even if pupils did use calculators to get the right answer, many were unable to interpret it because they did not have enough understanding of decimals, particularly where money is involved, the Qualifications and Curriculum Authority report said.
Despite having a special calculator paper, pupils are still tackling questions by trial and error using pencil and paper. These pupils are missing the point of the test, says the QCA report.
Professor David Reynolds, former chair of the Government's numeracy task force, said: "It is really quite extraordinary.
"This can probably be related to a lack of clarity in teachers' beliefs about the appropriate use of calculators and a historic lack of focus on this issue in teacher training."
The use of calculators in primary schools has always been controversial. The launch of the National Numeracy Strategy was clouded by a row over their use. The Department for Education and Employment claimed calculators would be banned from primary classrooms but was denounced by task force advisers who said this distorted their views.
The numeracy strategy, to be implemented from next September, suggests that their use be limited but the final decision is left to schools.