Ken Tompkin, a former headteacher and marker with the Edexcel board, said students may have lost marks needlessly because materials sent to help markers decide what score to give were inaccurate.
The controversy centres on an intermediate maths paper, taken by at least 100,000 pupils this year.
On question nine, candidates were asked to draw a scattergraph describing the relationship between the number of pages in each of 10 books and the weight of each book. The TES has seen an overlay sent to examiners, designed to tell them where pupils' answers must lie on the graph in order to be judged correct.
But the axes on the overlay did not line up with those on question papers, making it impossible for examiners to position the overlay properly and so decide accurately if the pupils were correct.
Mr Tompkin said: "It could have been placed in different positions by different examiners, giving different results depending on how it was used.
"I had to give no marks to some of the candidates even though it was marginal. I could not be sure that giving them nought was right."
Mr Tompkin, former head of Hurstmere foundation school for boys, Bexley, south London, said the inaccuracies may have cost pupils three out of 100 marks. He said there were also problems with another question, worth two marks, where pupils had to mark a circle on a map. Again, the faulty overlay for examiners made it hard to gauge if they had got it right, he said.
Mr Tompkin raised a string of other complaints, including ambiguities in the mark scheme, in a meeting with Edexcel last month. But he said it appeared nothing had been done to make sure students were not disadvantaged.
The intermediate maths paper was one of several to be transferred to traditional marking at the last moment in July, after computer problems hit Edexcel's trial of on-screen marking of one million GCSE and A-level papers.
Mr Tompkin suggested the problems may have come about because Edexcel had intended examiners to mark the paper concerned on-screen this year, rather than traditionally.
Stevie Pattison-Dick, Edexcel media affairs manager, said support materials, including overlays, had been produced 18 months ago, before any decision was taken on whether the paper would be marked on-screen.
She said Mr Tompkin had been the only examiner to raise these complaints and that the board had been happy with his marking.