Criticism of the tests, used by almost 200 schools in Britain, emerged after a leading specialist school was found to have flouted the rules for almost a decade.
The George Spencer foundation school, one of the country's first specialist technology colleges, has been banned from selecting pupils enrolling in September 2007.
It has been ordered to rewrite its admissions policy after complaints that it used aptitude tests - designed to identify pupils with "potential" - to pick the brightest.
The ruling by the Office of the Schools Adjudicator reflects deep-rooted concerns over the controversial measurements, used by half of the Government's new academies and roughly 160 specialist schools. It is likely to renew pressure to scrap selection, two years after the Commons education select committee said the process was deeply flawed.
At present, schools can select 10 per cent of pupils by aptitude - often defined as pupils' potential to perform - in one subject. But critics question whether aptitude can be differentiated from "ability".
A report into the George Spencer school, Nottinghamshire, found that measures supposed to identify pupils with a "gift or talent" in technology were flawed. Chris Whetton, head of assessment and measurement at the National Foundation for Educational Research, was asked by the adjudicator to investigate.
He found local primary heads were referring children to George Spencer based on existing "ratings of attainment", not their potential. Mr Whetton told The TES: "The essential difference is that aptitude is fitness for a particular thing, whereas ability is about generalised intellectual functioning. But unless schools can call upon a psychologist, they are going to find it very difficult to distinguish between the two."
Susan Jowett, principal, said its admissions system had worked efficiently for many years, adding: "The school would wish, in the long term, to continue to select 10 per cent of the new intake on aptitude in technology.
The aptitude v ability debate will continue."
Three years ago, Philip Hunter, the chief adjudicator, criticised 10 schools in Hertfordshire after complaints that they were selecting on ability rather than aptitude. He said at the time: "One of the difficulties is the law uses these two words as if they were separate things, and actually they are not."
Alan Parker, who made the ruling on George Spencer, said it was not known how many more specialist schools were breaking admissions rules, because adjudicators could only investigate when complaints were made.
Selection by ability is banned, although schools with specialist status in modern languages, music, arts or sport can select a proportion of pupils.
The Department for Education and Skills denied there was any confusion.
"Although there is no statutory definition of 'aptitude', it is generally described as being a gift or talent that can be developed, whereas 'ability' is seen as being related to achievement," said a spokesman.