There was no evidence to support the assertion that they produce better results, professors Stephen Gorard, James Tooley and Richard Pring told MPs this week.
The Commons education select committee is conducting an inquiry into secondary school diversity including specialists, city academies, faith schools, city technology colleges and foundation schools.
Professor Pring, from Oxford University, said the specialist policy was daft. Referring to specialist sports schools, he said: "There is no justification for having PE experts in Wantage."
Professor Gorard, from the University of Wales, Cardiff, said the White Paper on diversity was based on flawed evidence. He criticised research produced by the Technology Colleges Trust, which purports to show the sector's better GCSE results, because it failed to look at how the schools performed before they became specialist.
He claimed specialist schools led to more segregation and there was no evidence that they had superior GCSE results.
He said: "If you take away the extra money that specialist schools get, what is left that people want?"
Professor Tooley of Newcastle University, who supports a voucher system where parents get state funding to send their children to the school of their choice, said: "The current system is top down diversity imposed by the Government. Parental choice would not necessarily lead to diversity, it could lead to ranked uniform schools."