Reforms must check the power of Chris Woodhead, the chief inspector, MPs heard this week.
Mr Woodhead's Office for Standards in Education should have a management board and a chief executive, according to Greg Wilkinson, who was formerly responsible for the Audit Commission's contribution to local authority inspections.
OFSTED is the only inspectorate constituted as a non-ministerial government department, said Mr Wilkinson, now an education consultant with Anderson Consulting.
He told the education select committee, currently scrutinising OFSTED's work, that: "While it may in theory be accountable to the Prime Minister and Parliament, it is questionable whether either have the time or the information to scrutinise OFSTED's performance and hold it to account."
Mr Wilkinson criticised the inspection reports on local education authorities for producing conclusions based on weak evidence. He cited cases of critical judgments on local authorities' overall performance based on the views expressed by heads or teachers at only one or two schools.
Local authorities also complained of wild judgments, bias, prejudice and discourteous inspectors carrying out evaluations at the fringes of their competence. In written evidence to MPs they criticise an uneven quality of accredited inspectors, and an expensive service which has lost teachers' and local authorities' respect.
Southampton, one of the councils which wrote to the all-party select committee, said: "There is little evidence that OFSTED is providing an objective view of the education system to ministers.
"The evidence should be allowed to speak for itself, rather than be selectively trumpeted to match the music of the moment."
It added: "The core cadre of HMIs give good value. The itinerant players are so variable in quality that overall their evaluations have limited credibility."
Gloucestershire urged abandonment of the full framework for the inspection of all schools. High performing schools needed different measures against which they can be judged, it said.
"This will enable the exposure of the smug school in the good neighbourhood which routinely achieves results which may be high but which add little value to pupils' existing attainments."
Graham Lane, chairman of the Local Government Association, told MPs the LGA believed the chief inspector of schools needed a high profile to communicate the national importance of education. He said OFSTED represented value for money. Its costs amount to 0.6 per cent of the total education budget. But the LGA believes OFSTED should be managed by a board and there should be clearer rules for appointing the chief inspector.