Inspection agency Estyn said the elected members' "poor track record" of taking difficult decisions on issues such as school reorganisation was a barrier to future improvements in services.
Last year, the city's Liberal Democrat-controlled council made an abrupt U-turn when it shelved plans to close Dylan Thomas comprehensive. It was to merge with Bishop Gore as a way of tackling surplus places.
Estyn conceded that some aspects of school reorganisation had been completed successfully, but the reversal of earlier decisions "contributed towards the negative perceptions of the administration by schools".
Estyn made the comments in a new report on Swansea's provision for additional learning needs, child protection and looked-after children.
The report said the council had an inconsistent record regarding its strategic decision-making on education over "many years".
"On occasion, decision-making has been hampered by undue hesitancy and caution and at other times by undue haste and a lack of appropriate consultation. The current administration remains relatively inexperienced and has yet to establish a track record of effective decision-making," said inspectors.
The report said Swansea's councillors were committed to and supportive of education. It described the leadership given by senior officers as "strong and purposeful" and praised much of its provision.
The inspectors said the special educational needs service had effective leadership with managers at all levels providing clear strategy. Statements of special educational need, which apply to 3.4 per cent of the school population, are well-written and prompt few tribunal cases.
Mainstream schools, which cater for 98 per cent of statemented children, cover a broad range of special educational needs. Some 150 teachers in 98 per cent of schools have been trained through the Dyslexia Friendly School initiative.
And eight of the authority's 15 secondary schools host specialist teaching facilities for children with disabilities such as Asperger's syndrome or high-functioning autism.
The LEA has been less successful developing schools' capacity to provide for pupils with a wider range of emotional, behavioural and social disorders.
And many permanently excluded children with statements attend well below 25 hours of alternative education.
Estyn said there were also no important shortcomings in the authority's provision for child protection and looked-after children.
Support for looked-after children is "good and improving" and most children in care achieve in line with expectations.
But their actual attainment is low. In 2004, only 28 per cent of children leaving care at 16 achieved one GCSE or GNVQ - and their increasing number is placing more pressure on staff.
In 2004-05, 374 children were in care, up from 265 in 2001-02.
Mike Day, Swansea's cabinet member for education, welcomed the report and said its recommendations were being analysed.
A council spokesman noted that two schools had closed over the summer, and that a new education scrutiny committee had ordered a study on school occupancy levels and possible solutions to surplus places.
"Our education department has a fine record in providing learning for all ages. The many positive points highlighted in this Estyn report reflect this," he added.