Inspectors, who visited 29 colleges offering agriculture courses, say they have been affected by high course costs, falling demand, large catchment areas and low numbers.
Around 13,000 students followed agriculture and horticulture courses in the last academic year. The report says: "Colleges have striven to meet the substantial challenges they face and imaginative strategies have been adopted to achieve growth and increase efficiency.
"Despite strenuous efforts, some colleges have experienced difficulty in managing change effectively, and this has had an adverse effect on the quality of students' experience and the standards they achieve.
"The colleges now see themselves as serving the wider needs of local students interested in tending plants andor animals rather than responding to the specific demands of the agricultural and horticultural industries.
"The period of uncertainty and change is by no means over. Many specialist colleges face further possible reductions in funding for each student and increased competition between colleges makes projected growth rates uncertain.
"Skilful and imaginative management is required if the needs of employers and students, often in remote rural communities, are to continue to be met. "
The overall quality of teaching in colleges was found to be sound or good. Practical teaching was singled out for praise.
But there were concerns that students' contact with teachers had been cut - often without consulting employers.
* Education minister James Paice has urged farmers to embrace the Government's new system of National Traineeships, to be launched in September.
He told a London conference: "Our way forward is to maximise the production we get from our wages bill; to avoid saddling employers with burdens like the Social Chapter, and to compete on the basis of our skills.
"More producers now see the benefits rather than the costs of training. Some have been recognised as Investors in People, but many still need to learn the golden rule that training pays."
Also at the conference, which was organised by training group ATB-Landbase, Tim Boswell, minister for rural affairs, praised the achievements of agriculture colleges.
He added that he hoped colleges would "continue to seek to improve and develop the service they provide".
"Many are looking to the future and striking strategic alliances with local universities in order to retain viability and the provision of high quality education," he said.