Too many teachers do not have a strong grasp of the concepts they teach, and even students who get good grades at GCSE are not properly prepared for A-level.
The watchdog's damning report called for a re-think of the way the subject is taught and for an end to "teaching to the test".
Miriam Rosen, Ofsted's director of education, said: "Too many students do not expect to understand mathematics. The current approach to teaching (the subject) is not giving students the understanding they require, and this must change."
Teachers' tendency to prepare pupils for tests without ensuring they understood the subject was exacerbated by textbooks written to match specific syllabuses, the watchdog said.
In one class, pupils could see nothing wrong after they calculated the circumference of a circle with a diameter of 5cm to be 1.6cm.
The report said: "It became clear that the students could not confidently picture, draw or make with their hands a circle of diameter 5cm, nor did they visualise the circumference of the circle as the distance around the circle. Students had been taught to use a formula but not to understand its meaning or purpose."
Ofsted's findings are based on visits to 26 schools and colleges last year.
Most teaching was at least satisfactory in preparing for exams, but more than a third of schools had a shortage of suitably qualified maths teachers. Even staff with good subject knowledge often had a restricted range of teaching strategies, and many lessons lacked the imagination and flair necessary to get the best from students.
The report said: "Teachers were effective in showing students what to do, but mathematics became an apparently endless series of algorithms for them, rather than a coherent and interconnected body of knowledge."
Ofsted also found that many teachers spent too long talking, and gave pupils too little time to work things out for themselves.
The inspectorate called on the Department for Education and Skills and the Qualifications and Curriculum Authority to ensure that the review of GCSE and A-level changes exams in a way which will encourage greater understanding.
Jim Knight, schools minister, said this would happen, but added: "These conclusions are based on a small-scale survey. We do not accept that tests can be passed without properly mastering and understanding the subject - and preparing pupils to demonstrate mastery of the curriculum does not mean 'teaching to the test'."
"Evaluation: mathematics provision for 14 to 19-year-olds" is available at: www.ofsted.gov.uk