The government has promised to halve class sizes in primary and middle schools for key subjects, starting this spring.
With many teachers facing classes of up to 40 pupils - the legal limit - the announcement was welcomed by unions and reformists.
The education ministry will begin recruiting teachers this April for the small classes and plans to introduce them across the country over the next five years.
Tokyo's schools will be the first to offer the classes of no more than 20 pupils, initially in three subjects chosen from English, Japanese literature, maths, and science.
The new classes will be created by dividing, for example, two classes of 30 pupils into three. The classes will be split to enable teachers to teach pupils with similar ability.
Reformers hope the changes will mean more individual attention for pupils and encourage children to be more autonomous and creative.
The current education system has often been criticised for its over-regimentation. Rote learning geared to passing exams wins at the expense of promoting creativity interpretation and independent thought.
Japan's teacher unions welcomed the plan as it means jobs for many of their unemployed members. A declining birth rate had seen a sharp fall in demand for teachers over the past few years.
To keep costs down, the government wants eventually to reduce teacher numbers to current levels in 10 years' time. "The idea is to maintain the number of teachers in view of the continuing drop in student numbers owing to the falling birthrate," said a ministry spokesman.
However, over the next five years many more teachers will be needed to meet the initial demand. The education ministry says it has the approval of the finance department for 22,500 more teachers by 2005, starting with 4,500 extra primary and middle-school teachers in April.
But local authorities are worried about where the extra cash will come from to pay for the teachers.
Currently, the government funds half the cost of each teacher's salary, with local councils paying the rest. The ministry has said it would retain this system so local government would have to bear increasing staff costs if they hire more teachers for smaller classes, officials said.