The Department for Education and Skills would lose up to 2,700 staff if the Tories won power.
Oliver Letwin, shadow chancellor, said the department "typifies the fat and bloated bureaucracy that Labour has created" and that few would miss its "paper-shuffling empires".
He would also scrap the National College for School Leadership and six other educational organisations because he believes they are a waste of money.
The Conservatives revealed plans this week to make pound;5.7bn of efficiency savings in education. It claims these would give schools pound;15bn more each year than they would get under Labour.
The Government has already announced plans to make pound;4.4 bn savings in education. This includes dismissing more than a thousand DfES staff.
The Tories also intend to make big cuts at the Office for Standards in Education, cutting spending on inspections by more than pound;881 million.
They would abolish the National College for School Leadership, which costs up to pound;111m a year to run, and let schools choose where to buy their own training. The Well Being Project, Implementation Review Unit, Office for Fair Access, Union Learning Fund, Adult Learning Inspectorate and Sector Skills Development Agency would all perish.
The party also pledges to spend pound;1.2bn less on local authorities because it plans to make all schools grant-maintained.
But the Liberal Democrats say Tory cuts would be disastrous for special needs children.
The National College for School Leadership defended its work, saying surveys showed nearly three-quarters of headteachers felt it had a positive impact on education.