A survey for the programme found that nearly two-thirds of four-year-olds started formal education in primary reception classes this September.
Parents are increasingly reluctant to start children's formal education too early, but are "blackmailed" into doing so, the programme claims.
Parent Sally Harris said: "The thought that I might have a choice is a joke really - there is no choice." She is sending her four-year-old son Max to St John's primary school in Weybridge, Surrey, despite feeling he isn't ready.
"He can't hold a pen and he can't write - it's farcical", she said. The school told her Max's place was ready, and she was convinced she would lose it if she did not send him, she added.
John Horrell, St John's head, admitted schools may not be the best place for four-year-olds. But he said: "I have no choice about taking four-year-olds. I need to fill classes because I need to pay salaries and balance the books - each child carries an amount of money on their head."
Mr Blunkett told Panorama he accepted pressure by schools on parents was "a genuine problem", but that tougher guidelines on admission procedures would ensure no child lost a place by delaying starting school until five.
The programme also features Professor Kathy Sylva, head of the Government's early-years research programme. She said: "There is a lot of research that children under six, and especially under-fives, really have the best foundation for formal learning, literacy and numeracy, in a more play-based early childhood programme."
Panorama - Failing at Four, will be on BBC 1 on Monday at 10pm.