An untimely knee injury put paid to those aspirations. But the man who was inspired to take up athletics by the sight of future Olympic long jump champion Lynn Davies has, in a sense, attained his own personal gold medal by winning the Department for Education and Skills governor of the year title at last month's Welsh teaching awards.
"I didn't think I had the remotest chance of winning. I guess the best I was hoping for was some kind of commendation," says Mr Down.
"It's fair to say I was gobsmacked - when I made my acceptance speech I felt a bit of a fraud."
Staff at Chepstow's The Dell primary school never had such doubts. They describe Mr Down's award as fitting recognition for his tireless work behind the scenes on top of his job as an accountant, and demanding role as deputy leader of Monmouthshire council.
"Graham is the unsung hero of our school. He's an extremely effective worker who makes sure we never rest on our laurels," says headteacher Keith Rowlands.
"As a schoolboy learning how to play rugby I would have died for my PE teacher, and that is exactly the same way I feel now about Graham. All of us are absolutely thrilled for him."
Father-of-two Graham started at The Dell as a parent-governor in 1993. His own children are in their 20s now, but his enthusiasm for the school shows no sign of waning.
"The role of a governor is to set standards and policy for the school," he says. "We try to support the school leadership team and to bring a non-academic perspective into the academic arena.
"My own background is in finance and I like to think I can use some of that experience for the benefit of the school.
"By being a governor I can give something back. If you're involved with a good school like The Dell there's also a great sense of pleasure and fulfilment."
Thirty years after his own athletic ambitions were cruelly ended, Graham remains an enthusiastic armchair fan. "I can still visualise every step of that great 800m duel between Seb Coe and Steve Ovett at the 1980 Olympics,"