Most damningly, critics say the Emmanuel Schools Foundation, the charity he set up to run a city technology college and two academies, teaches creationism in science lessons as a way of furthering his evangelical agenda.
But meeting the TES at his King's academy in Middlesbrough, Sir Peter said he has had enough. The accusations, he claimed, were not only false but now threaten the future of academies nationwide.
"The creationist issue has derailed the academy agenda because people are worried that anyone who has different beliefs can build a school and teach what they want," he said. "Creationism is not taught in my schools. That is stark raving crazy. I am not a creationist and I did not build academies to indoctrinate anyone. I am a Christian and have decided to put some of the wealth I have been blessed with into schools."
This week the controversy continued as the Emmanuel Schools Foundation announced that Stephen Layfield, head of science at Emmanuel college, had resigned as a director from Truth in Science, a controversial group of academics and clergy.
The group recently sent a teaching pack and DVD to 5,000 secondary schools promoting the concept of "intelligent design", an off-shoot of creationism, which casts doubt on Darwin's theory of evolution. Last week the Department for Education and Skills said that this was not part of the national curriculum.
Mr Layfield's links with the group has cast doubt on the impartiality of science teaching in Sir Peter's academies.
But Sir Peter said: "Steve Layfield is not standing up in his science lessons and talking about six-day creationism. I have it in writing from him that creationism is not being taught in science. Truth in Science has nothing to do with me and their lesson packs did not go out with my blessing."
Following Mr Layfield's resignation from Truth in Science, the directors of the Emmanuel Schools Foundation released a statement. It said: "Recently, the private involvement of the head of science as a director of Truth in Science has been interpreted by some as reflecting the views of Emmanuel College and its teaching of science. This is not the case."
Sir Peter's best defence is the success of Emmanuel college. It has gone from being one of the worst-performing schools in the country at GCSE to one of the best. King's and Trinity academy, in Doncaster, are also performing well and both are over-subscribed by pupils from a variety of different faiths.
Sir Peter plans to build four more academies, the next in Blyth, Northumberland. But the creationist tag threatens the reputation of his existing schools and his plans for expansion.
The proposed academy in Blyth has run into significant opposition. Teachers union NASUWT has collected more than 1,000 signatures against the plan. It claims people are worried about the religious reputation of Sir Peter's schools and that an academy will divide the town's children.
Previous opposition against a second Emmanuel academy in Doncaster resulted in the plans being shelved, but Sir Peter is sure this will not happen again.
"We have seen it before. There is all this concern, but as soon as it is built there is absolute support locally for what is going on. The people petitioning are presenting a one-sided view," he said.
There is no doubt that the Emmanuel schools have a strong religious ethos.
Principals and senior management teams must be Christians, and all children are issued with a copy of the Bible. At King's there was a recent debate entitled "Is atheism a realistic belief?"
Sir Peter said the constant criticism has caused him a lot of anxiety, so why doesn't he abandon high-profile academies and use his wealth to support other educational charities?
"When you look at academies, you realise that this is the best education a child can get," he said. "There is no other way I could give money that would be as effective as this."
MAN WHO MADE IT
Sir Peter Vardy, 59, went to Durham Chorister school and then the private Durham school, which he left at 15 with one O-level in music.
He then went to work for his father's car dealership, Reg Vardy, where he did stints pumping petrol and panel-beating before moving into sales.
In 1976, following his father's death, Sir Peter took over the business and built it up before floating it on the stock exchange in 1989. He sold all of his shares in the business in February this year for around pound;120 million.
A father of two sons and a daughter, Sir Peter is married to Lady Margaret.
He was knighted in 2001 for services to business and education in the North East.