The housemaster at Harrow school regularly had his mail screened and received police protection because of death threats from angry supporters.
In all, Mr Elleray refereed 1,780 games.
He also described how his job at the pound;7,450-a-term school was his escape from the world of soccer.
The 50-year-old, who retired last season, said he often wore a disguise so he could safely leave a ground after a game.
Mr Elleray, director of boarding at Harrow, whose motto is Stet Fortuna Domus ("May the fortune of the house stand") spoke in Dublin this week to promote his book, The Man in the Middle.
He said: "Some fans develop an intense dislike and hatred of you. Some are so passionate that if you do something against their team, it is like you did something against their family."
Mr Elleray, who sent off Roy Keane four times, said: "At one point I was getting hate letters and death threats from a small minority of Manchester United fans.
"Fans expect me to be right every time I blow my whistle, and they don't remember my right decisions."
Mr Elleray said referees are more high-profile now than they were 15 years ago.
"Pierluigi Collina is known as the bald guy with the staring eyes - but I'm known as that guy that teaches at Harrow," he said.
He believes he is lucky to have been able to go back to Harrow and switch off. "One wonders how I would have coped if I'd hadnothing in my life apart from refereeing," he said.
Overall, the veteran of soccer said he had mixed feelings about his retirement from the game.
"I miss the knot of nerves in my stomach when I wake up or the adrenaline rush when I set foot on the pitch of a big game," he said.
"I don't miss the abuse. I don't miss having to train on a cold, wet November evening."