The father of four, with almost 25 years' experience as a care worker, was tried at Liverpool Crown Court in 1999, accused of abusing eight boys in his care.
He was eventually convicted of 22 out of 47 charges made against him, including indecent assault and serious sexual assault.
The social worker was in Wakefield prison with some of the most notorious inmates in the country, including the mass murderer Harold Shipman, before being transferred to Risley in Warrington.
But back in his native north-west he was beaten up three times by fellow prisoners.
He served almost four years behind bars before he was freed in March last year after London's Court of Appeal heard that the evidence of two of his accusers was unsafe.
Mr Williams-Rigby, 59, said: "I had no means of disproving what they were saying - it was their word against mine and I was not strong in the witness box.
"It is a sad, sad state of affairs that allegations like this can be made and innocent people are left to rot in jail attempting to clear their names."
Mr Williams-Rigby was only freed after his solicitor, Chris Saltrese, managed to track down two other men who shared dormitories with two of the boys making the allegations. One was serving a life sentence at the time for murder, but investigating officers had failed to find him to corroborate their case.
The two men said the abuse had not taken place and all guilty verdicts against the care worker were quashed.
Mr Williams-Rigby is now pursuing legal action against the police.