I'm totally opposed to grouping schools under one governing body, for just the reasons you give so clearly. It used to be widespread, very party-political and much abused.
I think it would destroy the real commitment to one school you obviously have - don't give that up without a fight just because of slight recruitment problems; many schools have much worse. I also share your concern about the sort of people it might attract and the transport problems.
But above all it would cut across accountbility: governors' duty to carry the can for one school.
In poor areas some schools have serious recruitment problems and others, like yours, minimal ones. Therefore it can't just be poverty; the style of the school in dealing with parents must make a big difference.
Governors whose work makes a difference will attract others with the same aims. A talking shop won't. Nobody wants to join a dud show.
Heads have a part to play here. I feel recruitment problems arise from governors being sidelined or just a formality , rather than because of social problems or excessive workload. Heads can change that if they want to, and will be repaid handsomely.
One recruitment tactic is for the outgoing parent-governor to write to parents in hisher own hand (photocopied and not altered) encouraging them to stand. Heads and governors should inform parents regularly about things the governing body have done; complain where party political influence is to blame for LEA vacancies; and suggest possible candidates.
Joan Sallis's new book, "Heads in Partnership: Working with your Governors for a Successful School", is published by Pearson Education