A conference in Edinburgh last week run by Children in Scotland and One Parent Families Scotland heard that "only a tiny scattering" of men work as carers with young children.
Ian Maxwell, chairman of One Parent Families, said that pay and career prospects had to be improved. At present there were too few men to set up a formal body to support them.
Celia Carson, national development officer for Children in Scotland, said that suspicions had to be overcome about men's motives. The fact that police checks were carried out on all childcare workers, including women, had to be emphasised to parents.
Routinely, no one worked alone with children, Ms Carson added.
Ron Summers, headteacher of Sighthill primary in Edinburgh, said that men carers should explain their motives and the benefits they brought to young children. "It's up to them to say, damn it, I enjoy it."