Some pupils are being turned off physical education because they are overweight, shy or have a poor body image, according to school inspectors.
To re-engage these pupils - who are predominately girls - there must be more choice and more "high quality collaborative opportunities" to build confidence, particularly in secondary, HMIE suggested in a report published last week.
Traditional sports such as rugby, football and hockey may "inadvertently exclude" some groups of pupils, inspectors said.
Activities such as playing with a Frisbee, handball, cheerleading, dance and "boxercise" may be more successful in motivating pupils to become more physically active.
The report said: "As teachers, we need to further our awareness of the barriers some young people face when coming to physical education, improve breadth and choice of experiences to meet their needs, and promote an increased awareness of opportunities available after school and in the community for pupils to participate in physical activity.
"Specific skills which help include experiences where learners have to work in pairs, small groups and teams to provide solutions to problems and challenges, allowing them to perform more successfully."
The report, Physical Education - a portrait of current practice, includes examples of how PE can be used to develop the four capacities of A Curriculum for Excellence.