What you said
I would say to the boy, "No PE equals no break or lunchtime. You have two minutes to make your decision." If he wanders off, you need to alert someone, but the consequence still stands. You cannot tolerate this - other pupils will start joining in if you do, and you are responsible for the group's safety as well as learning.
The expert view
If a pupil simply refuses to join in with PE, there are two approaches, and you can do them both simultaneously.
1. The Sherlock Holmes. There may be some underlying explanation for this behaviour other than merely obstreperousness. He may be embarrassed about changing or being bullied in the bathrooms, or it could be the result of a million other triggers that catalyse rebellion.
While this does not justify him sitting out PE, what it does entail is the teacher amending the problem if one exists - for example, by making sure that bullying is stopped. A chat and some reassurance could go a long way.
2. The Undertaker. Like the former World Wrestling Federation's famed Bret "Hitman" Hart, most pupils simply need to see that the school isn't prepared to put up with their behaviour and that, if they persist, they will incur the full gamut of wrath the school possesses.
So, follow the standard path of referrals, sanctions, phone calls and escalations. And persist: it may take some time for this young man to work out that it is worth it to comply. While he may not see the point of physical activity, his heart and lungs will thank him over a lifetime if you instil the habit in him. Good luck.
Tom Bennett is author of The Behaviour Guru and Not Quite a Teacher. His latest book, Teacher, is out now, published by Continuum. http:behaviourguru.blogspot.com
Post your questions at www.tes.co.ukbehaviour.