The publication of this year's tables has put her Hackney school back in the spotlight - this time because its results have plummeted.
Just over a quarter of 11-year-olds at Gainsborough primary reached the required standard in English maths and science tests this year - down from around two-thirds last year. Translated into numbers of children, this means about 20 of the 1998 pupils made the grade compared to about nine this year.
The difference is enough to make Gainsborough the school with the sixth biggest drop in results in the past year - its scores are the 54th worst in the country.
Ms Milsom, who became head in 1997, believes the school is still well on the road to recovery but years of upheaval have had a lasting impact.
The high 1998 performance had seemed particularly remarkable - only two years previously inspectors had identified serious weaknesses and test results had sunk to rock bottom.
Last year Ms Milsom credited the introduction of basic management structures and the numeracy strategy's teaching techniques for the high test scores.
Although she still believes these had a significant impact, she concedes that the importance of other factors - deprivation, poor English and attendance - are still not fully recognised. One in 12 of the school's pupils comes from a nearby travellers' site and has problems with literacy and truancy.
Ms Milsom said: "With a small school like this your cohort is a class of around 30 children and you are judged on their performance as if that tells the whole story of the school."
Ms Milsom also says that last year's 11-year-olds were an exceptionally bright group. If last year's results are ignored, the school's test scores can be seen to be consistently improving, although slowly, she added.
The class of 1998 contained more girls than boys, more autumn-born children, more native English speakers and fewer very poor children.
In 1999 more than 70 per cent of the class qualified for free school meals, 20 per cent were not fluent English-speakers and there were more boys than girls.
Ms Milsom said: "You can change the nature of the school and its management systems relatively quickly but actually changing results of a school will take longer to achieve."