From the turn of the 20th century, the middle classes fled to greener pastures and it became home to successive waves of immigrants fleeing persecution or poverty.
The elegant terraces were replaced by overcrowded high-rise estates and social indicators now place the borough as the fourth most deprived in England. Despite numerous regeneration schemes, the area is still associated with high unemployment, flourishing street crime, drug abuse, failing schools and high rates of teenage pregnancy.
Almost three-quarters of its 194,000 population live in council or rented accommodation and unemployment is around 14 per cent - almost twice the London average. Latest police figures also rank the area as the second worst for violent crime.
Run by a council torn by allegations of corruption and mismanagement, its 58 primary and nine secondary schools have struggled to avoid the bottom of league tables. In 1995, Hackney Downs boys' comprehensive famously became the first in the country to be closed by the Government for failing its pupils.
More than 80 per cent of Hackney's pupils are from black and other ethnic minorities, with more than 100 different languages represented in schools.
The proportion of pupils gaining five or more high grades at GCSE has risen by five percentage points to 32 per cent in the past two years. But Hackney's 11-year-olds are still falling further behind their "statistical neighbours" in English and maths tests.
Last year, the borough received its third damning Office for Standards in Education report in three years and former chief inspector Chris Woodhead promptly dubbed it "the worst in the country".
But it was not always so. Sir Alfred Sherman, founder of the Centre for Policy Studies and a former pupil of Hackney Downs who was at school in the 1920s and 1930s, said: "In my schooldays, Hackney was considered a place of opportunity, full of immigrants who worked hard to make a success of themselves.
"The standards in the schools were much higher, the discipline was far stricter and the teachers were better paid and more well-respected.
"Nevertheless, it was assumed that if you became successful you would move on. I left soon after my 17th birthday."
Sir Arthur Gold, 84, former chairman of the British Olympic Association, completed his primary and secondary education in Hackney.
He said: "I loved growing up in Hackney. The stereotypical images of unemployment, violence and drugs just don't ring true for me.
"Most of my friends have gone on to achieve great things, and I think it is marvellous that these children are so optimistic too."
Dr Michael Goldstein, vice-chancellor of Coventry University, was a Hackney schoolboy in the 1940s and 1950s.
He said: "As a child, I rarely left the Stoke Newington area except maybe to go on a bike ride to Southend. It was ethnically diverse back then too, but I didn't regard it as unsafe at all.
"I left in my early 20s but regularly came back to visit. It has certainly got much dirtier, busier and noisier, but it has an excitement and buzz that it didn't have so much back then."