Those first-day feelings of nervousness and enthusiasm were compounded at the Brooklyn Community High School of Communication Arts and Media (BCAM) by the fact that the entire school was new, an experiment in small and democratic secondary education launched by idealistic teachers in a tough, poor area of New York City.

“I’m giddy,” said one, Kevin Greer, after he welcomed students to his English class. “They’re going to feel sorry for kids who don’t go to our school.”

Down the hall, another teacher, Lavie Raven, made up a new name for his class in the humanities. “The Next Sun Rising,” Raven dubbed it. “Why not?” he asked, smiling. It was a new year in a new school. “Call it anything you want.”

Also in those enthusiastic first few weeks, a social worker carried chairs outside to the paved playground encircled by a chain-link fence and convened a self-empowerment group for girls who, in this inner-city neighbourhood called Bedford-Stuyvesant, suffered from low self-esteem about their futures and their bodies.

“Before, I had a terrible outlook on life,” said one of these girls, Lateefah. “Now I’m not wasting my time on nothing. I’m going straight for what I need.”

Other students, too, seemed to like the school immediately. “It’s fun and small and it’s loose,” said one, Moses, who added that he was convinced it would help him to reach his unwavering goal of going to university.

Reality bites

Three years later, things were very, very different.

Moses had grades so low he was at risk of dropping out, provoking his mother to break down in sobs. Lateefah had left in frustration. So had many of the teachers, including Raven. Some students stole a laptop computer in the middle of the school day. Another was caught in the crossfire of a nearby robbery and shot in the arm, prompting an impassioned debate over whether to set up metal detectors.

Of the 104 students in the inaugural class, only 60 were left, and only half of those were on track to graduate.

“I’m not as optimistic as I was when we started,” said Greer, clearly deflated. “For years I loved my job and now I don’t. There’s nothing to love about trying to do some stuff and having everybody resist it.”

BCAM’s story is told in an ambitious documentary film, The New Public, which exposes the realities of urban education in the US and is making the rounds of American teaching associations and teacher training institutions.

The moral is simple, says Anand Marri, an associate professor at Teachers College, Columbia University, one of the nation’s foremost teacher training providers: there are no right answers to the question of how to fix the system.

“Unlike other documentaries where there’s a very clear direction, this film allows for multiple viewpoints on the role of community, the role of parents, the role of kids, the role of teachers, the role of administrators as ways of thinking about urban public education,” says Marri, who once taught in secondary education himself and is so impressed with The New Public that he is incorporating it into the Columbia teacher training curriculum.

“Everybody feels like they know high schools because they went to high school,” he says. “But teaching high school was the hardest job I’ve ever done. And this is showing that it’s not easy. You have to understand the cultural context and the milieu that you’re working in.”

That was a particular challenge in Bedford-Stuyvesant, a slowly gentrifying, predominantly black area of Brooklyn, whose previous starring role on film was in Spike Lee’s Do the Right Thing. Although the once-high crime rate has been substantially reduced, it’s still a tough place to grow up.

Ninety-six per cent of the students at BCAM are black or Hispanic. Many arrived on that first day in 2006 scoring well below grade level in reading and maths. Some were bullied on their way to and from school. A few had parents - most often mothers - who pushed them to excel, but others enjoyed little support at home.

All this contributed to a descent from the school’s early idealism, which included a weekly dance through the corridors every Friday afternoon, to the harsh reality of discipline problems so persistent that a second dean was appointed to deal with them.

Although the energy of the original teachers “was incredibly infectious”, the documentary’s director Jyllian Gunther says that “the take-away is that, yes, it’s very complicated, and that the issues are bigger than school”.

Still, the teachers in that first year were certain they could overcome these obstacles. Greer, for example, who had taught in much bigger schools in the equally troubled neighbourhoods of South Central Los Angeles and the Bronx, called BCAM “a school that, if we were students, we would dream of going to”.

There were energetic showcase nights at which students showed off their singing, dancing, drumming and yoga skills. The idea was to take the students’ culture into account, says James O’Brien, a former DJ and the founding principal of BCAM. “Far too often schools imprint themselves on the student rather than acknowledging who the student is,” he explains.

That proved harder than anticipated. Many on the faculty, including O’Brien, lived in the neighbourhood but they couldn’t overcome its influence - something the students seemed to understand more than the adults did. Often poignant and insightful about their circumstances, with sudden and scene-stopping insight, they were also stubborn, obstinate and occasionally disrespectful. They sensed the enthusiasm going out of things.

One expressed his intention to get kicked out. “I don’t know what to do about that,” a teacher said. Many in Greer’s class in that first year were yawning and visibly bored; he wasn’t getting through to them, he realised.

“If you saw the faces of those kids - I thought, ‘Are you kidding me?’” Marri says.

When Lateefah overcame her shyness to perform a rap she wrote in front of the whole school but then gave up during a science experiment she didn’t understand, it was an example of the roller-coaster ride of educating teenage students.

“That’s a great metaphor not only for what happens in that school but what happens in many start-up schools,” Gunther says. “A teacher’s day is a roller coaster. You have one great class, one terrible class. A student has the same experience. It’s realistic and bittersweet.”

A dose of the bitter side came in a meeting of the faculty at which a weary O’Brien conceded that graduation rates had not improved from an abysmal 40 per cent, and that students were underperforming on required standardised tests.

“We’re not proud of those numbers,” he said, mindful that the school had fallen short of its early promise. “We dropped the ball on some fundamental things.”

After the shooting of its student, BCAM became sidetracked into a debate over whether to require daily scans through metal detectors. Outraged parents complained that this would send a message that their children were assumed to be criminals. When students stole a laptop, there was a spirited discussion about whether or not to allow them back.

“That’s all real. It’s real in the battlefield of public education,” O’Brien says.

Inside story

Showing the battlefield through a camera’s lens, and from within schools, in fly-on-the-wall documentaries and films, has become one of the most effective ways of igniting conversation about American public education. And this trend has been echoed in the UK with recent series such as Educating Yorkshire. The 2010 documentary Waiting for Superman, about the shortcomings of US education, the frustrations of urban families and the power of teaching unions, enjoyed unexpected popular success.

Gunther knew several of the original teachers at BCAM, who asked her to teach a documentary film-making class there; some of the footage in her film was shot by students.

“Most educational films have an agenda or are pushing a policy, or they’re the story of a single hero, like a principal, and not a sense of the bigger picture,” she says.

Her intention was “basically just to do a film that was not a top-down hero kind of thing. Everyone wanted people to know that we’re in a period where teachers are being blamed and policymakers are making policy without being in the schools and having a sense of what is going on inside them.”

The New Public - which drew its budget from private and government grants and two crowdfunding campaigns - has been televised nationally and screened at 40 universities and by teacher associations. Gunther believes audiences will see that, “Wow, here’s a parent who did everything they could and their student went down and up, and a teacher who had years and years of experience and still struggled.”

Gunther says that, unlike Waiting for Superman, her film does not have a political message. All the same, it has appeared against the backdrop of a scathing report by the National Council on Teacher Quality. The report blasted US teacher preparation programmes for an “incoherence” in the way they cover classroom management, including strategies for handling misbehaviour and encouraging student interest.

What The New Public preaches, Marri says, “is knowing where the kids are coming from and understanding what they bring to the table, and incorporating that”.

Even the students at BCAM with the most potential didn’t make it in some cases. Others did and went on to university - including Moses, barely. (Lateefah is now studying to become a paralegal.) Greer changed his class to make it more accessible to his students, who re-engaged as a result.

BCAM, which now has 469 students, remains exactly in the middle of New York schools in terms of performance, according to city statistics. Only a little more than half its students pass science, two-thirds maths and half global studies on standardised tests. Only half go on to university within a year and a half of leaving. But graduation rates, although still below city and national averages, have improved to 70 per cent.

“I wouldn’t characterise that as failure,” Marri says. “I would characterise that as, ‘Give it time and see what happens.’”

And when the first handful of students graduated, a huge roar went up. “You made it,” O’Brien said.

WATCH A CLIP

The producers of The New Public have provided TES Connect with nearly 10 minutes of footage, to give teachers around the world a flavour of what it is about their film that has engaged the US education community.

To view these scenes, go to tesconnect.comthenewpublic.