Many of the horrors of the 20th century are well known to us all: the impassive sadness of the inmates of German concentration camps at the end of the Second World War, the terror of napalmed children in Vietnam, and the pain of the victims of the Biafra famine. But what about the Japanese action in China in 1937 (80,000 raped, 400,000 massacred)? Or the occupation of Berlin by Soviet troops in 1945 (130,000 rapes, 10,000 suicides)?

Modern historical memory is selective, and responds neither to facts nor to ideology. As Susan Sontag points out in her marvellous new book, it’s the photographs that count. Photographs are special because, however much artistry goes into them, there is a large element of chance. Photography is the only art where the difference between amateur and professional is often undetectable: the point is to have the luck to be in the right place, pressing the button at the right time.

The aim of the photographer is not so much to make a good-looking image as to say: “This is how it was, and I was there when it happened, and it could have been you.” It does not matter if a photograph is grainy and grimy, as long as it gives an impression of authenticity. Witnesses who saw two aeroplanes flying into the Twin Towers on September 11, 2001, could not believe their eyes, saying, “it felt like a movie”. As Sontag notes with typical acuteness, this phrase seems to have replaced “it felt like a dream” as an indicator of weightless inconsequence.

Film and television are the currency of fantasy rather than reality: to be real, in the modern political imagination, is to have been photographed.

The political passions of our time are overwhelmingly negative. They are aroused not by the dream of a new Jerusalem, nor by the prospect of improvements in health, wealth, literacy and social justice. “Votes for women” and “homes for heroes” are yesterday’s popular slogans; today’s are on the lines of “never forget” and “never again”. The forces they appeal to are of repulsion rather than attraction; repulsion in particular from the graphic atrocities of war. And if Sontag is right, our negatively motivated politics is essentially the politics of the photograph. “The very notion of atrocity, of war crime,” she says, “is associated with the expectation of photographic evidence.”

Photographs are relics as well as records. They have a fascination that no painting could match. But maybe their vividness is a kind of obscenity, serving to haunt and torment us, or goad us into uncomprehending indignation, rather than to explain. Perhaps things were better in the past, when people could take time arguing over complex political problems, rather than making up their minds with the speed of a photographer’s flash.

There is also the danger that photographs of horror will desensitise us, initial shock decaying into apathy. These are arguments Sontag put forward when she started writing about photography some 30 years ago. Now she is less sure. The analysis she once endorsed has burgeoned into “fancy rhetoric” about the cult of the image and the death of reality.

But if photographs alienate us, they can also give us pause: “there’s nothing wrong with standing back and thinking”.

There can be few books about photography that contain no reproductions of photographs. But Regarding the Pain of Others needs none. Prose of such crisp intelligence turns the old saying on its head: every word is worth a thousand pictures.

Jonathan Ree is an author and philosopher. His books include I See a Voice: a philosophical history of language, deafness and the senses (HarperCollins)