How should the countries of Europe deal with Islam and the Turks? Should they try placing restrictions on interaction with the latter? The idea has a longer history than most people realise. Sir Noel Malcolm reminds us early on in his new book that the Lateran Council of the Catholic Church in 1189 issued a canon banning sales of the strategic goods of those days – weaponry, iron and timber – and also the piloting of ships. They were perhaps distant predecessors of the sanctions currently being discussed in the US Congress. But the canon did nothing to stop the advancing Turks, who went on in the next two-and-a-half centuries to destroy the Byzantine Empire, conquer Istanbul, and make the city their springboard for invasions and conquests in eastern Europe and the Mediterranean world.

In 1450 – at the start of this magisterial volume – scholars, philosophers, theologians and diplomats across Europe struggled to understand Islam and the Ottoman apogee, a threatening rival civilisation. By its conclusion, in the mid-18th century, they were discussing the reasons why it had grown weaker and why the Turkish Empire was becoming a candidate for imperial expansion.

Useful Enemies is the second of three books by Malcolm that open new windows into our understanding of relations between Turkey and Western Europe in the early modern period. Malcolm’s previous book on Islamic and Turkish contact with Europe, Agents of Empire, was a colourful account of the warfare and derring-do of the contest between Venice and the Ottomans in the Balkans. A third book is on the way, dealing with “actual human interactions”: Muslim slaves in Europe and Christian captives in Europe, Moriscos and missionaries.

Both the volumes published so far are wonders of erudition, tracing ideas and attitudes back to their origins. No language or text is too obscure or too detailed to avoid Malcolm’s lucid dissection. No document seems to have a context that he does not know and succinctly explain. So although some of the views cited about the Ottoman Empire in early modern Europe are already familiar to specialist readers, this book traces widely scattered ideas across the literature of Christian Europe, drawing them together into a sharply focused continuous narrative which reflects not just the evolution of the Ottoman Empire, but the development of modern political philosophy in Europe and shifts in European political attitudes, such as a growing interest in religious toleration and a dislike of despotism.

Despite the fact that the word “enemies” is part of the book’s title, this is not an unqualified account of “otherisation”, demonisation and hostility – although, to Malcolm’s clear distaste, all of these are regularly encountered. A distinguished Hobbes scholar, Malcolm became interested in the Balkans and Islam because of the war in Bosnia in the 1990s. Entering a field entirely new to him, he reversed some of the previous interpretations on the subject. Overall he detects “a whole gamut of attitudes” including “fascination, admiration and envy”.

Nevertheless, European admiration seems to have been mostly for the military qualities of the early Ottomans and the way troops were disciplined. However, confronted by a larger and dangerous enemy state, early-modern European views of Ottoman Turkey can read somewhat like 20th-century Kremlinology. One of the best surveys of Turkey during this period is

Sir Paul Rycaut’s The Present State of the Ottoman Empire (1666), a comprehensive and still very readable survey of Ottoman government and society. It was a remarkable achievement, written after living only five years in Turkey, but it exudes an intense dislike and fear of a dangerous alien empire.

Attitudes were nevertheless changing. A turning point came in the second half of the 16th century with the emergence of a “new paradigm”. The late-medieval notion of the Turkish Empire as “a malevolent and oppressive regime, inspired at least in part by an evil religion” gave way to something much more subtle. The Ottoman Empire was viewed as “a comparatively well-ordered system” of government, judiciary and civil and military conditions. “…[S]ignificant aspects of civil life were better arranged than their equivalents in Western Christendom. It was a system, apparently, in which ordinary people enjoyed real material benefits, and were generally content to obey their rulers.” This sounds not unlike the arguments that intelligent Chinese make in the West today about the advantages of a system in which decisions are swiftly implemented without the delays and compromises of democracy.

This perception (which conflicts dramatically with almost all European views of Turkey in the final Ottoman century after 1800 when “backwardness” had become a further new paradigm for Western observers) could be used in the 17th and 18th centuries, either to make contrasts with Western rulers that were unflattering to the latter, or even to urge the adoption of Ottoman methods of doing things.

As the Enlightenment arrived, some people even contrasted Trinitarian Christianity unfavourably with what they thought was the much more rational Unitarianism of Islam. Voltaire was among those writers who saw Islam as a device for “decentring” Christianity and opening it up for criticism. Henry Stubbe, a late-17th-century English thinker, and Henri de Boulainvillier, an early-18th-century French one, both argued – though seemingly did not dare to publish – that Mohammed was not an impostor, but “an exemplary religious leader”.

By then, however, other writers were groping for explanations of the increasingly visible weakness of the Ottomans while Europe advanced. The Empire now appeared as a despotism that had been weakened and curtailed.

On the other hand, obstructing a despot might actually protect the system he headed. Montesquieu suggested that things blocking the way of despotism might be the very ones that enabled it to survive, whereas: “Reducing everything to the despot’s will might produce total instability and self-destruction.”

Why were some states, such as the Ottomans, more despotic than others? As scientific explanations superseded moral ones, a variety of possible factors was cited. Montesquieu thought that the explanation must be that climate and material conditions determined political behaviour. The Ottomans, being products of a warm climate, would never conquer Northern Europe, because “soldiers from the south could not tolerate its low temperatures”. But as Malcolm points out, other writers were aware that there were extremes of climate in Turkey, and that the Greeks lived in the same climate as the Turks while being noticeably less martial.