In this paper, we propose an account of metaphor identification on the basis of contextual coherence. In doing so, we build on previous work by Nicholas Asher and Alex Lascarides that appeals to rhetorical relations in order to explain discourse structure and the constraints on the interpretation of metaphor that follow from it. Applying this general idea to our problem, we will show that rhetorical relations are sometimes insufficient and sometimes inadequate for deciding whether a given utterance is a case of metaphor. They are insufficient, since rhetorical relations fall short at times of providing a basis for disambiguating between literal and metaphorical interpretations. In such cases, contextual information other than previous discourse needs to enter the picture. To this effect, we bring the idea of external consistency into play. Beyond that, though, we will argue that rhetorical relations are sometimes inadequate to account for coherence, if conceived as relations among sentences only. The reason is that extra-linguistic elements of the situation in which the sentence is uttered may be crucial for getting at the preferred interpretation. To account for these cases, we allow rhetorical relations to connect both with previous discourse and with extra-linguistic situations. In our final refinement of the notion of contextual coherence, we forfeit any appeal to rhetorical relations in favour of Questions Under Discussion (QUD). We defend the view that this account does not only explain the same sort of cases. What is more, it solves the issue of metaphor identification in impoverished contexts.